How to automatically replace zlib with the new zlib-ng: I need help and user contributions to write a tutorial

[netcorefan1]

This is a tutorial on how to automatically force each packages which depends on the old zlib to use the new zlib-ng (compiled with ZLIB_COMPAT flag) in order to take advantage of the next gen instructions. Running zlib-ng natively is still the preferred way, but requires changes to the source code and this is not something that all developers are willing to do (at the time of writing, there is no port that uses zlib-ng).

I initially proposed two different implementations (#30918 and #30937), but unfortunately there are a couple of issues that cannot be solved. Although is not possible to get this into the upstream, it can still be done locally by the user and (hope) safely. This tutorial shows each step and is not production ready. Probably there are bugs I'm not aware of and needs improvements, suggestions and fixes by the experts.

This is for Windows x64 dynamic linkage (the default triplet on Windows). Other configurations requires adjustments.

---

Required user environment variables (paths can be adjusted to user preference):
VCPKG_OVERLAY_PORTS C:\dev\vcpkg\overlay-ports\zlib
VCPKG_OVERLAY_TRIPLETS C:\dev\vcpkg\overlay-triplets
VCPKG_DEFAULT_TRIPLET x64-windows-custom

Required files:
C:\dev\vcpkg\overlay-ports\zlib\portfile.cmake (not sure if an empty one is still fine):
set(VCPKG_POLICY_EMPTY_PACKAGE enabled)

C:\dev\vcpkg\overlay-ports\zlib\vcpkg.json:
The only purpose of zlib-ng dependency is to kick off update (and rerun our overlay port each time a new version of zlib-ng is published in the repo).

{
    "name": "zlib",
    "version": "1.0.0",
    "port-version": 1,
    "dependencies": [
      {
        "name": "vcpkg-cmake",
        "host": true
      },
      "zlib-ng" 
    ]
}

C:\dev\vcpkg\overlay-triplets\x64-windows-custom.cmake:
this is where all the processing happens. It is basically a copy of the original x64-windows triplet with an additional if() condition that runs all the code if the port name is zlib

set(VCPKG_TARGET_ARCHITECTURE x64)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE dynamic)

if(PORT STREQUAL "zlib")
    # ---- This code detects the port location. Checking CURRENT_PORT_DIR seems to be not safe (see discussion 31077) ----
    set(VCPKG_ENV_PASSTHROUGH VCPKG_OVERLAY_PORTS)
    set(overlay_ports $ENV{VCPKG_OVERLAY_PORTS})
    foreach(overlay_port IN LISTS overlay_ports)
        cmake_path(GET overlay_port FILENAME file_name)
        if (file_name STREQUAL "zlib")
            set(overlay_port_location ${overlay_port})
            break()
        endif()
    endforeach()
    if (NOT overlay_port_location)
        message(FATAL_ERROR "Unable to identify overlay port location!")
    endif()    
    # --------

    # Replace our custom empty portfile with a copy of the original zlib-ng
    file(COPY "${VCPKG_ROOT_DIR}/ports/zlib-ng/portfile.cmake" DESTINATION ${overlay_port_location})

    ## Alternative implementation through a symlink (instead of a file copy)
    # if(IS_SYMLINK "${overlay_port_location}/portfile.cmake")
    #     message("-- Skip creating symlink (IS_SYMLINK is true)...")
    # else()
    #     message("-- Creating symlink (${VCPKG_ROOT_DIR}/ports/zlib-ng/portfile.cmake -> ${overlay_port_location}/portfile.cmake)...")
    #     file(CREATE_LINK "${VCPKG_ROOT_DIR}/ports/zlib-ng/portfile.cmake" "${overlay_port_location}/portfile.cmake") # better a symbolic link?
    # endif()

    # finally configure cmake for compat build
    set(VCPKG_CMAKE_CONFIGURE_OPTIONS -DZLIB_COMPAT=ON)
endif()

Usage

Install a package that depends on zlib as usual:
vcpkg install freetype

Building zlib[core]:x64-windows-custom...
-- [OVERLAY] Loading triplet configuration from: C:\dev\vcpkg\overlay-triplets\x64-windows-custom.cmake
-- Installing port from location: C:\dev\vcpkg\overlay-ports\zlib
-- Using cached zlib-ng-zlib-ng-2.0.6.tar.gz.
-- Cleaning sources at C:/dev/vcpkg/buildtrees/zlib/src/2.0.6-f38bf28750.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source C:/dev/vcpkg/downloads/zlib-ng-zlib-ng-2.0.6.tar.gz
-- Using source at C:/dev/vcpkg/buildtrees/zlib/src/2.0.6-f38bf28750.clean
-- Found external ninja('1.11.0').
-- Configuring x64-windows-custom
-- Building x64-windows-custom-dbg
-- Building x64-windows-custom-rel
-- Fixing pkgconfig file: C:/dev/vcpkg/packages/zlib_x64-windows-custom/lib/pkgconfig/zlib.pc
-- Using cached msys-mingw-w64-i686-pkg-config-0.29.2-3-any.pkg.tar.zst.
-- Using cached msys-mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst.
-- Using msys root at C:/dev/vcpkg/downloads/tools/msys2/9a1ec3f33446b195
-- Fixing pkgconfig file: C:/dev/vcpkg/packages/zlib_x64-windows-custom/debug/lib/pkgconfig/zlib.pc
-- Installing: C:/dev/vcpkg/packages/zlib_x64-windows-custom/share/zlib/copyright
-- Performing post-build validation
Stored binaries in 1 destinations.

Considerations and concerns

1. As @dg0yt kindly made pointed me out, vcpkg track the input of a given port through ABI hashing and artifacts can share the same cache key. I'm not sure if this may, or may not generate side effects. During trials I experienced several times changes to code ignored, but I saw no impact on functioning. I will wait for suggestions from experts because I have not the required knowledge to handle such low level stuff.

2. In x64-windows-custom.cmake there is a commented section of code which shows an alternative implementation instead of a file copy. It uses a symbolic link. I'm not sure on what would be the better solution.

3. I would have preferred to avoid a custom overlay, but I didn't found a solution. A custom triplet is where I can change build configuration options and copy the original port file into the overlay folder (and get my package automatically updated without manually touching the source). A symbolic link could work, but its creation would have required running a pre-configuration external script while my main goal was to keep everything inside vcpkg chain.

4. The main purpose is to replace each old zlib dependency with the new zlib-ng. If a project needs to reference some special zlib version compiled outside the vcpkg ecosystem, then special handling is needed (just like any other external library that needs to coexist with an existing vcpkg package that provides its own vcpkg compiled version).

5. An empty portfile.cmake for the overlay port seems needed in order to avoid error. If there is some policy (applicable within the custom triplet) to ignore such error, then this useless file could be avoided.

[autoantwort]

Why this strange triplet magic? Why is an overlay port for zlib that depends on zlib-ng not enough?

[netcorefan1]

Why this strange triplet magic? Why is an overlay port for zlib that depends on zlib-ng not enough?

See point 3 of "Considerations" section. I added more details. I could have avoided the custom triplet by making an exact copy of the original portfile.cmake with ZLIB_COMPAT=ON flag added to the configuration options, but...
...how to handle updates? Each time zlib-ng get updated, I will have to manually do the same copy-paste-edit operation. The custom triplet is to make this happen automatically.

[dg0yt]

This tutorial leads into the same problems that were described in #31077 (reply in thread).
The triplet file shouldn't use anything which isn't already part of ABI hashing. This means you cannot use filepaths, no matter how you get them.
Keep the triplet file simple and stable.

[netcorefan1]

Thanks for your comment. I was hoping this was not the case.
As long as the port name which is requested to install is really zlib (and not something different), I can be sure that the extraction code returns a valid filepath (overlay_port_location), but I suppose this does not matter.
Problem is here:

# Replace our custom empty portfile with a copy of the original zlib-ng
file(COPY "${VCPKG_ROOT_DIR}/ports/zlib-ng/portfile.cmake" DESTINATION ${overlay_port_location})

I need a way to dynamically create the portfile before the overlay triplet ends execution. If I have understood well, doing this operation inside if(PORT STREQUAL "zlib") is already late because the build system has already performed ABI hashing, right? What if I do this outside the if() in something like a singleton style initialization?

See my response to @autoantwort . If I have to manually replace the portfile each time zlib-ng get updated, then it defeats the entire purpose of the tutorial. I really can't believe that a simple file copy operation can't be made part of the build process. Even a symbolic link solution would be fine (see commented code just below the file copy) because the portfile is just a clone of the original one and Cmake options are configured separately by the overlay triplet through VCPKG_CMAKE_CONFIGURE_OPTIONS. I just favored the file copy because it leaves room for editing (in the remote case that would be needed in the future).

[dg0yt]

Make the copy before calling vcpkg if you want to automate. But zlib-ng isn't updated everyday, and it might be a good idea to test an update before rolling out. Remember, vcpkg CI doesn't test your overlay configuration.

[netcorefan1]

This would require running a pre-configuration external script while my main goal was to keep everything inside vcpkg chain. What's wrong with moving initialization outside the if() condition?

if (not IS_SYMLINK portfilepath)
    file(CREATE_LINK "${VCPKG_ROOT_DIR}/ports/zlib-ng/portfile.cmake" "${portfilepath}/portfile.cmake")
endif()
if(PORT STREQUAL "zlib")
    # configure cmake for compat build
    set(VCPKG_CMAKE_CONFIGURE_OPTIONS -DZLIB_COMPAT=ON)
endif()

With the above, even in a fresh vcpkg install, even if zlib is the first package going to be installed, my initialization code will still run before zlib port ABI hash is determined:

Installing 1/3 vcpkg-cmake... # symbolic link created here
Installing 2/3 zlib-ng...
# ABI hash is determined before installing the port, but since we placed our portfile in step 1/3, I suppose it should be safe
Installing 3/3 zlib... 

Am I missing something obvious?

As a last resort, moving initialization outside vcpkg is still better than nothing, but I hope I can at least use a symbolic link and leave to vcpkg all the rest. Yes, zlib-ng isn't updated everyday, but since all I have to do is adding a flag, I would say that chances of issues are pretty much absent and it would be easier a "fire and forget".

[netcorefan1]

VCPKG toolchain integration issues

-- Setting up FREETYPE
-- Could NOT find ZLIB (missing: ZLIB_LIBRARY) (found version "1.2.11")
CMake Warning at C:/dev/vcpkg/scripts/buildsystems/vcpkg.cmake:852 (_find_package):
  Found package configuration file:

    C:/dev/vcpkg/installed/x64-windows-custom/share/freetype/freetype-config.cmake

  but it set FREETYPE_FOUND to FALSE so package "FREETYPE" is considered to
  be NOT FOUND.  Reason given by package:

  FREETYPE could not be found because dependency ZLIB could not be found.

Call Stack (most recent call first):
  CMakeLists.txt:114 (find_package)

I had to manually set FREETYPE_DIR to get the above log and discover that zlib was the source of the issue. I'm afraid this happens because my overlay port depends on zlib-ng which doesn't provide any CMake targets. When I switched to the old zlib (which provides targets), configuration went smooth.
What should I do? Should I open an issue of type "Request a feature or improvement to a port"? Or try to fix myself ? For a regular port, with the help of samples and docs I should be able to sort this out, but this is an overlay port and I learned well that overlays are hell!

[dg0yt]

I learned well that overlays are hell!

No. They just need the same care (from you!) which maintainers put in supported ports. There is no "fire and forget".
Premature shortcuts are hell.

What should I do?

You must solve this sourself. Your overlay port and custom triplet is not a supported configuration. If you think that there is a problem in vcpkg, try to create a minimal reproducible example. Sometimes this is enough to spot an error.

And IMO please remove the tutorial. It doesn't provided a reliable and tested user experience.

[netcorefan1]

No. They just need the same care (from you!) which maintainers put in supported ports. There is no "fire and forget".

Then my efforts and care must have been not enough if after so much time I still didn't found a way to achieve the goal (at least not in the way I was hoping). "fire and forget" may sound like a bad thing at first, but it isn't, at least not in my intentions. It is an attempt to create a maintenance free port (which is not mandatory or something that make the project renounceable).

Premature shortcuts are hell.

I assume this is a way to say "Yes, a shortcut while Installing 1/3 vcpkg-cmake... is safe under ABI hashing point of view, but it is premature because you're going to run a section of code each time you install a package even when does not need to be installed." If so, then it is perfectly fine for me. This is a local thing and I don't need and expect the same perfection and accuracy that would be needed like if it was an official vcpkg feature.

You must solve this sourself. Your overlay port and custom triplet is not a supported configuration. If you think that there is a problem in vcpkg, try to create a minimal reproducible example. Sometimes this is enough to spot an error.

I'm not sure if this can be considered a problem with vcpkg and strictly related to my overlay. It seems to have something to do with this already reported issue. I have done some adjustments and seems to work, but I'm holding back from sending a pull request because I don't get ... is compatible with built-in CMake targets in the log and I'm trying to find out what are the requirements that allows to pass such check.

It doesn't provided a reliable and tested user experience.

I totally agree with you and I have specified in multiple points of the description that this was not production ready, but a work in progress that needed user contributions. If you was expecting a reliable and tested user experience, than I must have been not clear enough. I've just changed the title with a more appropriate description. This time people will be aware that this is a work in progress that need users contributions even before they open the url.

And IMO please remove the tutorial.

I must admit that I was not sure if it was a good idea to write it. It would have been easier to just make my local changes privately and then just ask for help if and when would have occurred issues not easily fixable by me. However, in the spirit of sharing, I went ahead because I imagined that would have been of benefit for the community along with the possibility to make something better with the help of people. Plus, as an additional bonus, this would have been a chance for me to learn more.
Let see how will evolve first. I'm sorry if this created some frustration and I can perfectly understand you.

[dg0yt]

AFAIU you tried to automate what you didn't completely resolve manually before.

is compatible with built-in CMake targets will only be printed if a port installs a usage file. This is provided by the maintainer.
(There is also a heuristics for ports which install CMake config, but such targets aren't built-in.)

Port zlib also comes with another important addition, vcpkg-cmake-wrapper.cmake. The vcpkg toolchain uses this script to wrap find_package(ZLIB) calls because CMake's built-in Find module doesn't correctly deal with the debug and release variants for vcpkg.

As I said, try to create minimal reproducible examples. e.g. a CMake project which does find_package(ZLIB) required. You can add set(CMAKE_FIND_DEBUG_MODE 1) to get more diagnostic output if you need to trace an error.

[netcorefan1]

AFAIU you tried to automate what you didn't completely resolve manually before.

I think you're right. I have done the mistake to think that the toolchain would have worked straight away.

is compatible with built-in CMake targets will only be printed if a port installs a usage file. This is provided by the maintainer. (There is also a heuristics for ports which install CMake config, but such targets aren't built-in.)

Port zlib also comes with another important addition, vcpkg-cmake-wrapper.cmake. The vcpkg toolchain uses this script to wrap find_package(ZLIB) calls because CMake's built-in Find module doesn't correctly deal with the debug and release variants for vcpkg.

As I said, try to create minimal reproducible examples. e.g. a CMake project which does find_package(ZLIB) required. You can add set(CMAKE_FIND_DEBUG_MODE 1) to get more diagnostic output if you need to trace an error.

Many thanks for the detailed explanation. This helped me to prepare a reproducible example. This time I have completely excluded the automation part. The attached zipped file is a complete overlay port alone that can run without any configurations:
vcpkg install zlib --overlay-ports zlib_overlay_extracted_path
Basically I added the same files and lines of code from the regular zlib with no changes (files and header names are the same).
I don't know what is the way to test a toolchain, but I tried to compile again emgucv which was previously giving me such problem and it worked included is compatible with built-in CMake targets (yes, I am using the regular zlib for trials related to the issue opened here, I just used my custom port once to see if it was working).

zlib_overlay.zip
Let me know what you think, thanks!