GitHub Windows Server 2022 runners randomly selecting between image 20241113.3.0 and 20241021.1.0

[Alan-Jowett]

Description

The newer image of Server 2022 contains an updated version of the WDK (version 26100.0). I can either build my project to target the older WDK version or the newer one, not both. Having runners randomly select a WDK version makes it impossible to build reliably.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• macOS 15
• macOS 15 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

Working:
https://github.com/actions/runner-images/releases/tag/win22%2F20241021.1

Failing:
https://github.com/actions/runner-images/releases/tag/win22%2F20241113.3

Is it regression?

Yes - Works on 20241021.1.0, fails on 20241113.3.0

Expected behavior

Builds should consistently use either 26100 or 22621 of the WDK, not toggle between them

Actual behavior

Builds randomly pick one version or the other.

Repro steps

Have driver that targets 22621 of the WDK.
Build

Sometime succeeds and sometimes fails.

[binarymaster]

Having the same problem here, configure step fails:

    LINK: command "C:\PROGRA~1\MICROS~2\2022\ENTERP~1\VC\Tools\MSVC\1429~1.301\bin\HostX64\arm\link.exe /nologo CMakeFiles\cmTC_19d21.dir\testCCompiler.c.obj /out:cmTC_19d21.exe /implib:cmTC_19d21.lib /pdb:cmTC_19d21.pdb /version:0.0 /machine:ARM /debug /subsystem:console /MANIFEST:EMBED,ID=1" failed (exit code 1104) with the following output:
    LINK : fatal error LNK1104: cannot open file 'kernel32.lib'

    ninja: build stopped: subcommand failed.

• Failing one: https://github.com/reactos/reactos/actions/runs/11871236840/job/33083517488 (20241113.3.0)
• Working one: https://github.com/reactos/reactos/actions/runs/11871236840/job/33083516572 (20241113.1.0)

[Alan-Jowett]

Can I get this triaged ASAP? This is causing an outage for our project. Projects can't target both old and new WDK and by randomly switching between them it makes it impossible to build.

[BotellaA]

Save issue on our side. It breaks cmake cache files by providing random compiler path from one image to the other.

[gdryke]

Apologies for the issue here. We do a slow and gradual roll out of the new image versions, which can sometimes lead to version disconnects like this.

However, this persistent issue is not normal, and is because a few components failed to update to the new image late on Friday. We're going to get the last areas updated first thing tomorrow morning, and that should clear up this version flip flopping, going forward to 20241113.3.0.

[BotellaA]

Do you have any update or ETA for the stable image runner?

[gdryke]

We've had issues smoothing out the underlying pools, but we're tackling it with priority today.

[smoogipoo]

I hate to be a broken record but... any updates? We're still unable to build our net8.0-android project on the windows-latest runner...

[binarymaster]

Having the same problem here, configure step fails

I've been able to workaround the problem I reported here earlier by sticking to the previous Windows SDK 10.0.22621.0.

[Alan-Jowett]

I was able to work around the problem by pulling the Windows SDK and WDK via nuget instead of relying on the provided versions. This permits my project to consistently target 26100.

See: microsoft/ebpf-for-windows@0a14ab3

[gdryke]

Everyone should be consistently on the newer version of 20241113.3.0, or moving to an even newer version: 20241125.1.1, which should only have minor version changes, nothing major.

As mentioned before, apologies for the long span of time with flip flopping versions, that was related to an issue on our end. But as a reminder this can happen during image version updates, which we roll out gradually.

I'm going to close this issue out, but if anyone still has problems with version flip flopping, please reopen it!