I don’t really expect help with this given how nebulous the problem is, but I figured I’d put it out there just in case since this will likely take a problematically long time to figure out.

For one of my projects I use actions on GitHub hosted runners to handle CI/CD. This project makes use of Doxygen, which does not come by default with the Windows Server 2022 image.

So, with that in mind, here is a simplified version of the job that performs the main build for this project

jobs:
  build-qx:
    name: Build Qx - Windows (Debug/Release)
    strategy:
      matrix:
        qt_linkage: [shared, static]
    runs-on: windows-latest
    env:
      vs_dir: C:/Program Files/Microsoft Visual Studio/2022/Enterprise
      cmake_gen: Ninja Multi-Config
    outputs:
      qt_shared_artifact_name: ${{ steps.get_artifact_name.outputs.qt_shared_artifact_name }}
      qt_static_artifact_name: ${{ steps.get_artifact_name.outputs.qt_static_artifact_name }}
    steps:
    (...)
    - name: Cache Qt Build
      id: cache-qt
      uses: actions/cache@v3
      with:
        path: ${{ env.qt_install_dir }}
        key: ${{ env.qt_install_name }}
    (...)
    - name: Install Doxygen & Graphviz
      run: |
        choco install doxygen.install --version 1.9.4
        choco install graphviz
    - name: Checkout Qx
      uses: actions/checkout@v3
      with:
        path: ${{ env.qx_src_suffix }}
        fetch-depth: 0 # Required for verbose versioning to work correctly
    - name: Build/Install Qx
      working-directory: ${{ env.qx_src_dir }}
      shell: cmd
      run: |
        echo "Setup C++ Build Environment..."
        CALL "${{ env.vs_dir }}\Common7\Tools\VsDevCmd.bat" -arch=amd64
        echo "Configure CMake using Qt wrapper..."
        CALL "${{ env.qt_cmake }}" -G "${{ env.cmake_gen }}" -S "${{ env.qx_src_dir}}" -B "${{ env.qx_build_dir }}" -D QX_DOCS_TARGET=ON
        echo "Changing to build directory..."
        cd "%qx_build_dir%"
        echo "Building Qx debug..."
        cmake --build . --target all --config Debug
        echo "Building Qx release..."
        cmake --build . --target all --config Release
        echo "Building Qx docs..."
        cmake --build . --target qx_docs --config Release
        echo "Installing Qx Debug"
        cmake --build . --target install --config Debug
        echo "Installing Qx Release/Docs..."
        cmake --build . --target install --config Release
        echo "Packaging Qx..."
        cpack -C "Debug;Release"
        echo "Build complete."
    (...)

This was working fine. What started happening is that, at first intermittently, documentation would fail to build on the Windows Server 2022 hosts. Checking the logs, when CMake ran I see that instead of the usual:
Found Doxygen: C:/Program Files/doxygen/bin/doxygen.exe (found version "1.9.4...

I see:
Found Doxygen: C:/ProgramData/chocolatey/bin/doxygen.exe (found version "1.9.1...

and then the build later fails because of Doxygen features I use that were added after 1.9.1.

I first assumed that something must have happened with the chocolatey package, but it hasn’t been touched and the logs for that Doxygen installation step are nearly identical (other than run-to-run variance) between the runs that see the right version vs. the wrong version. It shows the setup executable for 1.9.4 being ran in both cases. So all-in-all I’m not sure what’s up with the path change.

Needless to say I was thoroughly confused, but decided to focus on the fact that at first this problem started happening sporadically, yet as of now happens every time. I even have a single run where the first path of the Windows build matrix found the correct Doxygen version and the second path found the wrong Doxygen version.

After some cross-referencing, I was able to figure out a consistent demarcation point:

It seems that these mixed success runs happened to occur during the rollout of the Windows Server 2022 20220619.1 image, as some of the runs were executed on a host with 20220614.0, while others were on a host with 20220619.1 (and very recently a few with 20220626.1).

100% of the runs where CMake found the correct version of Doxygen (1.9.4) occurred on a runner with 20220614.0, while 100% of the runs where the wrong version of Doxygen (1.9.1) that seemingly pops into existence out of nowhere is found occurred on a runner with 20220619.1 or 20220626.1.

My workflow has not been altered at all during any of this, especially evident by the fact this correlation of Windows Server 2022 image version was observed during a single matrix run as noted earlier.

Figuring that some related tool must have been changed with the 20220619.1 update, I went over to compare the two ReadMe files and found… nothing.

  <a href="https://github.com/actions/virtual-environments/commit/96c2ea89565b32d23595ce34732dc7539d67166e" target="_blank" rel="noopener nofollow ugc">github.com/actions/virtual-environments</a>

<div class="github-info">
  <div class="date">
    committed <span class="discourse-local-date" data-format="ll" data-date="2022-06-23" data-time="14:37:16" data-timezone="UTC">02:37PM - 23 Jun 22 UTC</span>
  </div>

  <div class="user">
    <a href="https://github.com/apps/github-actions" target="_blank" rel="noopener nofollow ugc">
      <img alt="github-actions[bot]" src="https://user-images.githubusercontent.com/24661585/181096619-22ed8fc9-0783-4235-8744-449a15826ee0.png" class="onebox-avatar-inline" width="20" height="20">
      github-actions[bot]
    </a>
  </div>

  <div class="lines" title="changed 1 files with 18 additions and 16 deletions">
    <a href="https://github.com/actions/virtual-environments/commit/96c2ea89565b32d23595ce34732dc7539d67166e" target="_blank" rel="noopener nofollow ugc">
      <span class="added">+18</span>
      <span class="removed">-16</span>
    </a>
  </div>
</div>

Chocolatey wasn’t updated, CMake wasn’t updated, Powershell wasn’t updated, even the build of Windows Server 2022 wasn’t updated.

The only things that were updated that are even within the same universe of relevance were:

• vcpkg
• Microsoft.VisualStudio.Component.Windows11SDK.22621

But vcpkg should have no bearing on any of this unless its somehow used indirectly by something else, and I really doubt that a new Windows SDK version would cause such an issue (I’d hope), and I don’t even think it’s used by default.

I changed my find_package(Doxygen...) call so that it specifically requires 1.9.4 in hopes that for whatever bizarre reason two version of Doxygen were being installed and that I could at least just ignore the wrong one, but unfortunately that didn’t work. It seems that after the image update, only 1.9.1 gets installed as I now get:
Could NOT find Doxygen: Found unsuitable version "1.9.1", but required is at least "1.9.4" (found C:/ProgramData/chocolatey/bin/doxygen.exe...

I’m fairly lost here. This basically just doesn’t make sense.

I don’t supposed anyone has a clue as to what’s going on.