Actions taking very long for Windows Release Build

We at gtsam are using Github Actions to build and run our project. While the Debug build for the Windows environment takes a reasonable 20-30 minutes, the Release build takes north of 3 hours which really slows down reviewing and code-merging.

Can anybody help with this? Is there something in CMake we can update to make the build faster?