I ended up just using the git tag directly and disable GitVersion when the build is triggered from a tag with if: startsWith(github.ref, ‘refs/tags/’) != true. I perform the stable/unstable versioning logic with actions/github-script as follows:

const gemVersion = (function() {
  const ref = '${{ github.ref }}';
  const tagPrefix = 'refs/tags/';
if (ref.startsWith(tagPrefix)) {

// If a tag ref is being built, just return the tag verbatim

return ref.substring(tagPrefix.length);

}
const escapedBranchName = '${{ steps.gitversion.outputs.escapedBranchName }}';

const commitsSinceVersionSource = '${{ steps.gitversion.outputs.commitsSinceVersionSourcePadded }}';

const preReleaseLabel = escapedBranchName + '.' + commitsSinceVersionSource;

const majorMinorPatch = '${{ steps.gitversion.outputs.majorMinorPatch }}';

return majorMinorPatch + preReleaseLabel;

})();
core.setOutput('version', gemVersion);

It would be much easier if I could just do git fetch --tags in the build to have the tag being built locally, but as that seems impossible, this is at least a suitable workaround.