After talking with GitHub support this was the solution I was given that worked for me:

Hello Eduardo,

I heard back from engineering and the reason that the alerts are open is because they are open in a previous stale configuration that is no longer updated. This happened when the analysis file was renamed and left the analyses under the old name stale.

This can be detected through the API, by using the list analyses request (Code scanning - GitHub Docs). From that response you can see multiple analyses that has “deletable”: true and with different values in the category field.

To fix it you should delete all the stale configurations using the delete API (Code scanning - GitHub Docs). Note that if a particular configuration has had several deliveries, then older analyses become deletable when the most recent one is deleted (the delete works as an undo operation) and they also need to be deleted."

This problem happened to me because I had ran an initial CodeQL scan and then renamed the workflow for CodeQL.

Hope this helps others.