I had the same issue. Based on the browser logs I figured out that it had to do with it wanting to sign in using SSO for an organization that has nothing to do with the repository in question but fails to open the dialog:
For example, this if from FireFox:
An iframe which has both allow-scripts and allow-same-origin for its sandbox attribute can remove its sandboxing. actions
Content Security Policy: Ignoring ‘x-frame-options’ because of ‘frame-ancestors’ directive.
Content Security Policy: The page’s settings blocked the loading of a resource at https://github.com/PerMalmberg/Smooth/actions (“frame-ancestors”).
And this from Chrome:
Refused to display 'https://github.com/login/oauth/authorize?response_type=code&redirect_uri=https%3A%2F%2Flaunch-editor.github.com%2Fauth%2Fcallback&state=<code>&client_id=<another code>' in a frame because an ancestor violates the following Content Security Policy directive: "frame-ancestors 'none'".
I got the request to sign in using SSO for the same organisation when signing in to this forum too. (what if I’m part of several orgs with forced SSO?!)
Now that I am signed in with SSO, I can access my Github Actions page again.
Clearly forced SSO from an organization has too wide reach, in addition to the failure to open the dialog.