Self-hosted runner queued forever

Suddenly, all the self hosted runner were queued forever.

Contact the support and no any feedback for more than 1 day now.

I created a brand new vm, it works for several run and then not any more.

How do I debug self-hosted runner?

2 Likes

BTW, Is there a solution to run the action locally? not tied to github’s service. It is really a good product, except when it is not working.

All of sudden we’re starting to have a lot of issues like this too using self-hosted runners. Things have been working fine for months. Now some random jobs just hang on “Starting your workflow run”…

2 Likes

Thanks for your feedback. I am not alone. Github seems to pretend the problem does not exists and did not respond to my support ticket for several days.

@swuecho, @rubenfonseca,

Can you check if the self-hosted runners you want to use are ‘Idle’ status when you trigger the workflows to run on the self-hosted runners?

When a runner is in use, its status is ‘Active’.
If the runners are ‘Offline’, try to restart them on the hosted machines.

To view more details, you can reference here:
https://docs.github.com/en/actions/hosting-your-own-runners/monitoring-and-troubleshooting-self-hosted-runners

Any update, feel free to tell us.

tried to stop and restart several times. sometimes restart take a long time.

after restart, status was ‘idle’. with new push, the runner will be active. However, in the runner, the test is not running ok.

in _diag/Work-xxx.log

[2020-07-23 02:54:44Z INFO Worker] Waiting to receive the job message from the channel.
[2020-07-23 02:55:15Z ERR  Worker] System.OperationCanceledException: The operation was canceled.
   at GitHub.Runner.Common.Extensions.WithCancellation[T](Task`1 task, CancellationToken cancellationToken)
   at GitHub.Runner.Common.StreamString.ReadInt32Async(CancellationToken cancellationToken)
   at GitHub.Runner.Common.ProcessChannel.ReceiveAsync(CancellationToken cancellationToken)
   at GitHub.Runner.Worker.Worker.RunAsync(String pipeIn, String pipeOut)
   at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)

I tried to remove the runner and reconfig, still the same problem. The problem appear about 2 or 3 days ago.

In Runner_xxx.log


[2020-07-23 03:50:05Z WARN GitHubActionsService] Authentication failed with status code 401.
WWW-Authenticate: Bearer^M
Strict-Transport-Security: max-age=2592000^M
X-TFS-ProcessId: 511c643b-787c-4920-8f42-7030245e8883^M
ActivityId: 027a165a-0130-4d77-9488-91056885edec^M
X-TFS-Session: 5d296720-964b-4af3-9329-ba50c5c10d75^M
X-VSS-E2EID: 32b1f9ed-4237-4660-adfd-639ee918504a^M
X-VSS-SenderDeploymentId: 193695a0-0dcd-ade4-f810-b10ad24a9829^M
X-TFS-ServiceError: The+user+%27System%3aPublicAccess%3baaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa%27+is+not+authorized+to+access+this+resource.^M
X-VSS-S2STargetService: 0000005A-0000-8888-8000-000000000000/visualstudio.com^M
X-MSEdge-Ref: Ref A: 742070CF566E4F50B71E984A6B7F4FC8 Ref B: TYBEDGE0318 Ref C: 2020-07-23T03:50:05Z^M
Date: Thu, 23 Jul 2020 03:50:05 GMT^M

Might be the Auth problem? but I do not think I change any auth related setting and I tried to re-run the old successful ones. not work either.

@swuecho, @rubenfonseca,

I have created an internal issue ticket to help you report the problems to the appropriate engineering team for further investigation and evaluation.
If they have any progress, I will notify you in time, and sometimes the appropriate engineers may directly reply you here.

1 Like

for those who are interested in a self managed solution.

  1. act https://github.com/nektos/act

with act, you can run github action locally.

  1. gocd https://www.gocd.org/

gocd by default does not support branch test model, you have to install github-pr-poller and github-pr-status plugins. With these two plugin support, it is as good as github action if not better.

Sounds like a serious dilemma I’m new here and do not posses the skill set to offer any solution I’m going to keep follow g and lead I g thanx

thanks diamond art

Having the exact same problem.

Everything was working on friday.

Today it started having this issue.

[2020-08-24 10:12:39Z WARN GitHubActionsService] Authentication failed with status code 401.
WWW-Authenticate: Bearer
Strict-Transport-Security: max-age=2592000
X-TFS-ProcessId: 958e827d-3664-47e0-a387-ec73e2fa8e41
ActivityId: e6f384e8-77f1-48e1-ae52-95b94a932856
X-TFS-Session: 154da78c-ab65-434c-9468-044f135b0bf4
X-VSS-E2EID: 4275a647-a0df-44bd-b96d-162c32377a6d
X-VSS-SenderDeploymentId: a07ab14e-025a-39c3-8d53-788cd7ce207f
X-TFS-ServiceError: The+user+%27System%3aPublicAccess%3baaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa%27+is+not+authorized+to+access+this+resource.
X-VSS-S2STargetService: 0000005A-0000-8888-8000-000000000000/visualstudio.com
X-MSEdge-Ref: Ref A: 21345C36025B49E3B5A679618A496835 Ref B: FRAEDGE1518 Ref C: 2020-08-24T10:12:39Z
Date: Mon, 24 Aug 2020 10:12:39 GMT

any help with this?