Queued jobs not executing

Hi,

during the past week or so, jobs queued in one of our private repos after a push or PR are no longer being executed on our self-hosted linux runners. They get queued, and after a day or so of waiting they get cancelled with the error message This request was automatically failed because there were no enabled runners online to process the request for more than 1 days.. The whole setup worked before, and the macos jobs are still working.

The “Settings” tab shows several idle Linux runners. As I read the error message, the runners are recognized as being eligible, but for some reason the jobs are not assigned.

One of the runners is configured on the organization level. It receives jobs from two repositories, and the jobs from the other repositories do get executed on it, so the runner itself is still working.

I have tried restarting the runner, manually upgraded it to the the latest version, and even removed and re-registered it, without any effect.

Any ideas what’s wrong here?

Things are working again. The reason is not entirely clear, apparently one or more (but not all) runners were having problems with a full disk and an outdated version.

Nevertheless it would have been helpful if

  • there was any indication of the problem(s) either in the runner logs or on the github website
  • failure of one or more runners didn’t stall the entire project, i. e. jobs should have been queued on the working runners