Self hosted runner autoscaling

Code to Cloud GitHub Actions
#1

Backstory, i’m looking to move from Buildkite where we use https://github.com/buildkite/elastic-ci-stack-for-aws

The github actions local runner, seems to be a toy in its current state.  Its missing the following:

  1. A way to set up many runner in an automated way
  2. A local events system, so we can protect an instance from being scaled in while its doing work
  3. A way to tell the worker to shutdown after its done with its current job
  4. A api that gives you the # of outstanding actions to run so you can make scaling decisions
24 Likes
#2

I’m also looking for the same thing. @Github actions team, do you have any information about this?

1 Like
#3

Yes, we’re working on an API that provides the registration token so that people will be able to automate setup.

7 Likes
#4

This is really needed, especially for parallel tests environment. Also, project level OR organization level runners that are usable across repositories.

1 Like
#5

Looks like #1 this is now supported, but #2, #3 and #4 are not…

https://github.blog/2020-02-06-manage-secrets-and-more-with-the-github-actions-api/#self-hosted-runners

#6

I’m running into this as well – I can programatically register self hosted runners and query for pending or in progress runs, but the current APIs for workflows and workflow runs does not show which ones run on self-hosted runners. You can query the API for queued and in progress runs (to determine how many runners you should scale up to), but non-self-hosted workflows can not be filtered out to get an accurate number of required self-hosted runners (#4 from above).

#3 appears to be solved by using run.sh --once.

#7

Hi @jeichorn 

Could you please tell a bit more why you want to move off buildkite?

I’m currently selecting between buildkite and github actions as our next CI tool and I’d appreciate if you can share your experience.

#8

@jeichorn Wondering about this as well.

#9

@alexebirdand  @vgrigoruk, might be too late to help you but the answer is:

Cost and Integration.

I have a team of 30 people in github.  Giving them access to the buildkite UI is $15 a month per user, github is $4 a month a user.  Buildkite is another account to maintain, another UI to learn etc.

1 Like
#10

Niek wrote a blog-post on auto-scaling github self-hosted action runners on AWS.

It’s open-sourced.

3 Likes
#11

Thanks @jeroenknoops I hadn’t seen that work by Niek, it looks like he worked around the limitations of runners to make autoscaling possible :grinning:

#12

We’re using it now for hundreds of repositories. It works very nice.

1 Like
#13

does anyone run auto-scaling in GCP?

#14

For GCP, here is a blog post about it: https://github.blog/2020-08-04-github-actions-self-hosted-runners-on-google-cloud/

#15

I tried but couldn’t get it working, having problems with Github permissions, created an issue on the repo, please could you check.

Thanks.

#16

Hey, there:

Just wondering how do you try to remove all the runners? I didn’t any API for this. For the API of force delete the runner from org doesn’t work.

#17

We’re now trying to terminate runners by using the AWS API. If a runner is working, this will prevent the runner from getting removed. We’re talking with GitHub to get this with the API.

#18

how does that work? If we terminate the instance directly, how do you do that?

  • Confirm the runner is running a job
  • prevent the termination

Looking forward to your reply. Thanks.