GitHub Community
Is that possible to run job which takes more than 6 hours on self-hosted runner? #26679
-
|
In my CI/CD setup, one step is to retrain a deep learning model. We setup our own GPU-box as a self-hosted runner to run this step. However, the training takes about one day, which is much longer than the current 6-hours cap GitHub Actions allowed. Is there any way to workaround this limitation? |
Beta Was this translation helpful? Give feedback.
Replies: 14 comments 2 replies
-
|
Yes, you can use self-hosted runner to run more than 6 hours job.
Please refer to this official blog: |
Beta Was this translation helpful? Give feedback.
-
|
This is interesting. My test shows the jos is killed after 6 hours even running on self-hosted runner. Here is a snapshot of the summary of the run. As it shows, the job was killed after 6 hours. Just in case one might suspect the job had not been run on self-hosted runner. Here is the screenshot of the shell where I run the self-hosted runner agent. As can be seen from the timestamps, the job (the 3rd and 4th ones) were killed after 6 hours. |
Beta Was this translation helpful? Give feedback.
-
|
The maximum number of minutes to let a job run before GitHub automatically cancels it. In default is 360 minutes. Please try to specify a larger minutes to timeout-minutes under your job. |
Beta Was this translation helpful? Give feedback.
-
|
I was very confused. It didn’t work for me. Even “Both jobs. <job_id> .steps.timeout-minutes” and “jobs. <job_id> .timeout-minutes” be set to 7200, The jobs was still canceled by GitHub automatically. My YAML file: https://github.com/Jianlin-lv/buildx-demo/blob/master/.github/workflows/build_container.yml https://github.com/Jianlin-lv/buildx-demo/actions/runs/105486327 |
Beta Was this translation helpful? Give feedback.
-
|
Solution is false. |
Beta Was this translation helpful? Give feedback.
-
|
+1 . Can some one please update if we can run any jobs more than 6hrs ? |
Beta Was this translation helpful? Give feedback.
-
|
Just to let all the folks know, Yes its possible to run job/steps for more than 6 hrs , provided you are using self hosted runners .for ex: Use job-timeout: 1000 parameter to allow jobs/steps to run for 1000 mins in your workflow. Cheers !! |
Beta Was this translation helpful? Give feedback.
-
|
Can someone help with this? This solution doesn’t work. I’m still getting a 6h timeout even after putting timeout-minutes in the step and job level with a self hosted runner. As aside note I’m using reusable workflows (reusable workflows can’t have timeout-minutes property) |
Beta Was this translation helpful? Give feedback.
-
|
Do we have any updates on the solution. Self hosted runners even with timeout parameter being more than 6 hours doesnt work. It still cancels the job after 6 hours. Any solutions here ? |
Beta Was this translation helpful? Give feedback.
-
|
I'm running into the same issue with a self-hosted runner. |
Beta Was this translation helpful? Give feedback.
-
|
Still running into this. Any movement on allowing longer than 360 min jobs on self hosted runners? We would like to run stress tests on custom hardware, which shouldn't affect extra runner infrastructure GitHub makes for Actions. |
Beta Was this translation helpful? Give feedback.
-
|
I can report that still the limit is in place and I'm unable to do a longer build beyond 6h. |
Beta Was this translation helpful? Give feedback.
-
|
I can report that still the limit is in place and I'm unable to do a longer build beyond 6h. |
Beta Was this translation helpful? Give feedback.
-
|
We were actually able to get longer than 6h builds on self-hosted runners recently with something like the following: The key seems to be set to it at the job level, not step level |
Beta Was this translation helpful? Give feedback.
-
|
@tt-rkim what is the maximum timeout value have you been able to test this for? One of the limits is 35 days, I wonder if this is able to get to that or cross that limit. |
Beta Was this translation helpful? Give feedback.
-
|
We have been able to get up to 25 hours, unfortunately no more than 35 days. I am unsure how to get around the total workflow time... unfortunately my suspicion is because the workflow is a GitHub-managed construct in their architecture or whatever, this doesn't sound like something you could get around. Or at least, I have no ideas currently on how to . |
Beta Was this translation helpful? Give feedback.
The maximum number of minutes to let a job run before GitHub automatically cancels it. In default is 360 minutes.
Please try to specify a larger minutes to timeout-minutes under your job.
https://help.github.com/en/actions/automating-your-workflow-with-github-actions/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes