Too many workflows are referenced, limit is 20

I am running a complex workflow that requires combining different reusable workflows and I am hitting this limit too many workflows are referenced, limit is 20.

Is there any plan to increase this limit or workaround for it?

Some alternatives (that do not really work for me but might work for other people):

  • SUPPORTED: Composite actions: I am using them already as part of the reusable workflows
  • NOT SUPPORTED: A good alternative would be to call a reusable workflow within a reusable workflow, which is something that “by the doc” cannot be done.
  • HACK: Another potential solution is using a matrix strategy like the one outlined here; this does not work for me because I cannot chain workflow dependencies in the way I want. (A bit more background: let’s say I have to run workflow A with 7 variants and workflow B with 9 variants, and the dependencies between As and Bs are not 1-1, and I want to avoid waiting for all A jobs to finish before starting B workflows).

Great question. Since we need to ingest every called workflow, we added a 20 workflow limit to ensure that we didn’t expose the runner to denial-of-service attacks.

We can look at increasing this number - how many workflows are you trying to reference?

Thanks @ethomson !

The objective for us is to run workflows (or set of workflows) that would require approximately 100-400 jobs and include long build chains (depth 5-10, around 50 jobs per “independent” set of jobs in the chain). I reckon this can be solved with some composition of workflow_dispatch and downloading artifacts from the previous workflow (in the chain) using GitHub - dawidd6/action-download-artifact: A GitHub Action to download an artifact associated with given workflow and commit or other criteria. I indeed don’t want to put everything in one single workflow, but using reusable workflows has the advantage of having the whole build chain shown in one view in the workflow run page, and allows us to define dependencies between jobs in an easy way.

Again, the point is that I cannot use a matrix within the called workflow, because I cannot make jobs in the caller workflow depend on one single matrix configuration, resulting in me having to wait all the jobs defined in the matrix to be completed before e.g. running tests.

For now it should be sufficient to increase the limit to 30-40, though would it be possible to increase this limit in the future if needed? If so, how much approximately? 50-100? Even more?

Since we are running many of our workflows on self-hosted runners, would it make sense to let people configure this parameter in the actions settings page (e.g. at the repo or org level)? Does the way reusable workflows are implemented not allow this?

Thank you very much, glad do add some examples if the use case is not clear enough

1 Like

+1. We’re looking at over 50 “calls” to reusable workflows.

1 Like

Also having the same issue. I just need 30 in my case.

Using a monorepo setup with 7 microservices. Created a build and a deploy reusable workflow just to discover this limit when I enabled the staging environment. Really a bummer after spending all the energy battling yaml and all the env variables and secrets prop drilling.

Is the limit increase something we can request per account basis, or is something that github team is looking at?

+1.

From a selfish perspective, I am looking at 25-30 calls to reusable workflows.

That would solve my immediate problem but I’d also echo the comment above about mono-repos - that approach is common and could easily result in a lot of calls to reusable workflows (maybe 50 or more).

Thanks.

@ptc-rcapraro @mhirota-impinj @gusfune @cipriancaba @CallumHibbert

Can you help me understand whether you need to call the same reusable workflow 20+ times, or whether you need to call 20+ distinct reusable workflow files?

In my case there are 3 or 4 distinct reusable workflow files that I would like to call 20+ times.

1 Like

@ericsciple In my case, it’s more like calling 4 distinct reusable workflows 10 times each

1 Like

Same workflow, multiple times. There’s a point where splitting the caller workflow and using worfklow_run to create the chain might make more sense (based on the dependencies between jobs).

1 Like

Thanks folks for the discussion! Work is in progress to change this to a distinct count

@ajaykn fyi

2 Likes

Thanks everyone for the patience.
We have rolled out a fix to count per distinct workflow.
This should fix most of the cases mentioned in the thread like referring same workflow multiple times.

Please reach out to us, if you are having the scenario referring more than 20 distinct workflows.
We can prioritise accordingly. cc @ericsciple

3 Likes

This has solved my scenario, thanks very much.

1 Like