Actions/cache - Fan in from matrix jobs #26665
-
I’m trying to build a Workflow to manage a relatively intensive build job. Specifically, I need to shard the build into manageable pieces, and then run the sharded build on self-hosted runners for memory capacity. Once builds are complete, I need to pull them into a merge job and continue with processing. In a CI space, this should be a fan-out/fan-in pattern, and while those are generally tricky, I’m having particular trouble setting this up in Github Actions. The shards are straightforward: I have a setup job that produces a matrix JSON, and then the individual builds are run in a strategy. I’ve been trying to set up I have two problems:
For debug logs related to #1, in the build job I’m seeing:
and then in the merge job
These are jobs running in the same workflow, so issues like cross-branch builds shouldn’t be an issue. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
The cache is not meant for reliable transfer, it’s meant to avoid duplicate work on a best-effort basis. So I’d advise against this approach even if it happened to be working right now. 😅 What you should do instead is to use artifacts, see Storing workflow data as artifacts - GitHub Docs. Artifacts are meant exactly for the kind of scenario you describe, where data needs to be passed between parts of a build (or out of it). If your artifacts are large and you don’t want to keep using the storage, you can set a short retention period when uploading the artifact. |
Beta Was this translation helpful? Give feedback.
The cache is not meant for reliable transfer, it’s meant to avoid duplicate work on a best-effort basis. So I’d advise against this approach even if it happened to be working right now. 😅
What you should do instead is to use artifacts, see Storing workflow data as artifacts - GitHub Docs. Artifacts are meant exactly for the kind of scenario you describe, where data needs to be passed between parts of a build (or out of it).
If your artifacts are large and you don’t want to keep using the storage, you can set a short retention period when uploading the artifact.