I'm trying to figure out if there's a way to dynamically create workflow yamls from a script, and then immediatelly executing it. The use-case is being able to dynamically add or remove steps across the repo when things change, instead of running all steps on every commit.
For example; if I've got multiple test modules I want to *programatically* decide which ones are useful for the current commit-range, and run only those. Currently, I use a single script that folds each step with `::group::` and `::endgroup::`, but this makes it hard to find which steps are actually succeeding and which ones are not. I'd much rather have these as proper steps, or preferably jobs, to show success/failure at a higher level instead of having to dig through the logs to discover where things fail.
Has anyone tried to do something similar?
Take a look here and here. In both cases logic step ("ReDefine") is running our application which creates single log file. Depends what's found in there, only interesting steps are enabled. First link was a boring commit, so nearly all steps are disabled. In second one, there are things which might be interesting for us, so most of steps are enabled. It takes a second to see if commit is boring or not. Plus, it prevents steps using `grep` from failing whole job when match can't be found.
Now. Your logic step could follow same ::set-env + if scheme.
- name: Logic run: | if [ $need_test_1 -eq 1 ]; then echo ::set-env name=TEST_1::true fi if [ $need_test_2 -eq 1 ]; then echo ::set-env name=TEST_2::true fi # ... if [ $need_test_N -eq 1 ]; then echo ::set-env name=TEST_N::true fi shell: bash - name: Test 1 if: env.TEST_1 == 'true' run: test/1.sh shell: bash - name: Test 2 if: env.TEST_2 == 'true' run: test/2.sh shell: bash # ... - name: Test N if: env.TEST_N == 'true' run: test/N.sh shell: bash
That should be enough for a quick POC. Later, when whole thing grows, you might want to move "Logic" step into its own script, which [at least for me] would be easier to mantain. Key here is to populate `env` context, that way or another, so it can be used later with if.
Note that if you feel your logic is getting too complex, and environment variables starts to limit you, there's also ::set-output available, which could help if one logic step is not enough, or any other reason.
- name: Logic 1 id: logic1 run: | if [ $need_test_1_1 -eq 1 ]; then echo ::set-output name=TEST_1::true fi if [ $need_test_1_2 -eq 1 ]; then echo ::set-output name=TEST_2::true fi
shell: bash - name: Logic 2 id: logic2 run: | if [ $need_test_2_1 -eq 1 ]; then echo ::set-output name=TEST_1::true fi if [ $need_test_2_2 -eq 1 ]; then echo ::set-output name=TEST_2::true fi
shell: bash - name: Test 1.1 if: steps.logic1.outputs.TEST_1 == 'true' run: test/1.sh --casual shell: bash - name: Test 1.2 if: steps.logic1.outputs.TEST_2 == 'true' run: test/2.sh --casual shell: bash - name: Test 2.1 if: steps.logic2.outputs.TEST_1 == 'true' run: test/1.sh --harsh --mode shell: bash - name: Test 2.2 if: steps.logic2.outputs.TEST_2 == 'true' run: test/2.sh --harsh --mode shell: bash
^ Kinda silly, and way too long example (there are better ways to achieve same result) but it should show few differences between ::set-env and ::set-output usage.
Hi Wipe, thanks for your response! Perhaps I was unclear, but I know I can do this - what I want to do is to *not* declare any steps ahead of time. Otherwise, if someone adds a test it also needs to be added to CI, and the same goes for all other CI steps I might want to do. This leads to duplication, and requires twice the effort for each task I need.
I.e., given that I already have the first part, I want to from the same step create jobs for each task I've identified as necessary.