Steps in parallel?

Is it possible to run given steps in parallel?

For example, we have a number of dependencies to build before we can build our component, and then test our component.

It would be nice if the dependency build steps could first run in parallel, followed by our component build step, and the test step.

25 Likes

Not at this moment. Only jobs can run in parallel, but steps always run sequentially.

I don’t know if there are any plans to ever support this.

3 Likes

Hi, is this still the case that the steps within the job cannot run in parallel.

If that is the case, I wonder if there is any related documentation on best structuring the jobs to achieve the same goal (basically, turn the tasks into jobs) 

2 Likes

If your steps are shell commands can you run them as a background task using the ‘&’ at the end of the command line.  You won’t be told when they finish.  I also don’t know if the workflow run will know to wait for the job to finish before it kills the instance.

If you REALLY need parallelism, remember that an action allows you to write code.  That code could trigger tasks in multiple threads.  Seems like a lot of work, but the ability should be there.

1 Like

As @amezick mentioned it is indeed possible via background tasks. And you can wait for them!

At our CI, we do something like this:

launch_as() {
  local cmd_name=$1
  shift
  (time $@ || echo $cmd_name >> fail.txt) 2>&1 > $cmd_name.txt
}

time npm ci
launch_as lint npm run lint &
time npm run build
launch_as build1 npm run build:build1 &
launch_as build2 npm run build:build2 &
launch_as build3 npm run build:build3 &

wait
cat lint.txt
cat build1.txt
cat build2.txt
cat build3.txt
if [ -f fail.txt ]; then
  cat fail.txt
  exit 1
fi
1 Like

Or you can use GNU Parallel!

2 Likes

I am looking forward seeing steps to be runnable in parallel. so that the setup steps can run once (installing dependencies, setting up database etc) but then everything else can be run in parallel

1 Like