Two of our applications now have CI jobs running on Actions. The setup was easy and we're looking forward to using Actions for more applications.
We noticed that the test suites currently take a lot longer to run when compared to Circle CI. To a large degree this is attributable to a lack of dependency caching (which we're glad to see is already being addressed). However, we have also noticed that the test suite itself takes notably longer (3m05s on Circle vs 5m15s on Actions)
Is this simply a lower spec machine that Actions are running on? Circle runs on a 2CPU/4096MB box.
It probably doesn't make a difference, but we're running Rails rspec tests. I know its still early days for Github actions, so perhaps comparisons like this are just premature.
The virtual machines are 2 cores with 7 GB RAM - for Windows and Linux these are the "Standard_DS2_v2" VM types in Azure. https://help.github.com/en/articles/virtual-environments-for-github-actions
I don't know offhand why you would see different performance characteristics (beyond the caching, of course) but it would be interesting to understand your workload to understand why it's slower on Actions.
That is interesting.
The only thing I can think of is that our tests are currently quite chatty with the postgres database (which we are running as a service). Perhaps there is some kind of network latency to the db service. When I get a chance I will try do another benchmark with only tests that dont hit the database and see if that makes a difference.
Aha, interesting. Also, just curious about an apples-to-apples sort of comparison: how are you running Postgres on your other CI provider? If it's not in a container, I wonder if there's a possible latency there, as well.
We're running Postgres in the same way on our other provider. Apart from the small differences in their schemas, the CI yml files look effectively identical. If you'd like I can share those yml files here
I doubt it has anything to do with container / network overhead, as in our case we're using an SQLite database in most tests, and they're still way slower compared to Travis CI.
Travis CI VM specs: https://docs.travis-ci.com/user/reference/overview/#virtualisation-environment-vs-operating-system
We "solved" the performance issue by using `tmpfs`, e.g.
services: postgres: image: postgres:10-alpine options: >- --mount type=tmpfs,destination=/var/lib/postgresql/data --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
We've also faced a similar issue here: https://github.community/t5/GitHub-Actions/Running-steps-in-a-container-has-too-much-unexpected-over...
I've not been able to figure out the drastic performance difference (< 5 minutes on Travis CI vs. 18 minutes on GitHub Actions workflow) for our Behat tests.