Two of our applications now have CI jobs running on Actions. The setup was easy and we're looking forward to using Actions for more applications.
We noticed that the test suites currently take a lot longer to run when compared to Circle CI. To a large degree this is attributable to a lack of dependency caching (which we're glad to see is already being addressed). However, we have also noticed that the test suite itself takes notably longer (3m05s on Circle vs 5m15s on Actions)
Is this simply a lower spec machine that Actions are running on? Circle runs on a 2CPU/4096MB box.
It probably doesn't make a difference, but we're running Rails rspec tests. I know its still early days for Github actions, so perhaps comparisons like this are just premature.
The virtual machines are 2 cores with 7 GB RAM - for Windows and Linux these are the "Standard_DS2_v2" VM types in Azure. https://help.github.com/en/articles/virtual-environments-for-github-actions
I don't know offhand why you would see different performance characteristics (beyond the caching, of course) but it would be interesting to understand your workload to understand why it's slower on Actions.
That is interesting.
The only thing I can think of is that our tests are currently quite chatty with the postgres database (which we are running as a service). Perhaps there is some kind of network latency to the db service. When I get a chance I will try do another benchmark with only tests that dont hit the database and see if that makes a difference.
Aha, interesting. Also, just curious about an apples-to-apples sort of comparison: how are you running Postgres on your other CI provider? If it's not in a container, I wonder if there's a possible latency there, as well.
We're running Postgres in the same way on our other provider. Apart from the small differences in their schemas, the CI yml files look effectively identical. If you'd like I can share those yml files here