Help
cancel
Showing results for 
Search instead for 
Did you mean: 
Copilot Lvl 3
Message 1 of 9

Insight into VM performance

Two of our applications now have CI jobs running on Actions. The setup was easy and we're looking forward to using Actions for more applications.

 

We noticed that the test suites currently take a lot longer to run when compared to Circle CI. To a large degree this is attributable to a lack of dependency caching (which we're glad to see is already being addressed). However, we have also noticed that the test suite itself takes notably longer (3m05s on Circle vs 5m15s on Actions)

 

Is this simply a lower spec machine that Actions are running on? Circle runs on a 2CPU/4096MB box.

 

It probably doesn't make a difference, but we're running Rails rspec tests. I know its still early days for Github actions, so perhaps comparisons like this are just premature.

8 Replies
GitHub Staff
Message 2 of 9

Re: Insight into VM performance

The virtual machines are 2 cores with 7 GB RAM - for Windows and Linux these are the "Standard_DS2_v2" VM types in Azure. https://help.github.com/en/articles/virtual-environments-for-github-actions

 

I don't know offhand why you would see different performance characteristics (beyond the caching, of course) but it would be interesting to understand your workload to understand why it's slower on Actions.

Copilot Lvl 3
Message 3 of 9

Re: Insight into VM performance

That is interesting.

 

The only thing I can think of is that our tests are currently quite chatty with the postgres database (which we are running as a service). Perhaps there is some kind of network latency to the db service. When I get a chance I will try do another benchmark with only tests that dont hit the database and see if that makes a difference.

GitHub Staff
Message 4 of 9

Re: Insight into VM performance

Aha, interesting.  Also, just curious about an apples-to-apples sort of comparison: how are you running Postgres on your other CI provider?  If it's not in a container, I wonder if there's a possible latency there, as well.

Copilot Lvl 3
Message 5 of 9

Re: Insight into VM performance

We're running Postgres in the same way on our other provider. Apart from the small differences in their schemas, the CI yml files look effectively identical. If you'd like I can share those yml files here

Copilot Lvl 3
Message 6 of 9

Re: Insight into VM performance

@ethomson, in https://github.community/t5/GitHub-Actions/Performance-on-Windows/m-p/39011 you will find a comparison between Travis, AppVeyor and GHA. As you see, there seems to be no issues with GNU/Linux or macOS. But on windows, GHA is x3.5 slower.

Pilot Lvl 1
Message 7 of 9

Re: Insight into VM performance

I doubt it has anything to do with container / network overhead, as in our case we're using an SQLite database in most tests, and they're still way slower compared to Travis CI.

Travis CI VM specs: https://docs.travis-ci.com/user/reference/overview/#virtualisation-environment-vs-operating-system

Pilot Lvl 1
Message 8 of 9

Re: Insight into VM performance

We "solved" the performance issue by using `tmpfs`, e.g.

    services:
      postgres:
        image: postgres:10-alpine
        options: >-
          --mount type=tmpfs,destination=/var/lib/postgresql/data
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
Highlighted
Pilot Lvl 1
Message 9 of 9

Re: Insight into VM performance

We've also faced a similar issue here: https://github.community/t5/GitHub-Actions/Running-steps-in-a-container-has-too-much-unexpected-over...

I've not been able to figure out the drastic performance difference (< 5 minutes on Travis CI vs. 18 minutes on GitHub Actions workflow) for our Behat tests.