Ubuntu runners regulary fail while installing dependency packages

Hi,

From time to time, but regularly, jobs on Ubuntu runners fail at the “install dependencies” steps. The apt command to install required packages fails with errors like:

Err:2 http://azure.archive.ubuntu.com/ubuntu bionic/main amd64 doxygen amd64 1.8.13-10
  Unable to connect to azure.archive.ubuntu.com:http:
Err:3 http://azure.archive.ubuntu.com/ubuntu bionic/universe amd64 libann0 amd64 1.1.2+doc-6
  Unable to connect to azure.archive.ubuntu.com:http:

Of course, the whole job fails and the workflow is marked as failed.

One example is available here. I get those errors from time to time.

It is obviously an operational error. Once is understandable. Network errors happen. But I have seen that too many times to consider it as a standard network error.

Anyone with similar errors?

1 Like

Hi @lelegard ,

Thank you for reaching this out! I confirmed this issue has been reported here: https://github.com/actions/virtual-environments/issues/675

Our team has fixed:

we just heard back from Canonical that they identified the issue as being due to load and added resources to deal with the traffic. This should mean the issues are resolved now.

I forked your sample repo and checked for 6 times, ubuntu runner can install dependency packages successfully now. Please kindly check again for your workflow.

Thanks a lot!

Hi,

Thank you for your answer. Like many people, I have heard of this temporary one-shot disruption of Azure. However, I do not think that this is related. It looks like repetitive transient network issues over a longer period. I can see that on Windows runners too. In all cases, the problem occurs in the “install dependencies” step, where a few packages are downloaded. I have also seen once the network problem during the “git pull” operation.

I appreciate that you tried on my repo with 6 successes in a row. But if you carefully read my initial report, the problem is transient and occurs in series. So, it may succeed 10 or 50 times (especially if run in a row) and fail the next time. So, this is unfortunately the kind of problem which is impossible to consider as fixed when you try it a couple of times with success. We need to wait until a couple of weeks without problem to consider it as fixed.

Here are a list of other similar network errors on runners at other times, other days, before and after my initial report:

https://github.com/tsduck/tsduck/runs/578310527?check_suite_focus=true

https://github.com/tsduck/tsduck/runs/547328229?check_suite_focus=true

https://github.com/tsduck/tsduck/runs/477325875?check_suite_focus=true

Other occurrences existed but are now erased since I manually restarted the jobs afterwards and the network errors are no longer in the logs. From now on, I will no longer try to restart failing jobs because of network errors in order to keep the errors in the log.

I can also confirm that from time to time I see network related issues - not necessarily Azure relaated, and not necessarily dependencies related. It is simply sporadic errors when some step attempts to access a remote URL. 

I have reported this to GitHub with examples, but it is hard to reproduce.

However, I can say that in the past week or two I haven’t encountered this problem.

Hi @lelegard ,

Thanks for your reply!  According to the latest comment in the issue provided, ‘apt-spy2’ is disabled, and other users reported that no more azure.archive.ubuntu.com-related failures now.  Could you please check for your workflow?

It’s good news although it may need some time for confirmation.

This one failed again in the “install dependencies” step. Note that this is a Windows runner, not a Ubuntu one. This is not the first time I see this problem with Windows. So, I confirm that this is a network issue, not an Ubuntu repository server issue because it appears on both platforms.

It is a bit unsafe that the CI tasks fail from time to time (let’s say once a week) because of networking issues. Nothing vital, of course, but that creates an uncumfortable feeling of unreliability.

1 Like

Hi,

Installation of dependencies before build broke again last night on a Windows runner because of networking errors. Can you please do something?

Hi @lelegard ,

I notice the installation of dependencies works fine in your repo now, sorry for the intermittent fail.

For such network issue, it’s recommended to raise a ticket here where you can discuss with dev team directly.

Thanks.