"Cannot assign requested address" using localhost:<PORT> inside container #26331

esabol · 2022-05-17T09:26:10Z

esabol
May 17, 2022

This is driving me crazy. I’ve trying to migrate from Travis CI to GitHub Actions CI for months, but I cannot get our test suite to pass when building and testing inside of a container. If I don’t use a container and use use “os: ubuntu-”, the tests pass. If I use a container, however, I get the dreaded “Cannot assign requested address” error. My googling brought up this issue:

  <a href="https://github.com/actions/virtual-environments/issues/3185" target="_blank" rel="noopener nofollow ugc">github.com/actions/virtual-environments</a>

Reverse name lookup is broken for current hostname in ubuntu VMs

<div class="github-info">
  <div class="date">
    opened <span class="discourse-local-date" data-format="ll" data-date="2021-04-14" data-time="15:59:36" data-timezone="UTC">03:59PM - 14 Apr 21 UTC</span>
  </div>

    <div class="date">
      closed <span class="discourse-local-date" data-format="ll" data-date="2021-04-26" data-time="11:55:20" data-timezone="UTC">11:55AM - 26 Apr 21 UTC</span>
    </div>

  <div class="user">
    <a href="https://github.com/lhotari" target="_blank" rel="noopener nofollow ugc">
      <img alt="lhotari" src="https://user-images.githubusercontent.com/22986767/181096978-cf8dae92-ea05-43da-90fb-909a5c6f8795.jpeg" class="onebox-avatar-inline" width="20" height="20">
      lhotari
    </a>
  </div>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
      awaiting-deployment
    </span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
      OS: Ubuntu
    </span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
      Area: Common Tools
    </span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
      investigate
    </span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
      external
    </span>
</div>

**Description**

Reverse name lookup is broken for the current hostname. Thi…s leads to odd issues. Here are some reports from GitHub Community:

This is a problem report in apache/pulsar: apache/pulsar#10232
and workaround in apache/pulsar apache/pulsar#10233

I have suggested this general workaround to the problem:

- name: Add the current IP address, long hostname and short hostname record to /etc/hosts file
  run: |
     echo -e "$(ip addr show eth0 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1)\t$(hostname -f) $(hostname -s)" | sudo tee -a /etc/hosts

Question, Bug, or Feature?: Bug

Virtual environments affected

Expected behavior

hostname -i, nslookup $(hostname -s) and nslookup $(hostname -l) should return the IP address of the current hostname.

Actual behavior

hostname -i, nslookup $(hostname -s) or nslookup $(hostname -l) returns an IP address that changes and the IP address doesn't belong to the current host.

Repro steps

Add this command to some action and run it ip a; while [ 1 ]; do nslookup $(hostname); sleep 3; done

Example output

1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0d:3a:11:ff:4c brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.35/16 brd 10.1.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20d:3aff:fe11:ff4c/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: &lt;NO-CARRIER,BROADCAST,MULTICAST,UP&gt; mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:b1:57:12:f2 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   fv-az132-427.w5jmtgevnbgefpdoz44oxoejwc.bx.internal.cloudapp.net
Address: 10.1.1.83

Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   fv-az132-427.w5jmtgevnbgefpdoz44oxoejwc.bx.internal.cloudapp.net
Address: 10.1.1.83

Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   fv-az132-427.w5jmtgevnbgefpdoz44oxoejwc.bx.internal.cloudapp.net
Address: 10.1.1.83

Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   fv-az132-427.w5jmtgevnbgefpdoz44oxoejwc.bx.internal.cloudapp.net
Address: 10.1.1.83

Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   fv-az132-427.w5jmtgevnbgefpdoz44oxoejwc.bx.internal.cloudapp.net
Address: 10.1.0.37

Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   fv-az132-427.w5jmtgevnbgefpdoz44oxoejwc.bx.internal.cloudapp.net
Address: 10.1.0.37

Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   fv-az132-427.w5jmtgevnbgefpdoz44oxoejwc.bx.internal.cloudapp.net
Address: 10.1.0.37

Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   fv-az132-427.w5jmtgevnbgefpdoz44oxoejwc.bx.internal.cloudapp.net
Address: 10.1.0.37

In the above case, the reverse lookup returns addresses 10.1.1.83 and 10.1.0.37 which aren't the ip addresses of the current host. The address of eth0 is 10.1.1.35 .

But that was supposedly fixed and resolved last year. I thought maybe ubuntu:14.04 wasn’t fixed (since it’s ancient and wasn’t listed in the issue), so I tried the workaround that folks used before it was fixed, but I can’t seem to get it working.

Can anyone help me, please?

Here’s a failing run:

  <a href="https://github.com/gearman/gearmand/runs/6462158831" target="_blank" rel="noopener nofollow ugc">GitHub</a>

Build software better, together

GitHub is where people build software. More than 83 million people use GitHub to discover, fork, and contribute to over 200 million projects.

Here’s my workflow yml:

  <a href="https://github.com/gearman/gearmand/blob/e42d60e587d60ac396461a75d6f66d4744e3a565/.github/workflows/c-cpp.yml" target="_blank" rel="noopener nofollow ugc">github.com</a>

gearman/gearmand/blob/e42d60e587d60ac396461a75d6f66d4744e3a565/.github/workflows/c-cpp.yml

  <pre><code class="lang-yml">name: GitHub Actions CI

on:
push:
branches: [ master ]
pull_request:
branches: [ master ]

jobs:
build:

name: ${{ matrix.config.name }}
runs-on: ${{ matrix.config.os }}
container: ${{ matrix.config.container }}

strategy:
  fail-fast: false
  matrix:
    config:
      - {name: 'ubuntu-14.04 gcc-4.8', os: ubuntu-latest, container: 'ubuntu:14.04', cc: 'gcc', cxx: 'g++', tag: '4.8', toolchain: 'ppa:ubuntu-toolchain-r/test'}

This file has been truncated. show original

Thanks,
Ed

Answered by esabol

May 27, 2022

Never mind! I just got it to work. Finally!

For some reason our test suite code to find an available port isn’t working inside the container, but if I force it to use a specific port number that I know is available, then it works! All tests pass! Yippee!

View full answer

esabol · 2022-05-25T23:56:07Z

esabol
May 25, 2022
Author

Trying again… Can anyone help, please???

0 replies

airtower-luna · 2022-05-26T08:31:46Z

airtower-luna
May 26, 2022

Does the container image you’re using define an IPv6 address for localhost in /etc/hosts? The Docker containers on the runners don’t get IPv6, even on the loopback device.

2 replies

nickeldan Jan 24, 2023

Is there no way to configure that? That is, is there any way to configure the workflow get an IPv6 loopback address?

airtower-luna Jan 24, 2023

Unfortunately not. 😕 Without the job container you could try to reconfigure the Docker daemon enable IPv6 for the Docker networks at least, but I have no idea if that'd bump into any other issues.

See also: actions/runner-images#668 It's a slightly different topic but closely related, note the "wontfix" tag.

esabol · 2022-05-26T09:08:10Z

esabol
May 26, 2022
Author

I think so? I “cat /etc/hosts” and run some other network commands in the workflow. Here’s the output:

  <a href="https://github.com/gearman/gearmand/runs/6462158831#step:4:24" target="_blank" rel="noopener nofollow ugc">GitHub</a>

Build software better, together

GitHub is where people build software. More than 83 million people use GitHub to discover, fork, and contribute to over 200 million projects.

127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.18.0.2 e1755e97477e

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
5: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff
inet 172.18.0.2/16 brd 172.18.255.255 scope global eth0
valid_lft forever preferred_lft forever

‘hostname -i’ shows ‘172.18.0.2’
‘hostname -I’ shows '172.18.0.2 ’

0 replies

airtower-luna · 2022-05-26T11:44:36Z

airtower-luna
May 26, 2022

My points is, if your software is trying to bind to the IPv6 address for localhost it can’t work in a container with no IPv6. Might be a thing to investigate.

0 replies

esabol · 2022-05-26T18:39:03Z

esabol
May 26, 2022
Author

I’m not sure how to investigate it. It’s trying to bind to “localhost”, certainly. What do I need to change in /etc/hosts to get it to work?

0 replies

airtower-luna · 2022-05-26T18:48:17Z

airtower-luna
May 26, 2022

Look at the code: Does it use IPv6 or IPv4/IPv6 dualstack? If yes, you could give it an IPv4 only mode.

For a very quick test you could remove the localhost name for ::1 with a command like this in your workflow:

sed -i -e 's/^::1\slocalhost\s\(.*\)/::1 \1/' /etc/hosts

0 replies

esabol · 2022-05-26T23:00:38Z

esabol
May 26, 2022
Author

I got a “Device or file busy” error using “sed -i”, so I changed it to this:

        sed 's/^::1\s\s*localhost\s\(.*\)/::1 \1/' /etc/hosts > /tmp/hosts.temp
        cp /tmp/hosts.temp /etc/hosts
        rm /tmp/hosts.temp

Here’s the new contents of /etc/hosts inside the container:

127.0.0.1	localhost
::1 ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
172.18.0.2	7076974732ab

And here’s the output of ip addr:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
5: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.2/16 brd 172.18.255.255 scope global eth0
       valid_lft forever preferred_lft forever

I really had high hopes this would work, but it didn’t. 😦 However, the error has now changed from “Cannot assign requested address” to “Connection refused”! So progress, I think! Any other suggestions?

libtest/client.cc:273: in start() pid(23275) localhost:59298 ping(libtest/client.cc:273: Connection refused), additionally pid: 23315 is alive: true waited: 17 server started. exec: /__w/gearmand/gearmand/libtool --mode=execute /__w/gearmand/gearmand/./gearmand/gearmand --verbose=INFO --log-file=var/log/gearmand.log8ch4wj --pid-file=var/run/gearmand.pid2itON2 --port=59298 --exceptions -u root --listen=localhost stderr:

Link: 92

I don’t get why the same code and the same commands work just fine in Travis CI and in the Docker engine running on my Linux box, but I appreciate the assistance!!!

0 replies

esabol · 2022-05-27T00:37:19Z

esabol
May 27, 2022
Author

Never mind! I just got it to work. Finally!

For some reason our test suite code to find an available port isn’t working inside the container, but if I force it to use a specific port number that I know is available, then it works! All tests pass! Yippee!

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GitHub Community

"Cannot assign requested address" using localhost:<PORT> inside container #26331

{{title}}

Reverse name lookup is broken for current hostname in ubuntu VMs

Replies: 8 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

"Cannot assign requested address" using localhost:<PORT> inside container #26331

Replies: 8 comments · 2 replies

esabol May 25, 2022 Author

esabol May 26, 2022 Author

127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 172.18.0.2 e1755e97477e

esabol May 26, 2022 Author

esabol May 26, 2022 Author

esabol May 27, 2022 Author

Replies: 8 comments 2 replies

esabol
May 25, 2022
Author

esabol
May 26, 2022
Author

127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.18.0.2 e1755e97477e

esabol
May 26, 2022
Author

esabol
May 26, 2022
Author

esabol
May 27, 2022
Author