Eaddrinuse on running tests

Code to Cloud GitHub Actions
#1

Hey there,

my team is using Github Actions and all in all we are very happy. But a strange behavior occurs from time to time:
When we run our tests we start up some small scale local servers that mock our third party systems. So a test will start up a server, run its test and stop the server again.
In general this works fine, but sometimes we will get a random :eaddrinuse error.

Does somebody experience the same? Is there a way to avoid this? Further below I will add the config of our workflow.

on:
  push:
    branches: [master]

jobs:
  release:
    name: Stage Release (OTP ${{ matrix.otp }} | Elixir ${{ matrix.elixir }} | Node ${{ matrix.node }} | OS ${{ matrix.os }})
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest]
        otp: [23.x]
        elixir: [1.11.x]
        node: [12.x]
        postgres: [12]

    services:
      db:
        image: postgres:${{ matrix.postgres }}
        ports: ["5432:5432"]
        env:
          POSTGRES_PASSWORD: postgres
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:
      - name: Checkout Repository
        uses: actions/checkout@v2

      - name: Setup Elixir
        uses: erlef/setup-elixir@v1
        with:
          otp-version: ${{ matrix.otp }}
          elixir-version: ${{ matrix.elixir }}

      - name: Cache deps
        uses: actions/cache@v2
        with:
          path: deps
          key: ${{ runner.os }}-${{ matrix.elixir }}-mix-${{ hashFiles(format('{0}{1}', github.workspace, '/mix.lock')) }}
          restore-keys: |
            ${{ runner.os }}-${{ matrix.elixir }}-mix-

      - name: Install Dependencies
        run: mix deps.get

      - name: Cache _build/test
        uses: actions/cache@v2
        with:
          path: _build/test
          key: ${{ runner.os }}-${{ matrix.elixir }}-build-test-${{ hashFiles('**/mix.lock') }}
          restore-keys: |
            ${{ runner.os }}-${{ matrix.elixir }}-build-test

      - name: Run Tests
        run: mix test
...

Kind regards,
Peter :slight_smile:

#2

“Sometimes” sounds a bit like there may be a random timing issue involved, e.g. one server might not be fully stopped when another instance is started with the same port. Or if it’s a large (as in, running for a long time) test suite there might be an SO_REUSEADDR setting missing a socket somewhere, and sometimes the affected tests run with enough time between them, and sometimes they don’t.

#3

I guessed so myself, but all the tests that could cause this are running sequential. Each of the tests starts the server and ends them on exit.

The tests look like this

defmodule Somewhere.ServiceTest do
  use ExUnit.Case, async: false
  alias Somewhere.Service

  @valid_data "datadata"
 @invalid_data "invalid_datadata"
  describe "somewhere service api" do
    alias Somewhere.Service

    setup do
      Service.Server.start()
      on_exit(&Service.Server.stop/0)
    end

    test "pdf" do
      assert {:ok, pdf} = Service.get_document_pdf(@data)
      assert is_binary(pdf)
      assert {:error, ""} = Service.get_document_pdf(@invalid_data)
    end
  end
end

I think I will add a small delay into the setup script to ensure its no runtime issue within my application. I will let this thread know what happens after it :slight_smile:

Kind regards,
Peter

#4

That does not rule out a block because of port reuse. By default a port is blocked for a certain time after it is released, and you need to set the SO_REUSEADDR socket option to change that. Though I can’t see if your server might already be doing that.