Dotnet inside Docker giving Connection Refused error

I am running an API which I made using Dotnet Core and Swagger inside a Docker container. This API connects with another API hosted in a server.

This connection works perfectly when I run my API locally in my machine, however, when i run it inside a docker container I am presented with a System.Net.Http.HttpRequestException: Connection refused when I try to make a request to the server. However, if I do the exact same request from inside the docker container´s CLI using curl the request seems to work perfectly fine, so I suppose it´s a problem either dotnet or swagger is causing.

Futhermore, if I run the docker container while connected to a VPN (to the same network at which the server is at) the exception is not thrown and it all works perfectly fine.

My Dockerfile:

FROM mcr.microsoft.com/dotnet/core/aspnet:3.1

# Set up working directory
WORKDIR /app

# copy config and application file
COPY netcoreapp3.1/publish/ app/

# expose port used by api
EXPOSE 5004
ENV ASPNETCORE_URLS=http://+:5004


ENTRYPOINT ["dotnet", "app/DigitalTwin.Api.dll"]

The exception thrown :

Microsoft.AspNetCore.Server.Kestrel[13]

digital-twin-api | Connection id "0HM316M7IU0UH", Request id "0HM316M7IU0UH:00000003": An unhandled exception was thrown by the application.

digital-twin-api | System.Net.Http.HttpRequestException: Connection refused

digital-twin-api | ---> System.Net.Sockets.SocketException (111): Connection refused

digital-twin-api | at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)

digital-twin-api | --- End of inner exception stack trace ---