Nginx upstream connection refused - docker

Question

I have a docker deployment with 3 services (using docker-compose) and the following port mappings:
nginx (90 → 80)
node (3000 → 3000)
python (8001 → 8000)
Python is a demo aiohttp app (aiohttp-based) served on port 8000
The node app is a simple ssr frontend served on port 3000
Nginx acts as a reverse proxy and has this clause to route traffic to the python app:
location /api/ {
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
proxy_buffering off;
proxy_pass http://python:8000;
}
And this one to route to the node app:
location / {
proxy_pass http://node:3000;
include /etc/nginx/node_params;
}
The problem is that the none of the other two containers can connect to the python container:
$ docker-compose exec nginx curl 'http://python:8000/api/'
curl: (7) Failed to connect to python port 8000: Connection refused
Same by using the IP directly:
$ docker-compose exec node curl 'http://172.18.0.5:8000/api/'
curl: (7) Failed to connect to 172.18.0.5 port 8000: Connection refused
Checking open ports also fails:
$ docker-compose exec nginx nc -vz python 8000
$ <no response>
Only the python container can connect to itself:
$ docker-compose exec python curl 'http://python:8000/api/'
Response ok
$ docker-compose exec python nc -vz python 8000
python (172.18.0.5:8000) open
The other service (node) can be accessed normally. Pinging the container also works.
The only way it can be accessed is from outside the docker network by the mapped port (8001), i.e.:
$ curl http://localhost:8001/api/
Response ok
It works with any IP and even from another hosts over the internet:
$ curl http://my-app.mydomain.com:8001/api/
Response ok
I am also not able to reproduce this problem because the same project run on my local machine works completely fine. The only difference is that where I'm trying to run it it's using docker 17 (Docker version 17.06.0-ce, build 02c1d87) whereas my local machine runs docker 18 (Docker version 18.09.5, build e8ff056). Also the server is running fedora 24 vs fedora 29 on my machine.
What am I doing wrong?
This is my docker-compose.yml file
version: '3.7'
services:
python:
build: api
ports:
- 8001:8000
networks:
default:
aliases:
- python
restart: always
volumes:
- cdn:/app/cdn
frontend:
build:
context: nuxt
ports:
- 3000:3000
networks:
default:
aliases:
- node
restart: always
nginx:
build:
context: nginx
ports:
- 90:80
restart: always
volumes:
- cdn:/app/cdn
volumes:
cdn:
Edit:
$ docker inspect project_python_1
[
{
"Id": "98f3624ea0866665204167d9975b050977836b843c8294639e245897c0c8e44e",
"Created": "2019-05-07T14:03:17.714587695Z",
"Path": "/bin/sh",
"Args": [
"-c",
"cd src && python -m api"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 5268,
"ExitCode": 0,
"Error": "",
"StartedAt": "2019-05-07T14:03:18.860468562Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:6b9059304a2e0f5316204acaf37423a557dc8d14dbc3bc72e169430ff38df73c",
"ResolvConfPath": "/var/lib/docker/containers/98f3624ea0866665204167d9975b050977836b843c8294639e245897c0c8e44e/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/98f3624ea0866665204167d9975b050977836b843c8294639e245897c0c8e44e/hostname",
"HostsPath": "/var/lib/docker/containers/98f3624ea0866665204167d9975b050977836b843c8294639e245897c0c8e44e/hosts",
"LogPath": "/var/lib/docker/containers/98f3624ea0866665204167d9975b050977836b843c8294639e245897c0c8e44e/98f3624ea0866665204167d9975b050977836b843c8294639e245897c0c8e44e-json.log",
"Name": "/project_python_1",
"RestartCount": 0,
"Driver": "overlay2",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"cdn:/app/cdn:rw"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "project_default",
"PortBindings": {
"8000/tcp": [
{
"HostIp": "",
"HostPort": "8001"
}
]
},
"RestartPolicy": {
"Name": "always",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": [],
"CapAdd": null,
"CapDrop": null,
"Dns": null,
"DnsOptions": null,
"DnsSearch": null,
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": null,
"DeviceCgroupRules": null,
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": -1,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/013c07caf2f6fd59e99a7ec626355e8820d7fe6c0d2f83d5ed0fd2a0c2688ea9-init/diff:/var/lib/docker/overlay2/b1986769f12e6919ad34bb2184a4822a18d01c402b187d8caf7d1088f6020da1/diff:/var/lib/docker/overlay2/919b177579f26bde763973564af0a3762db5fb9d801b9804f5038fb9c60e4250/diff:/var/lib/docker/overlay2/22389c009280043fe76e9e2631e59aa3d6ee35a827613114e39db5f4d29783b7/diff:/var/lib/docker/overlay2/098414feeb05448f0b70dad272c9c81976171d7626e902c9325c5a454b666e59/diff:/var/lib/docker/overlay2/91cf4d7cef0ffb067991afc5b99ebb7ffee6fb02ce6e258304b23202a49d71a9/diff:/var/lib/docker/overlay2/7d13e7a43ebd06c9babf901e9630ff663c5036886df08038ccbda5f730e7c3a5/diff:/var/lib/docker/overlay2/f8db754b7d72fc8cd0fcfdd758a9491ffc1029e7cac0f5f884d8f0ca26aee253/diff:/var/lib/docker/overlay2/b0cb3c0f4b0d1eba56f353767142bdccbe08b9d15cddf0b52f2173cb771f850a/diff:/var/lib/docker/overlay2/228b0ee3f88b6b9ab9a436612f416acb02dd7196fb3870ba632c973f560ca75e/diff:/var/lib/docker/overlay2/ee2d7a211a67bc164f787443de343de51efc89e00592a7516acd26f1a02bf520/diff:/var/lib/docker/overlay2/40a529d74eb8c72cbc3e57db301678996e229b4b4de31a5b3f5642c44018c499/diff:/var/lib/docker/overlay2/95534c69b64738866cd6a87a73dda2f049a28745bea72dbd54c6fb6f662202e3/diff:/var/lib/docker/overlay2/69ce7a7e7ad79423e0abab05a3b4270a4a309686ab4410759e05248286799cb6/diff:/var/lib/docker/overlay2/6525630fd688dbae59699c3cf1246cc5a202e4a4265b6cc17e238cd90867ad54/diff:/var/lib/docker/overlay2/66f8ad83ba1c1bd4c719ebfc004b85f4b6aef9bb15fba5f5ea9b5a58f7eb198c/diff:/var/lib/docker/overlay2/a1ca64fad83b74d88984bd7378905308ed5e9bc142f9fb50392b4414b6076eb2/diff",
"MergedDir": "/var/lib/docker/overlay2/013c07caf2f6fd59e99a7ec626355e8820d7fe6c0d2f83d5ed0fd2a0c2688ea9/merged",
"UpperDir": "/var/lib/docker/overlay2/013c07caf2f6fd59e99a7ec626355e8820d7fe6c0d2f83d5ed0fd2a0c2688ea9/diff",
"WorkDir": "/var/lib/docker/overlay2/013c07caf2f6fd59e99a7ec626355e8820d7fe6c0d2f83d5ed0fd2a0c2688ea9/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "volume",
"Name": "cdn",
"Source": "/var/lib/docker/volumes/cdn/_data",
"Destination": "/app/cdn",
"Driver": "local",
"Mode": "rw",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "98f3624ea086",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"8000/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"DEBUG=1",
"PATH=scripts:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"LANG=C.UTF-8",
"GPG_KEY=0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D",
"PYTHON_VERSION=3.7.3",
"PYTHON_PIP_VERSION=19.1"
],
"Cmd": [
"/bin/sh",
"-c",
"cd src && python -m api"
],
"ArgsEscaped": true,
"Image": "project_python",
"Volumes": {
"/app/cdn": {}
},
"WorkingDir": "/app",
"Entrypoint": null,
"OnBuild": null,
"Labels": {
"com.docker.compose.config-hash": "0f0fe6053d92416fd77f6efba7e8282f385c447b8a8d40aa866554ee282896d7",
"com.docker.compose.container-number": "1",
"com.docker.compose.oneoff": "False",
"com.docker.compose.project": "project",
"com.docker.compose.service": "python",
"com.docker.compose.version": "1.24.0"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "397d60b1dbe4733910c9ae2c0dabc1bdb3046d784b25f8fb4f72c28f6d458ff2",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"8000/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "8001"
}
]
},
"SandboxKey": "/var/run/docker/netns/397d60b1dbe4",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"project_default": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"98f3624ea086",
"api",
"python"
],
"NetworkID": "4145a30ce48519a895707d607265635012341f73db63b9fedf6e86d68fad6641",
"EndpointID": "4b4bafed80cb88693e2c3f3c1b0268f95afefc3eb7e713ce88d20392d36fa85c",
"Gateway": "172.18.0.1",
"IPAddress": "172.18.0.5",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:12:00:05",
"DriverOpts": null
}
}
}
}
]

Answer 1

Okay, so I found the culprit, the problem was that the machine I was deploying to had a port mapping set up via firewalld, 8000→80, on the main interface eth0, and docker was using that when trying to access the container. I.e. When the nginx container tried to connect to the python container in port 8000, it was actually using 80 as upstream and thus failing. A workaround is to either remove the port mapping or using an unmapped port. I have no idea why would docker apply the same rules of the system's firewalld in its internal networks.
This is the output of firewall-cmd --list-all
FedoraServer (active)
target: default
icmp-block-inversion: no
interfaces: eth0
sources:
services: dhcpv6-client
ports: 22/tcp 9090/tcp 90/tcp 8001/tcp 3000/tcp
protocols:
masquerade: no
forward-ports: port=8000:proto=tcp:toport=80:toaddr=
source-ports:
icmp-blocks:
rich rules:
And this is the output of docker network inspect project_default:
[
{
"Name": "project_default",
"Id": "4145a30ce48519a895707d607265635012341f73db63b9fedf6e86d68fad6641",
"Created": "2019-05-07T09:03:17.425575867-05:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.18.0.0/16",
"Gateway": "172.18.0.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"02f1f96b74eb292eeff1eb623e725a41c2a14aa0fc40f727ba78e0a620812254": {
"Name": "project_nginx_1",
"EndpointID": "68c3c7fb40d2e56d6601136a123fc8b7834c0503e3da99be56fac40750247a37",
"MacAddress": ...,
"IPv4Address": "172.18.0.3/16",
"IPv6Address": ""
},
"1e93a55f0d329f4cc8beb681c3e17c6aec1ded73de5dca2fc1eaf49dae788516": {
"Name": "project_mongo_1",
"EndpointID": "3a0a6ae0dfdc922b5fa6032c492376643e4b61415743af7afae2de33576f3acf",
"MacAddress": ...,
"IPv4Address": "172.18.0.4/16",
"IPv6Address": ""
},
"39ac596559da13506abcce9941a06441f42bd1c2d153d118bd13ff9a57f8c538": {
"Name": "project_node_1",
"EndpointID": "6753668d5fb20d908660b48bb757f9b6755c5f4f0bae69c7e02f5431c8f0e575",
"MacAddress": ...,
"IPv4Address": "172.18.0.2/16",
"IPv6Address": ""
},
"98f3624ea0866665204167d9975b050977836b843c8294639e245897c0c8e44e": {
"Name": "project_python_1",
"EndpointID": "4b4bafed80cb88693e2c3f3c1b0268f95afefc3eb7e713ce88d20392d36fa85c",
"MacAddress": ...,
"IPv4Address": "172.18.0.5/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {
"com.docker.compose.network": "default",
"com.docker.compose.project": "project",
"com.docker.compose.version": "1.24.0"
}
}
]
This is the stripped output of docker ps:
PORTS NAMES
0.0.0.0:8001->8000/tcp project_python_1
27017/tcp, 28017/tcp project_mongo_1
0.0.0.0:90->80/tcp project_nginx_1
0.0.0.0:3000->3000/tcp project_node_1