Why I am unable to access service created by docker-compose? - docker

Question

I am building a micro-service based web app with Flask and Docker. Currently, I have 2 services running by docker-compose under the same default network.
mysql
product_api
docker-compose.yml file is --
version: '3'
services:
mysql:
image: mysql
environment:
MYSQL_USER: "mysqluser"
MYSQL_PASSWORD: "mysqlpassword"
MYSQL_ROOT_PASSWORD: "root"
MYSQL_DATABASE: "gadgetfreeiot"
container_name: mysql
ports:
- 3306:3306
restart: always
entrypoint: ['docker-entrypoint.sh', '--default-authentication-plugin=mysql_native_password']
product_api:
build: ./${SERVICE_ROOT:-src/services/product/api}
image: product_api:v1
container_name: product_api
volumes:
- ./${SERVICE_ROOT:-src/services/product/api}:${PROJECT_ROOT:-/usr/projects/gadgetfreeiot}/${SERVICE_ROOT:-src/services/product/api}
ports:
- 5000:5000
depends_on:
- mysql
environment:
username: "mysqluser"
password: "mysqlpassword"
host: "mysql"
port: "3306"
database: "gadgetfreeiot"
command: ["./wait-for-mysql.sh", "--", "python", "./run.py"]
docker ps output gives me --
johir#ubuntu:gadgetfreeiot$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e10329e26e5c product_api:v1 "./wait-for-mysql.sh…" 18 minutes ago Up 18 minutes 0.0.0.0:5000->5000/tcp product_api
7fed5a136123 mysql "docker-entrypoint.s…" 18 minutes ago Up 18 minutes 0.0.0.0:3306->3306/tcp, 33060/tcp mysql
Both services are running under gadgetfreeiot_default network. docker inspect gadgetfreeiot_default shows that both are under the same network --
johir#ubuntu:gadgetfreeiot$ docker inspect gadgetfreeiot_default
[
{
"Name": "gadgetfreeiot_default",
"Id": "67e09ae3a33c0ff4203eefe4fee6ba421d3f68564c6e32c7d1cd04e866ac6850",
"Created": "2018-10-18T14:56:05.86215576+03:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "192.168.0.0/20",
"Gateway": "192.168.0.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"14ffb8eaa0c17de0122de142dbcf7aa5455b41b47eadb197e8be200c2375fbb3": {
"Name": "mysql",
"EndpointID": "38ed4140ed728271194ee82f12b3d937c53166f6159ab4e6fcf2d8087039ed06",
"MacAddress": "02:42:c0:a8:00:02",
"IPv4Address": "192.168.0.2/20",
"IPv6Address": ""
},
"e013059b510e42933d33f7c3fb7e141a19a6c78a0e34d031e5fce5e104aa8697": {
"Name": "product_api",
"EndpointID": "fdbe0ed92d0e53d6fc1040a50b1898e2bb87b34384f80b98e638a3a89a57c4e1",
"MacAddress": "02:42:c0:a8:00:03",
"IPv4Address": "192.168.0.3/20",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {
"com.docker.compose.network": "default",
"com.docker.compose.project": "gadgetfreeiot",
"com.docker.compose.version": "1.22.0"
}
}
]
Now I am trying to access the product_api as well as mysql services from my host OS. In the meantime, I am also trying to access from one container to another (from product_api to mysql and from mysql to product_api). mysql is accessible from all 3 that is my host OS, product_api and mysql itself by --
mysql -h172.19.0.2 -P3306 -umysqluser -p
product_api is also able to access by --
mysql -hmysql -P3306 -umysqluser -p
Luckily I am able to access curl http://localhost:5000 shows from product_api --
johir#ubuntu:gadgetfreeiot$ docker exec -it product_api bash
root#e013059b510e:/usr/projects/gadgetfreeiot/src/services/product/api# curl http://localhost:5000
{
"message": "Endpoint not found",
"status": "failed"
}
Where curl http://172.19.0.3:5000 shows --
curl: (7) Failed to connect to 172.19.0.3 port 5000: Connection refused
That means product_api is up & running and only accessible by localhost or 127.0.0.1 from inside the product_api container not by IP or by service name from outside product_api container that is neither from my host OS nor from mysql container.
Finally, I checked -- active networks among 3 by netstat -tln --
# product_api container
root#e10329e26e5c:/usr/projects/gadgetfreeiot/src/services/product/api# netstat -tln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 127.0.0.11:40071 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:5000 0.0.0.0:* LISTEN
# mysql container
root#7fed5a136123:/# netstat -tln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 127.0.0.11:35507 0.0.0.0:* LISTEN
tcp6 0 0 :::33060 :::* LISTEN
tcp6 0 0 :::3306 :::* LISTEN
# host OS
johir#ubuntu:gadgetfreeiot$ netstat -tln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN
tcp6 0 0 ::1:631 :::* LISTEN
tcp6 0 0 :::5000 :::* LISTEN
tcp6 0 0 :::3306 :::* LISTEN
Note:
port 5000 is not open by tcp6 in product_api container. On the other hand port 3306 is open by tcp6 on mysql container.
Question: Why I am unable to access product_api service from my host OS or even from mysql (in constraint, why tcp6 is not exposing port 5000 for product_api service)?

Answer 1

The short answer is, you aren't unable to access, as you showed with curl http://localhost:5000. It just seems to be a bit confusing who (host vs containers) can access which IPs and resolve which hostnames.
In your docker-compose.yml you mapped the ports 3306 and 5000 to your respective containers. So docker added a port forward from any interface on your host (w/o the interfaces of the docker networks) to your containers (have a look at the output of sudo iptables-save | grep 5000 if you are interested in how it's done under the hood, it will look something like this:
-A DOCKER -d 192.168.0.2/20 ! -i br-e013059b510e -o br-e013059b510e -p tcp -m tcp --dport 5000 -j ACCEPT
Access
So from your host, you can reach your service via
curl http://localhost:5000
From another computer (assuming firewall settings allow) via
curl http://your.hostname:5000
From a container on the same docker network
curl http://product_api:5000 # or
curl http://product_api.gadgetfreeiot_default:5000 # or
curl http://192.168.0.3:5000
To have both container on the same docker network, adjust your docker-compose.yml like this:
services:
mysql:
[...]
ports:
- 3306:3306
networks:
- gadgetfreeiot
[...]
product_api:
ports:
- 5000:5000
networks:
- gadgetfreeiot
[...]
networks:
gadgetfreeiot:
DNS
Container hostnames like product_api are not resolvable on the host. They are however inside your containers. Inside a container you have an extra docker DNS server at 127.0.0.11 which can resolve what your host can resove, plus docker hostnames like product_api.gadgetfreeiot_default. Try
nslookup product_api.gadgetfreeiot_default
on your host and from inside the container
docker exec -it mysql bash
Check https://docs.docker.com/network/ for more info on that.
With regards to your note: netstat doesn't show you which port is "open", i.e. allowed by the firewall, but which port is bound to by a program. Whether a program binds to a port on an interface on IPv4, v6 or both, is up to the program. This is not related to docker networking.