unable to access server in docker container from outside host machine - docker

Question

I have a linux machine (Host-591) hosting 2 VM instances. In the first VM instance (Docker03), inside a container, I have a Flask web server hosted, which runs on tcp/81 and publishes it. The host machine, Docker03 maps it to port tcp/5003.
From Docker03, I am able to access the Flask server:
root#Docker03:/home/ubuntu/docker/app3# curl http://192.168.122.103:5003/
Hello Root!
root#Docker03:/home/ubuntu/docker/app3#
But from the host linux machine, I am not able to access the server.
[root#Host-591 ~]# curl http://192.168.122.103:5003/
^C
Ping from the host machine to the server in the container inside Docker03 works fine.
[root#Host-591 ~]# ping 192.168.122.103 -c 1
PING 192.168.122.103 (192.168.122.103) 56(84) bytes of data.
64 bytes from 192.168.122.103: icmp_seq=1 ttl=64 time=0.225 ms
--- 192.168.122.103 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.225/0.225/0.225/0.000 ms
A simple telnet from the Host machine to the Flask server's port shows TCP connection can be established:
[root#Host-591 ~]# telnet 192.168.122.103 5003
Trying 192.168.122.103...
Connected to 192.168.122.103.
Escape character is '^]'.
^]
telnet> q
Connection closed.
[root#Host-591 ~]# ifconfig virbr0
virbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:1a:d4:4d txqueuelen 1000 (Ethernet)
RX packets 30436 bytes 7466531 (7.1 MiB)
RX errors 0 dropped 24 overruns 0 frame 0
TX packets 42414 bytes 65991140 (62.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
The iptables rules on Docker03 VM instance looks like:
root#Docker03:/home/ubuntu/docker/app3# iptables -t nat -vL -n
Chain PREROUTING (policy ACCEPT 1 packets, 84 bytes)
pkts bytes target prot opt in out source destination
1 84 DOCKER-INGRESS all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
3 204 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 1 packets, 84 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 5 packets, 364 bytes)
pkts bytes target prot opt in out source destination
4 240 DOCKER-INGRESS all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
0 0 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 5 packets, 364 bytes)
pkts bytes target prot opt in out source destination
4 240 MASQUERADE all -- * docker_gwbridge 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match src-type LOCAL
0 0 MASQUERADE all -- * !docker_gwbridge 172.18.0.0/16 0.0.0.0/0
9 582 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- docker_gwbridge * 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-INGRESS (2 references)
pkts bytes target prot opt in out source destination
3 180 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5003 to:172.18.0.2:5003
1 84 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
In the DOCKER-INGRESS chain, the pkt count increases when I try to access the server from Docker03 instance, but does not increase when I try to access it from the host machine (Host-591).
Inside Docker03:
root#Docker03:/home/ubuntu/docker/app3# ifconfig
...
docker_gwbridge Link encap:Ethernet HWaddr 02:42:a5:66:fb:c6
inet addr:172.18.0.1 Bcast:0.0.0.0 Mask:255.255.0.0
inet6 addr: fe80::42:a5ff:fe66:fbc6/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:55 errors:0 dropped:0 overruns:0 frame:0
TX packets:48 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3768 (3.7 KB) TX bytes:3560 (3.5 KB)
ens3 Link encap:Ethernet HWaddr 52:54:00:4d:a9:67
inet addr:192.168.122.103 Bcast:192.168.122.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fe4d:a967/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:16582 errors:0 dropped:8 overruns:0 frame:0
TX packets:7988 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:27505482 (27.5 MB) TX bytes:773745 (773.7 KB)
...
root#Docker03:/home/ubuntu/docker/app3# docker service ls
ID NAME MODE REPLICAS IMAGE
jgsuip3oda4e app3_web replicated 1/1 app3-web:v1
root#Docker03:/home/ubuntu/docker/app3# docker service inspect jgsuip3oda4e
[
{
"ID": "jgsuip3oda4ef2soefj0ce2oh",
"Version": {
"Index": 26
},
"CreatedAt": "2018-08-20T16:13:40.627151395Z",
"UpdatedAt": "2018-08-20T16:13:40.628064367Z",
"Spec": {
"Name": "app3_web",
"Labels": {
"com.docker.stack.namespace": "app3"
},
"TaskTemplate": {
"ContainerSpec": {
"Image": "app3-web:v1",
"Labels": {
"com.docker.stack.namespace": "app3"
}
},
"Resources": {},
"Placement": {},
"ForceUpdate": 0
},
"Mode": {
"Replicated": {
"Replicas": 1
}
},
"Networks": [
{
"Target": "giz5m1weca0xjlcsxjnvm5e81",
"Aliases": [
"web"
]
}
],
"EndpointSpec": {
"Mode": "vip",
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 81,
"PublishedPort": 5003,
"PublishMode": "ingress"
}
]
}
},
"Endpoint": {
"Spec": {
"Mode": "vip",
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 81,
"PublishedPort": 5003,
"PublishMode": "ingress"
}
]
},
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 81,
"PublishedPort": 5003,
"PublishMode": "ingress"
}
],
"VirtualIPs": [
{
"NetworkID": "s067fap1788lt9le1nfc5l2yh",
"Addr": "10.255.0.3/16"
},
{
"NetworkID": "giz5m1weca0xjlcsxjnvm5e81",
"Addr": "10.0.0.2/24"
}
]
},
"UpdateStatus": {
"StartedAt": "0001-01-01T00:00:00Z",
"CompletedAt": "0001-01-01T00:00:00Z"
}
}
]
root#Docker03:/home/ubuntu/docker/app3# docker network ls
NETWORK ID NAME DRIVER SCOPE
giz5m1weca0x app3_webnet overlay swarm
a2a6a0d8d2eb bridge bridge local
3d5bf5444e12 docker_gwbridge bridge local
97d487b3203e host host local
s067fap1788l ingress overlay swarm
efb9d06c92a8 none null local
root#Docker03:/home/ubuntu/docker/app3# docker network inspect docker_gwbridge
[
{
"Name": "docker_gwbridge",
"Id": "3d5bf5444e12adb0d8ed307144de2047372b5f56b2dead9718b414c8e6afa75b",
"Created": "2018-08-20T12:04:26.440509262-04:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.18.0.0/16",
"Gateway": "172.18.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Containers": {
"9eb13ae864ef07243c9b6c89713680248db9ba8e4e914e2f0173923c38d87d6f": {
"Name": "gateway_9eb13ae864ef",
"EndpointID": "48e44bfe94366c783f8bc59d1ed1bc3b8cefbbe534cdb4bf7cedfc4852b91213",
"MacAddress": "02:42:ac:12:00:03",
"IPv4Address": "172.18.0.3/16",
"IPv6Address": ""
},
"ingress-sbox": {
"Name": "gateway_ingress-sbox",
"EndpointID": "a9e15a62d6a678b2beb078f2eb99933c48ce44ebf4d2cc2912090ef75a12b75d",
"MacAddress": "02:42:ac:12:00:02",
"IPv4Address": "172.18.0.2/16",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.bridge.enable_icc": "false",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.name": "docker_gwbridge"
},
"Labels": {}
}
]
root#Docker03:/home/ubuntu/docker/app3# docker network inspect app3_webnet
[
{
"Name": "app3_webnet",
"Id": "giz5m1weca0xjlcsxjnvm5e81",
"Created": "2018-08-20T12:13:40.787096192-04:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.0.0/24",
"Gateway": "10.0.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Containers": {
"9eb13ae864ef07243c9b6c89713680248db9ba8e4e914e2f0173923c38d87d6f": {
"Name": "app3_web.1.8cejzgd75xul8brdjjjjnq0rb",
"EndpointID": "b5717c1dff888d993ff9a573b7967f90165c35e35774ca479b5d37cf0821e00d",
"MacAddress": "02:42:0a:00:00:03",
"IPv4Address": "10.0.0.3/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4097"
},
"Labels": {
"com.docker.stack.namespace": "app3"
},
"Peers": [
{
"Name": "Docker03-03ead807e067",
"IP": "192.168.122.103"
}
]
}
]
Here is the docker compose file that I am using:
root#Docker03:/home/ubuntu/docker/app3# cat docker-compose.yml
version: '3'
services:
web:
image: "app3-web:v1"
ports:
- "5003:81"
networks:
- "webnet"
networks:
webnet:
root#Docker03:/home/ubuntu/docker/app3# netstat -tulpn | grep 5003
tcp6 0 0 :::5003 :::* LISTEN 1610/dockerd
The Dockerfile for the app looks like this:
root#Docker03:/home/ubuntu/docker/app3# cat web/Dockerfile
FROM python:3.4-alpine
ADD . /web
WORKDIR /web
RUN pip install --proxy <proxy_ip_address:port> --trusted-host pypi.python.org -r requirements.txt
EXPOSE 81
# set environment variable so that python does not buffer any output logs
ENV PYTHONUNBUFFERED 0
CMD ["python", "index.py"]
I tried to spin up another VM instance connected to the 192.168.122.0/24 network and access the Flask server, and it worked fine. It seems only access from outside the VMs is not working. Docker version I am using:
root#Docker03:/home/ubuntu/docker/app3# docker --version
Docker version 17.03.2-ce, build f5ec1e2
Any help is much appreciated.
Thanks,