Can't resolve hostnames between docker containers - docker

Question

I have two containers created in separate compose files (done for application isolation -- each application may have multiple containers defined in the compose file such as a backing database).
These containers are linked via an external network named "common".
An example compose file would be:
version: '2'
services:
rabbitmq:
image: "rabbitmq:3-management"
hostname: "rabbitmq"
container_name: "rabbitmq"
environment:
RABBITMQ_ERLANG_COOKIE: "SWQOKODSQALRPCLNMEQG"
RABBITMQ_DEFAULT_USER: "rabbitmq"
RABBITMQ_DEFAULT_PASS: "rabbitmq"
RABBITMQ_DEFAULT_VHOST: "/"
ports:
- "15672:15672"
- "5672:5672"
networks:
default:
external:
name: common
Docker versions:
root#server:~/# docker version
Client:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Mon, 10 Oct 2016 21:38:17 +1300
OS/Arch: linux/amd64
Server:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Mon, 10 Oct 2016 21:38:17 +1300
OS/Arch: linux/amd64
root#server:~/# docker-compose version
docker-compose version 1.8.1, build 878cff1
docker-py version: 1.10.3
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
The common network was created using:
docker network create common
Then I bring containers up using:
docker-compose up -d
Inspecting the network I get:
root#server:~# docker network inspect b5e8f81a8ea0
[
{
"Name": "common",
"Id": "b5e8f81a8ea063149298d2023be5740c8d971e0329a741abdafbac59fd882684",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "192.168.0.0/16"
}
]
},
"Internal": false,
"Containers": {
"6137864e71161b417f0659b88b3e17538fb60277ca818e58b255d5cc17932c3c": {
"Name": "db",
"EndpointID": "5d9800dedcc22bdb8e08a1d04712df4e2f2846f5448e4ce888f164f7877ce5a4",
"MacAddress": "02:42:c0:a8:00:03",
"IPv4Address": "192.168.0.3/16",
"IPv6Address": ""
},
"61b1288bf250196d02f3933c8bbde732fbcd24bdf3949c4f53b3b05aa87f3c7f": {
"Name": "rabbitmq",
"EndpointID": "484e3cc05e5a852150e6a7c429dc73d9ce4b097d4f53b07800c9998ef977565c",
"MacAddress": "02:42:c0:a8:00:02",
"IPv4Address": "192.168.0.2/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]
Other containers have crashed because they can't resolve the 'rabbitmq' hostname. Inspecting the crashed containers shows they are on the same network:
root#server:~# docker inspect bae5214bf619
"NetworkSettings": {
"Bridge": "",
"SandboxID": "54a8b0833204522a75cf2e4195c5838b336ab82438aa53d9f1f38276eb9f6061",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": null,
"SandboxKey": "/var/run/docker/netns/54a8b0833204",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"common": {
"IPAMConfig": null,
"Links": [
"db"
],
"Aliases": [
"rulesengine",
"bae5214bf619"
],
"NetworkID": "b5e8f81a8ea063149298d2023be5740c8d971e0329a741abdafbac59fd882684",
"EndpointID": "",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": ""
}
}
}
If I login to a container that is not trying to access the rabbitmq container and hasn't crashed, the hostname does not resolve. But, the ip address of the rabbitmq container is reachable.
root#server:~/David.Deployments# docker exec -it 6137864e7116 bash
root#db:/# ping rabbitmq
ping: unknown host
root#db:/# ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2): 56 data bytes
64 bytes from 192.168.0.2: icmp_seq=0 ttl=64 time=0.149 ms
64 bytes from 192.168.0.2: icmp_seq=1 ttl=64 time=0.043 ms
When signed into the rabbitmq container:
root#server:/# docker exec -it rabbitmq bash
root#rabbitmq:/# uname -n
rabbitmq
root#rabbitmq:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
445: eth0#if446: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:c0:a8:00:02 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.2/16 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::42:c0ff:fea8:2/64 scope link
valid_lft forever preferred_lft forever
root#rabbitmq:/# cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.0.2 rabbitmq
Are there any suggestions for how I can resolve this issue or things I should be checking?
Update 1
I have tried adding external_links, but the problem persists and no entry is added to /etc/hosts for the external hostname 'rabbitmq'

Answer 1

Per the discussion here: https://github.com/docker/docker/issues/13381
poga's suggestion to restart docker worked for me: systemctl restart docker

Answer 2

In my case the problem was that container was crashing when being accessed for the first time after being started because in the entrypoint script there was a command that was failing.