Container ifconfig & nslookup IPs don't match - docker

Question

I'm running a container called "front-end". Other containers communicate with this container by using the name front-end which gets resolved into the containers IP address. I noticed something that doesn't make sense to me about this container.
When I shell into the container and run "ifconfig", I see one IP address. When I run "nslookup front-end" I see another IP address. I would have though that these two IPs should have matched. See my terminal output below:
I'm doing this in swarm-mode. Here is how I start the containers:
docker service create --with-registry-auth --name first --network my-network aarch64/ubuntu:16.04 sleep 99999999999999999
docker service create --with-registry-auth --name second --network my-network aarch64/ubuntu:16.04 sleep 99999999999999999
Terminal output to confirm the containers are up and running:
odroid#odroid64-cluster4:~/ssd/weaveworksdemo$ docker service ls
ID NAME REPLICAS IMAGE COMMAND
7v04cpi9b8uq second 1/1 aarch64/ubuntu:16.04 sleep 99999999999999999
cru6diqlv5on first 1/1 aarch64/ubuntu:16.04 sleep 99999999999999999
swarm service and docker PS commands
odroid#odroid64-cluster4:~/ssd/weaveworksdemo$ docker service ps first
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR
4dqy376icyi8fow1y1owwjzj0 first.1 aarch64/ubuntu:16.04 odroid64-cluster3 Running Running 2 minutes ago
odroid#odroid64-cluster4:~/ssd/weaveworksdemo$ docker service ps second
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR
aurs8nznz35to7pjcikn9903h second.1 aarch64/ubuntu:16.04 odroid64-cluster4 Running Running 2 minutes ago
odroid#odroid64-cluster4:~/ssd/weaveworksdemo$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cd28f6ec30e1 aarch64/ubuntu:16.04 "sleep 99999999999999" 3 minutes ago Up 3 minutes second.1.aurs8nznz35to7pjcikn9903h
As can be seen. First is running on cluster3, and second it running on cluster4.
Now, let's take a look at the my-network inspection:
odroid#odroid64-cluster4:~/ssd/weaveworksdemo$ docker network inspect my-network
[
{
"Name": "my-network",
"Id": "3kjlgo564nkmnccliuz7rlxa3",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.10.0.0/24",
"Gateway": "10.10.0.2"
}
]
},
"Internal": false,
"Containers": {
"cd28f6ec30e1ab33c09ef20ef4ee3968d7e7d71f83608a7ebd995986f9c7a049": {
"Name": "second.1.aurs8nznz35to7pjcikn9903h",
"EndpointID": "bf3089daf09a9de1a9b245807d8b3cb3cbee1d3e28887f8f8b5a375ba72d9e18",
"MacAddress": "02:42:0a:0a:00:06",
"IPv4Address": "10.10.0.6/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "258",
"encrypted": ""
},
"Labels": {}
}
]
The reason you only see container service "second" is because I am shelled into cluster4. If I were to shell into cluster3, and run the network inspect command, I would see the service "first". This is how network overlays work in Docker. Notice the IP is 10.10.0.6
Alright, let's shell into "second" and run ifconfig, and nslookup.
odroid#odroid64-cluster4:~/ssd/weaveworksdemo$ docker exec -ti second.1.aurs8nznz35to7pjcikn9903h bash
root#cd28f6ec30e1:/# ifconfig
eth0 Link encap:Ethernet HWaddr 02:42:0a:0a:00:06
inet addr:10.10.0.6 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::42:aff:fe0a:6/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1424 Metric:1
RX packets:13 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1038 (1.0 KB) TX bytes:648 (648.0 B)
eth1 Link encap:Ethernet HWaddr 02:42:ac:12:00:03
inet addr:172.18.0.3 Bcast:0.0.0.0 Mask:255.255.0.0
inet6 addr: fe80::42:acff:fe12:3/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:7374 errors:0 dropped:0 overruns:0 frame:0
TX packets:6551 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:36597121 (36.5 MB) TX bytes:443087 (443.0 KB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:4096 Metric:1
RX packets:12 errors:0 dropped:0 overruns:0 frame:0
TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:876 (876.0 B) TX bytes:876 (876.0 B)
root#cd28f6ec30e1:/# nslookup second
Server: 127.0.0.11
Address: 127.0.0.11#53
Non-authoritative answer:
Name: second
Address: 10.10.0.5
Notice how ifconfig shows 10.10.0.6, and resolving the name "front-end" returns 10.10.0.5. Also, notice how network inspect matches up with the ifconfig output.
Can anyone explain why this is the case? I take back that this is all working. Having some issues with connectivity between nodes, not sure if this might have something to do with it.
One other note, I'm running two containers with different names, but I see the same thing even if I run a single container.

Answer 1

It's about Docker Virtual-IP
For example:
You name a service redis-service in your compose file.
You can connect to that redis server via redis-service.
If you replicate this redis service to five instances, then which ip address should redis-service resolve to?
redis-service will be resolved to a virtual ip, and five redis instances will be load balanced by this virutal ip.
It's the same even you have only one instance.