Can't access docker-swarm container by service name - docker

Question

I can't ping a service by it's service name from another container on the same overlay network in docker swarm. My steps are:
# docker swarm init
# docker network create -d overlay --attachable net1
# docker service create --name dns1 --network net1 tutum/dnsutils sleep 3000
# docker service create --name dns2 --network net1 tutum/dnsutils sleep 3000
This creates a 1 node swarm, a user defined overlay network and 2 services. I should be able to exec into 1 container and ping the other via service name but it does not work:
# docker exec -it dns1.1.6rned8409m9jkqoxgutzjz4y4 /bin/bash
root#05cba6fd8a0b:/# ping dns2
PING dns2 (10.0.5.5) 56(84) bytes of data.
From 05cba6fd8a0b (10.0.5.3) icmp_seq=1 Destination Host Unreachable
From 05cba6fd8a0b (10.0.5.3) icmp_seq=2 Destination Host Unreachable
From 05cba6fd8a0b (10.0.5.3) icmp_seq=3 Destination Host Unreachable
^C
--- dns2 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3062ms
I can ping the container directly either via the full hostname (dns2.1.idkledfjgd5dwknv6pirywpfk) or IP (10.0.5.6).
Environment Info:
# docker network inspect -v net1
[
{
"Name": "net1",
"Id": "ngzwl7l7m0zb5brvee21mvfcz",
"Created": "2020-12-14T22:05:25.962132239Z",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.5.0/24",
"Gateway": "10.0.5.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"05cba6fd8a0bc4e480b50f91fb395d27ee4998277d480109cb95249c38852909": {
"Name": "dns1.1.6rned8409m9jkqoxgutzjz4y4",
"EndpointID": "6bcc76c8688527fcf26d2ed313e351a54b8de69d28cde4388032849a2ff91a3e",
"MacAddress": "02:42:0a:00:05:03",
"IPv4Address": "10.0.5.3/24",
"IPv6Address": ""
},
"c1d9252f528b177ac397b7b9bf627996993ddc0f54aad3ee3862d93dcac407a3": {
"Name": "dns2.1.idkledfjgd5dwknv6pirywpfk",
"EndpointID": "fafd8335715737c26c83ff8a3e7c52a302eb48cbb6b7bb75e396ed6a483bfd31",
"MacAddress": "02:42:0a:00:05:06",
"IPv4Address": "10.0.5.6/24",
"IPv6Address": ""
},
"lb-net1": {
"Name": "net1-endpoint",
"EndpointID": "09e3b875528a05dc39a910b8cfe5cfd57756681c4aeffd56a0c9fb41d6bffd23",
"MacAddress": "02:42:0a:00:05:04",
"IPv4Address": "10.0.5.4/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4101"
},
"Labels": {},
"Peers": [
{
"Name": "4dc98c7e5f08",
"IP": "192.168.1.26"
}
],
"Services": {
"dns1": {
"VIP": "10.0.5.2",
"Ports": [],
"LocalLBIndex": 269,
"Tasks": [
{
"Name": "dns1.1.6rned8409m9jkqoxgutzjz4y4",
"EndpointID": "6bcc76c8688527fcf26d2ed313e351a54b8de69d28cde4388032849a2ff91a3e",
"EndpointIP": "10.0.5.3",
"Info": {
"Host IP": "192.168.1.26"
}
}
]
},
"dns2": {
"VIP": "10.0.5.5",
"Ports": [],
"LocalLBIndex": 270,
"Tasks": [
{
"Name": "dns2.1.idkledfjgd5dwknv6pirywpfk",
"EndpointID": "fafd8335715737c26c83ff8a3e7c52a302eb48cbb6b7bb75e396ed6a483bfd31",
"EndpointIP": "10.0.5.6",
"Info": {
"Host IP": "192.168.1.26"
}
}
]
}
}
}
]
and
# docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.4.2-docker)
Server:
Containers: 3
Running: 2
Paused: 0
Stopped: 1
Images: 7
Server Version: 20.10.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: x2o135d3kkfxw6lb6mfyx8s3h
Is Manager: true
ClusterID: v5x80quwm3vwsubwdd6pclj4r
Managers: 1
Nodes: 1
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 192.168.1.26
Manager Addresses:
192.168.1.26:2377
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.4.73-1-pve
Operating System: Ubuntu 20.10
OSType: linux
Architecture: x86_64
CPUs: 6
Total Memory: 15.62GiB
Name: dockerHost
ID: CCGD:MQRE:PGJJ:YRU5:M4IM:5INT:EGA5:IER3:22UL:7CI3:PZOU:EZZ2
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No blkio weight support
WARNING: No blkio weight_device support

Answer 1

For anyone looking at this in the future. The issue for me was that I was running docker in a LXC container on proxmox (ubuntu 20.04 template). I tested this in a ubuntu 20.04 VM and it works exactly as expected. I don't know exactly what the issue is or if it can be fixed, but essentially running this in a LXC container will not work.