Docker Swarm does not create container - docker

I'm trying to create 3 zookeeper services in my docker swarm. However only managed to create 2 of the 3 containers:
docker ps -a returns:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2c883f9148ff hyperledger/fabric-zookeeper:latest "/docker-entrypoint.…" 2 minutes ago Up 2 minutes 2181/tcp, 2888/tcp, 3888/tcp fabric_zookeeper1.1.td4wpq2t9uj5yjnw0q76gsqi0
068ef5d9075b hyperledger/fabric-zookeeper:latest "/docker-entrypoint.…" 2 minutes ago Up 2 minutes 2181/tcp, 2888/tcp, 3888/tcp fabric_zookeeper2.1.u3zr2o8lifcncjo6g2u2yqhwu
docker network ls return:
NETWORK ID NAME DRIVER SCOPE
0e17f2cd7e8d bridge bridge local
4f78c376719f docker_gwbridge bridge local
djds6rgg0pqc fabric overlay swarm
o1es27fz05i1 fabric_net overlay swarm
2f99d3b30b86 host host local
ls05jfjuekg0 ingress overlay swarm
e7d8a3ff8bb2 net_blockcord bridge local
42ec3d9a4f1b none null local
docker network inspect fabric_net return:
[
{
"Name": "fabric_net",
"Id": "o1es27fz05i1g9cjrq5nvv0ok",
"Created": "2018-10-26T07:41:49.436040523Z",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.6.0/24",
"Gateway": "10.0.6.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"068ef5d9075bc9c61b313b97cfbb36189401bc4eb72258b4346f659add5b3a0a": {
"Name": "fabric_zookeeper2.1.u3zr2o8lifcncjo6g2u2yqhwu",
"EndpointID": "3274a8bc693c742a0acedd786174a1c7ed4c2843cd28a6ff9140a2e977059657",
"MacAddress": "02:42:0a:00:06:11",
"IPv4Address": "10.0.6.17/24",
"IPv6Address": ""
},
"2c883f9148ff3b53228e8d02a8bd60db754cd2677155307e5db31f426e356223": {
"Name": "fabric_zookeeper1.1.td4wpq2t9uj5yjnw0q76gsqi0",
"EndpointID": "f58c3c303a6f2fe22ba410e0881f67ce002cbfc5e0afe9cd1104f7f11e2c6ecf",
"MacAddress": "02:42:0a:00:06:15",
"IPv4Address": "10.0.6.21/24",
"IPv6Address": ""
},
"lb-fabric_net": {
"Name": "fabric_net-endpoint",
"EndpointID": "d70a81ad2631c3b76feac7484599e0715c9b901d2ed72153a38105b236b4c882",
"MacAddress": "02:42:0a:00:06:02",
"IPv4Address": "10.0.6.2/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4103"
},
"Labels": {
"com.docker.stack.namespace": "fabric"
},
"Peers": [
{
"Name": "a2beaca62ca3",
"IP": "10.0.0.5"
},
{
"Name": "fa12393e1d65",
"IP": "137.116.149.79"
}
]
}
]
With my container showing only 2 of my 3 zookeepers
I first create an overlay network
docker network create --attachable --driver overlay fabric
and ran the below docker compose file using command:
docker stack deploy -c docker-compose-zookeeper.yaml fabric
docker-compose-zookeeper.yaml
# Copyright IBM Corp. All Rights Reserved.
#
# SPDX-License-Identifier: Apache-2.0
#
version: '3'
networks:
net:
services:
zookeeper0:
hostname: zookeeper0.example.com
image: hyperledger/fabric-zookeeper
ports:
- 2181
- 2888
- 3888
environment:
- ZOO_MY_ID=1
- ZOO_SERVERS=server.1=0.0.0.0:2888:3888 server.2=zookeeper1:2888:3888 server.3=zookeeper2:2888:3888
networks:
- net
zookeeper1:
hostname: zookeeper1.example.com
image: hyperledger/fabric-zookeeper
ports:
- 2181
- 2888
- 3888
environment:
- ZOO_MY_ID=2
- ZOO_SERVERS=server.1=zookeeper0:2888:3888 server.2=0.0.0.0:2888:3888 server.3=zookeeper2:2888:3888
networks:
- net
zookeeper2:
hostname: zookeeper2.example.com
image: hyperledger/fabric-zookeeper
ports:
- 2181
- 2888
- 3888
environment:
- ZOO_MY_ID=3
- ZOO_SERVERS=server.1=zookeeper0:2888:3888 server.2=zookeeper1:2888:3888 server.3=0.0.0.0:2888:3888
networks:
- net
docker info:
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 15
Server Version: 18.06.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: x8mooygnt8mzruof5c5d3p0vp
Is Manager: true
ClusterID: vmqqjuwztz3sraag3e8dgpqbl
Managers: 2
Nodes: 2
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.0.0.5
Manager Addresses:
137.116.149.79:2377
168.63.239.163:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-1023-azure
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.853GiB
Name: blockcord-staging2
ID: UT5F:4ZFW:4PRT:LGFS:JIV4:3YAD:DK5I:BIYL:FU6P:ZFEB:3OD3:U5EX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support

Found out that the container was created in my other nodes. But my container wasnt able to resolve address of the service

Related

Can't access docker-swarm container by service name

I can't ping a service by it's service name from another container on the same overlay network in docker swarm. My steps are:
# docker swarm init
# docker network create -d overlay --attachable net1
# docker service create --name dns1 --network net1 tutum/dnsutils sleep 3000
# docker service create --name dns2 --network net1 tutum/dnsutils sleep 3000
This creates a 1 node swarm, a user defined overlay network and 2 services. I should be able to exec into 1 container and ping the other via service name but it does not work:
# docker exec -it dns1.1.6rned8409m9jkqoxgutzjz4y4 /bin/bash
root#05cba6fd8a0b:/# ping dns2
PING dns2 (10.0.5.5) 56(84) bytes of data.
From 05cba6fd8a0b (10.0.5.3) icmp_seq=1 Destination Host Unreachable
From 05cba6fd8a0b (10.0.5.3) icmp_seq=2 Destination Host Unreachable
From 05cba6fd8a0b (10.0.5.3) icmp_seq=3 Destination Host Unreachable
^C
--- dns2 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3062ms
I can ping the container directly either via the full hostname (dns2.1.idkledfjgd5dwknv6pirywpfk) or IP (10.0.5.6).
Environment Info:
# docker network inspect -v net1
[
{
"Name": "net1",
"Id": "ngzwl7l7m0zb5brvee21mvfcz",
"Created": "2020-12-14T22:05:25.962132239Z",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.5.0/24",
"Gateway": "10.0.5.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"05cba6fd8a0bc4e480b50f91fb395d27ee4998277d480109cb95249c38852909": {
"Name": "dns1.1.6rned8409m9jkqoxgutzjz4y4",
"EndpointID": "6bcc76c8688527fcf26d2ed313e351a54b8de69d28cde4388032849a2ff91a3e",
"MacAddress": "02:42:0a:00:05:03",
"IPv4Address": "10.0.5.3/24",
"IPv6Address": ""
},
"c1d9252f528b177ac397b7b9bf627996993ddc0f54aad3ee3862d93dcac407a3": {
"Name": "dns2.1.idkledfjgd5dwknv6pirywpfk",
"EndpointID": "fafd8335715737c26c83ff8a3e7c52a302eb48cbb6b7bb75e396ed6a483bfd31",
"MacAddress": "02:42:0a:00:05:06",
"IPv4Address": "10.0.5.6/24",
"IPv6Address": ""
},
"lb-net1": {
"Name": "net1-endpoint",
"EndpointID": "09e3b875528a05dc39a910b8cfe5cfd57756681c4aeffd56a0c9fb41d6bffd23",
"MacAddress": "02:42:0a:00:05:04",
"IPv4Address": "10.0.5.4/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4101"
},
"Labels": {},
"Peers": [
{
"Name": "4dc98c7e5f08",
"IP": "192.168.1.26"
}
],
"Services": {
"dns1": {
"VIP": "10.0.5.2",
"Ports": [],
"LocalLBIndex": 269,
"Tasks": [
{
"Name": "dns1.1.6rned8409m9jkqoxgutzjz4y4",
"EndpointID": "6bcc76c8688527fcf26d2ed313e351a54b8de69d28cde4388032849a2ff91a3e",
"EndpointIP": "10.0.5.3",
"Info": {
"Host IP": "192.168.1.26"
}
}
]
},
"dns2": {
"VIP": "10.0.5.5",
"Ports": [],
"LocalLBIndex": 270,
"Tasks": [
{
"Name": "dns2.1.idkledfjgd5dwknv6pirywpfk",
"EndpointID": "fafd8335715737c26c83ff8a3e7c52a302eb48cbb6b7bb75e396ed6a483bfd31",
"EndpointIP": "10.0.5.6",
"Info": {
"Host IP": "192.168.1.26"
}
}
]
}
}
}
]
and
# docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.4.2-docker)
Server:
Containers: 3
Running: 2
Paused: 0
Stopped: 1
Images: 7
Server Version: 20.10.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: x2o135d3kkfxw6lb6mfyx8s3h
Is Manager: true
ClusterID: v5x80quwm3vwsubwdd6pclj4r
Managers: 1
Nodes: 1
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 192.168.1.26
Manager Addresses:
192.168.1.26:2377
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.4.73-1-pve
Operating System: Ubuntu 20.10
OSType: linux
Architecture: x86_64
CPUs: 6
Total Memory: 15.62GiB
Name: dockerHost
ID: CCGD:MQRE:PGJJ:YRU5:M4IM:5INT:EGA5:IER3:22UL:7CI3:PZOU:EZZ2
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No blkio weight support
WARNING: No blkio weight_device support
For anyone looking at this in the future. The issue for me was that I was running docker in a LXC container on proxmox (ubuntu 20.04 template). I tested this in a ubuntu 20.04 VM and it works exactly as expected. I don't know exactly what the issue is or if it can be fixed, but essentially running this in a LXC container will not work.

container running on docker swarm not accessible from outside

I am running my containers on the docker swarm. asset-frontend service is my frontend application which is running Nginx inside the container and exposing port 80. now if I do
curl http://10.255.8.21:80
or
curl http://127.0.0.1:80
from my host where I am running these containers I am able to see my asset-frontend application but it is not accessible outside of the host. I am not able to access it from another machine, my host machine operating system is centos 8.
this is my docker-compose file
version: "3.3"
networks:
basic:
services:
asset-backend:
image: asset/asset-management-backend
env_file: .env
deploy:
replicas: 1
depends_on:
- asset-mongodb
- asset-postgres
networks:
- basic
asset-mongodb:
image: mongo
restart: always
env_file: .env
ports:
- "27017:27017"
volumes:
- $HOME/asset/mongodb:/data/db
networks:
- basic
asset-postgres:
image: asset/postgresql
restart: always
env_file: .env
ports:
- "5432:5432"
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=password
- POSTGRES_DB=asset-management
volumes:
- $HOME/asset/postgres:/var/lib/postgresql/data
networks:
- basic
asset-frontend:
image: asset/asset-management-frontend
restart: always
ports:
- "80:80"
environment:
- ENV=dev
depends_on:
- asset-backend
deploy:
replicas: 1
networks:
- basic
asset-autodiscovery-cron:
image: asset/auto-discovery-cron
restart: always
env_file: .env
deploy:
replicas: 1
depends_on:
- asset-mongodb
- asset-postgres
networks:
- basic
this is my docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
auz640zl60bx asset_asset-autodiscovery-cron replicated 1/1 asset/auto-discovery-cron:latest
g6poofhvmoal asset_asset-backend replicated 1/1 asset/asset-management-backend:latest
brhq4g4mz7cf asset_asset-frontend replicated 1/1 asset/asset-management-frontend:latest *:80->80/tcp
rmkncnsm2pjn asset_asset-mongodb replicated 1/1 mongo:latest *:27017->27017/tcp
rmlmdpa5fz69 asset_asset-postgres replicated 1/1 asset/postgresql:latest *:5432->5432/tcp
My 80 port is open in firewall
following is the output of firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: eth0
sources:
services: cockpit dhcpv6-client ssh
ports: 22/tcp 2376/tcp 2377/tcp 7946/tcp 7946/udp 4789/udp 80/tcp
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
if i inspect my created network the output is following
[
{
"Name": "asset_basic",
"Id": "zw73vr9xigfx7hy16u1myw5gc",
"Created": "2019-11-26T02:36:38.241352385-05:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.3.0/24",
"Gateway": "10.0.3.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"9348f4fc6bfc1b14b84570e205c88a67aba46f295a5e61bda301fdb3e55f3576": {
"Name": "asset_asset-frontend.1.zew1obp21ozmg8r1tzmi5h8g8",
"EndpointID": "27624fe2a7b282cef1762c4328ce0239dc70ebccba8e00d7a61595a7a1da2066",
"MacAddress": "02:42:0a:00:03:08",
"IPv4Address": "10.0.3.8/24",
"IPv6Address": ""
},
"943895f12de86d85fd03d0ce77567ef88555cf4766fa50b2a8088e220fe1eafe": {
"Name": "asset_asset-mongodb.1.ygswft1l34o5vfaxbzmnf0hrr",
"EndpointID": "98fd1ce6e16ade2b165b11c8f2875a0bdd3bc326c807ba6a1eb3c92f4417feed",
"MacAddress": "02:42:0a:00:03:04",
"IPv4Address": "10.0.3.4/24",
"IPv6Address": ""
},
"afab468aefab0689aa3488ee7f85dbc2cebe0202669ab4a58d570c12ee2bde21": {
"Name": "asset_asset-autodiscovery-cron.1.5k23u87w7224mpuasiyakgbdx",
"EndpointID": "d3d4c303e1bc665969ad9e4c9672e65a625fb71ed76e2423dca444a89779e4ee",
"MacAddress": "02:42:0a:00:03:0a",
"IPv4Address": "10.0.3.10/24",
"IPv6Address": ""
},
"f0a768e5cb2f1f700ee39d94e380aeb4bab5fe477bd136fd0abfa776917e90c1": {
"Name": "asset_asset-backend.1.8ql9t3qqt512etekjuntkft4q",
"EndpointID": "41587022c339023f15c57a5efc5e5adf6e57dc173286753216f90a976741d292",
"MacAddress": "02:42:0a:00:03:0c",
"IPv4Address": "10.0.3.12/24",
"IPv6Address": ""
},
"f577c539bbc3c06a501612d747f0d28d8a7994b843c6a37e18eeccb77717539e": {
"Name": "asset_asset-postgres.1.ynrqbzvba9kvfdkek3hurs7hl",
"EndpointID": "272d642a9e20e45f661ba01e8731f5256cef87898de7976f19577e16082c5854",
"MacAddress": "02:42:0a:00:03:06",
"IPv4Address": "10.0.3.6/24",
"IPv6Address": ""
},
"lb-asset_basic": {
"Name": "asset_basic-endpoint",
"EndpointID": "142373fd9c0d56d5a633b640d1ec9e4248bac22fa383ba2f754c1ff567a3502e",
"MacAddress": "02:42:0a:00:03:02",
"IPv4Address": "10.0.3.2/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4100"
},
"Labels": {
"com.docker.stack.namespace": "asset"
},
"Peers": [
{
"Name": "8170c4487a4b",
"IP": "10.255.8.21"
}
]
}
]
Ran into this same issue and it turns out it was a clash between my local networks subnet and the subnet of the automatically created ingress network. This can be verified using docker network inspect ingress and checking if the IPAM.Config.Subnet value overlaps with your local network.
To fix you can update the configuration of the ingress network as specified in Customize the default ingress network; in summary:
Remove services that publish ports
Remove existing network: docker network rm ingress
Recreate using non-conflicting subnet:
docker network create \
--driver overlay \
--ingress \
--subnet 172.16.0.0/16 \ # Or whatever other subnet you want to use
--gateway 172.16.0.1 \
ingress
Restart services
You can avoid a clash to begin with by specifying the default subnet pool when initializing the swarm using the --default-addr-pool option.
docker service update your-service --publish-add 80:80
You can publish ports by updating the service.
Can you try this url instead of the ip adres? host.docker.internal so something like http://host.docker.internal:80
I suggest you verify the "right" behavior using docker-compose first. Then, try to use docker swarm without network specification just to verify there are no network interface problems.
Also, you could use the below command to verify your LISTEN ports:
netstat -tulpn
EDIT: I faced this same issue but I was able to access my services through 127.0.0.1
While running docker provide an port mapping, like
docker run -p 8081:8081 your-docker-image
Or, provide the port mapping in the docker desktop while starting the container.
I got into this same issue. It turns out that's my iptables filter causes external connections not work.
In docker swarm mode, docker create a virtual network bridge device docker_gwbridge to access to overlap network. My iptables has following line to drop packet forwards:
:FORWARD DROP
That makes network packets from physical NIC can't reach the docker ingress network, so that my docker service only works on localhost.
Change iptables rule to
:FORWARD ACCEPT
And problem solved without touching the docker.

Docker restartmanger prevents restart despite restart policy

I have a docker container that likes to shutdown without restarting, despite having a restart=unless-stopped policy set.
Other containers are running on the same host (with similar startup configuration parameters) which I don't have any problems with. The host is a node in a swarm on a somewhat unstable network, and the container is frequent user of the node network (talking to the master node) so I'm not surprised that it would fail regularly, but I expect it to restart itself.
This is due to the restartmanger. The docker inspect State.error shows a message which clearly came from docker and not my container. The logs show:
... time="2019-09-21T02:06:31.969473802Z" level=error msg="restartmanger wait error: Could not attach to network cqr3v2jode1boqh2yofqrh7bx: context deadline exceeded"
So it appears that -- occasionally -- when the container gets restarted the network is down and the manger decides stop restarting. The question becomes how to override this behavior.
docker info:
Client:
Debug Mode: false
Server:
Containers: 4
Running: 2
Paused: 0
Stopped: 2
Images: 43
Server Version: 19.03.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: wgn64s7lx9jvgw36gtlu0dsou
Is Manager: false
Node Address: 10.0.0.2
Manager Addresses:
10.0.0.1:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.19.66-v7+
Operating System: Raspbian GNU/Linux 9 (stretch)
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 874.5MiB
Name: sensors-2
ID: NTRC:WPLS:GH2P:ZTLM:EDAN:H7HB:HGP6:6G6A:3YVW:T2I7:TVJU:XV3N
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
Here are the relevant bits from docker inspect on the non-restarting container. Note that it has restarted a few times, it exited due to a network error, and the MaximumRetryCount is 0 (which I assume is unlimited). Most recently it wasn't up for long... but my understanding of unless-stopped is that docker would continue restarting the container, though it would increase the delay between restarts.
[
{
"Id": "fa7c59dfa38f25c70d4c1293db27965c2e76af950fa19a2097b4ce63e1af2be4",
"Created": "2019-06-24T05:25:10.792698029Z",
"Path": "/srv/bin/weather_collector_server",
"Args": [
"/etc/config.ini"
],
"State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 1,
"Error": "Could not attach to network cqr3v2jode1boqh2yofqrh7bx: context deadline exceeded",
"StartedAt": "2019-09-21T03:56:40.911764904Z",
"FinishedAt": "2019-09-21T03:58:07.234852939Z"
},
"Image": "sha256:ee0e5023f37917f074dd0bf03dca328833eafd117fe69041203533768a196789",
"ResolvConfPath": "/var/lib/docker/containers/fa7c59dfa38f25c70d4c1293db27965c2e76af950fa19a2097b4ce63e1af2be4/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/fa7c59dfa38f25c70d4c1293db27965c2e76af950fa19a2097b4ce63e1af2be4/hostname",
"HostsPath": "/var/lib/docker/containers/fa7c59dfa38f25c70d4c1293db27965c2e76af950fa19a2097b4ce63e1af2be4/hosts",
"LogPath": "",
"Name": "/weather_collector_server",
"RestartCount": 3,
"Driver": "overlay2",
"Platform": "linux",
...
"HostConfig": {
...
"RestartPolicy": {
"Name": "unless-stopped",
"MaximumRetryCount": 0
},
...
],
"NetworkSettings": {
"Bridge": "",
"SandboxID": "0e901219511bb618d66943a12af1e09d8bbcb78ca4caa0bad88880f21d843c55",
...
"Networks": {
"hostnet": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"fa7c59dfa38f"
],
"NetworkID": "cqr3v2jode1boqh2yofqrh7bx",
"EndpointID": "",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "",
"DriverOpts": {}
}
}
}
}
]

Two services cannot see each other through a swarm overlay

I feel like this is simple, but I can't figure it out. I have two services, consul and traefik up in a single node swarm on the same host.
> docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
3g1obv9l7a9q consul_consul replicated 1/1 progrium/consul:latest
ogdnlfe1v8qx proxy_proxy global 1/1 traefik:alpine *:80->80/tcp, *:443->443/tcp
> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
090f1ed90972 progrium/consul:latest "/bin/start -server …" 12 minutes ago Up 12 minutes 53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8500/tcp, 8301-8302/udp consul_consul.1.o0j8kijns4lag6odmwkvikexv
20f03023d511 traefik:alpine "/entrypoint.sh -c /…" 12 minutes ago Up 12 minutes 80/tcp
Both containers have access to the "consul" overlay network, which was created as such.
> docker network create --driver overlay --attachable consul
ypdmdyx2ulqt8l8glejfn2t25
Traefik is complaining that it can't reach consul.
time="2019-03-18T18:58:08Z" level=error msg="Load config error: Get http://consul:8500/v1/kv/traefik?consistent=&recurse=&wait=30000ms: dial tcp 10.0.2.2:8500: connect: connection refused, retrying in 7.492175404s"
I can go into the traefik container and confirm that I can't reach consul through the overlay network, although it is pingable.
> docker exec -it 20f03023d511 ash
/ # nslookup consul
Name: consul
Address 1: 10.0.2.2
/ # curl consul:8500
curl: (7) Failed to connect to consul port 8500: Connection refused
# ping consul
PING consul (10.0.2.2): 56 data bytes
64 bytes from 10.0.2.2: seq=0 ttl=64 time=0.085 ms
However, if I look a little deeper, I find that they are connected, just that the overlay network isn't transmitting traffic to the actual destination for some reason. If I go directly to the actual consul ip, it works.
/ # nslookup tasks.consul
Name: tasks.consul
Address 1: 10.0.2.3 0327c8e1bdd7.consul
/ # curl tasks.consul:8500
Moved Permanently.
I could workaround this, technically there will only ever be one copy of consul running, but I'd like to know why the data isn't routing in the first place before I get deeper into it. I can't think of anything else to try. Here is various information related to this setup.
> docker --version
Docker version 18.09.2, build 6247962
> docker network ls
NETWORK ID NAME DRIVER SCOPE
cee3cdfe1194 bridge bridge local
ypdmdyx2ulqt consul overlay swarm
5469e4538c2d docker_gwbridge bridge local
5fd928ea1e31 host host local
9v22k03pg9sl ingress overlay swarm
> docker network inspect consul
[
{
"Name": "consul",
"Id": "ypdmdyx2ulqt8l8glejfn2t25",
"Created": "2019-03-18T14:44:27.213690506-04:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.2.0/24",
"Gateway": "10.0.2.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"0327c8e1bdd7ebb5a7871d16cf12df03240996f9e590509984783715a4c09193": {
"Name": "consul_consul.1.8v4bshotrco8fv3sclwx61106",
"EndpointID": "ae9d5ef1d19b67e297ebf40f6db410c33e4e3c0266c56e539e696be3ed4c81a5",
"MacAddress": "02:42:0a:00:02:03",
"IPv4Address": "10.0.2.3/24",
"IPv6Address": ""
},
"c21f5dfa93a2f43b747aedc64a343d94d6c1c2e6558d81bd4a52e2ba4b5fa90f": {
"Name": "proxy_proxy.sb6oindhmfukq4gcne6ynb2o2.4zvco02we58i3ulbyrsw1b2ok",
"EndpointID": "7596a208e0b05ba688f318814e24a2a1a3401765ed53ca421bf61c73e65c235a",
"MacAddress": "02:42:0a:00:02:06",
"IPv4Address": "10.0.2.6/24",
"IPv6Address": ""
},
"lb-consul": {
"Name": "consul-endpoint",
"EndpointID": "23e74716ef54f3fb6537b305176b790b4bc4132dda55f20588d7ce4ca71d7372",
"MacAddress": "02:42:0a:00:02:04",
"IPv4Address": "10.0.2.4/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4099"
},
"Labels": {},
"Peers": [
{
"Name": "e11b9bd30b31",
"IP": "10.8.0.1"
}
]
}
]
> cat consul/docker-compose.yml
version: '3.1'
services:
consul:
image: progrium/consul
command: -server -bootstrap
networks:
- consul
volumes:
- consul:/data
deploy:
labels:
- "traefik.enable=false"
networks:
consul:
external: true
> cat proxy/docker-compose.yml
version: '3.3'
services:
proxy:
image: traefik:alpine
command: -c /traefik.toml
networks:
# We need an external proxy network and the consul network
# - proxy
- consul
ports:
# Send HTTP and HTTPS traffic to the proxy service
- 80:80
- 443:443
configs:
- traefik.toml
volumes:
- /var/run/docker.sock:/var/run/docker.sock
deploy:
# Deploy the service to all nodes that match our constraints
mode: global
placement:
constraints:
- "node.role==manager"
- "node.labels.proxy==true"
labels:
# Traefik uses labels to configure routing to your services
# Change the domain to your own
- "traefik.frontend.rule=Host:proxy.mcwebsite.net"
# Route traffic to the web interface hosted on port 8080 in the container
- "traefik.port=8080"
# Name the backend (not required here)
- "traefik.backend=traefik"
# Manually set entrypoints (not required here)
- "traefik.frontend.entryPoints=http,https"
configs:
# Traefik configuration file
traefik.toml:
file: ./traefik.toml
# This service will be using two external networks
networks:
# proxy:
# external: true
consul:
external: true
There were two optional kernel configs CONFIG_IP_VS_PROTO_TCP and CONFIG_IP_VS_PROTO_UDP disabled in my kernel which, you guessed it, enable tcp and udp load balancing.
I wish I'd checked that about four hours sooner than I did.

host name not working with docker swarm mode

I am using docker version 18.06.1-ce and compose version 1.22.0.
As per docker, it should be possible to call services using service names. This is working for me with docker compose without swarm mode, but on swarm mode it is not working. I have even tried setting aliases in my compose but no result.
Below is my docker-compose.yml
version: "3"
networks:
my_network:
external:
name: new_network
services:
config-service:
image: com.test/config-service:0.0.1
deploy:
placement:
constraints: [node.role == manager]
resources:
limits:
memory: 1024M
reservations:
memory: 768M
restart_policy:
condition: on-failure
healthcheck:
test: ["CMD", "curl", "-f", "http://config-service:8888/health"]
interval: 5s
timeout: 3s
retries: 5
ports:
- 8888:8888
networks:
my_network:
aliases:
- config-service
eureka-service:
image: com.test/eureka-service:0.0.1
deploy:
placement:
constraints: [node.role == manager]
resources:
limits:
memory: 1536M
reservations:
memory: 1024M
restart_policy:
condition: on-failure
healthcheck:
test: ["CMD", "curl", "-I", "http://eureka-service:8761/health"]
interval: 5s
timeout: 3s
retries: 5
ports:
- 8761:8761
depends_on:
- config-service
networks:
my_network:
aliases:
- eureka-service
When I inspect into my network I found
[
{
"Name": "new_network",
"Id": "s2m7yq7tz4996w7eg229l59nf",
"Created": "2018-08-30T13:58:59.75070753Z",
"Scope": "swarm",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.20.0.0/16",
"Gateway": "172.20.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"355efe27067ee20868455dabbedd859b354d50fb957dcef4262eac6f25d10686": {
"Name": "test_eureka-service.1.a4pjb3ntez9ly5zhu020h0tva",
"EndpointID": "50998abdb4cd2cd2f747fadd82be495150919531b81a3d6fb07251a940ef2749",
"MacAddress": "02:42:ac:14:00:02",
"IPv4Address": "172.20.0.2/16",
"IPv6Address": ""
},
"5cdb398c598c1cea6b9032d4c696fd1581e88f0644896edd958ef59895b698a4": {
"Name": "test_config-service.1.se8ajr73ajnjhvxt3rq31xzlm",
"EndpointID": "5b3c41a8df0054e1c115d93c32ca52220e2934b6f763f588452c38e60c067054",
"MacAddress": "02:42:ac:14:00:03",
"IPv4Address": "172.20.0.3/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]
Now if I connect into containers terminal and ping using the long name 'test_config-service.1.se8ajr73ajnjhvxt3rq31xzlm' it is able to ping but not 'config-service'.
I believe the issue you are experiencing is because you are using a swarm scoped bridge network, instead of an overlay network. I'm not sure if this configuration is supported. The DNS entry for the service when deployed in swarm mode is at the service level, not the individual containers. From my testing, that DNS entry, along with the code to setup a VIP, only appear to work with overlay networks. You may want to follow this issue if you really need your network to be configured as a bridge: https://github.com/moby/moby/issues/37672
Otherwise, the easy fix is to replace your network with an overlay network. You can remove your network aliases since they are redundant. And if you have other containers on the host that need to also be on this network, from outside of swarm mode, be sure to configure your overlay network as "attachable". If you have other applications currently attached to the network, you can replace that with a new network, or if you need to keep the same network name, swap it out in two phases:
# create a temporary network to free up the new_network name
docker network create -d overlay --attachable temp_network
docker network connect temp_network $container_id # repeat for each container
# finish the above step for all containers before continuing
docker network disconnect new_network $container_id #repeat for each container
# remove the old bridge network
docker network rm new_network
# now create a new_network as overlay
docker network create -d overlay --attachable new_network
docker network connect new_network $container_id # repeat for each container
# finish the above step for all containers before continuing
docker network disconnect temp_network $container_id #repeat for each container
# cleanup the temporary network
docker network rm temp_network
If everything is running in swarm mode, then there's no need for --attachable. After that, you should be able to start your swarm mode stack.
Try to list your services with a docker service ls command. Because if you use stack and give a name to your stack the service name will be nameofstack_config-service
And I see in your inspect test_eureka-service.1xxxxxx so the service name should be test_eureka-service
This is a known issue with version 18.06:
https://github.com/docker/for-win/issues/2327
https://github.com/docker/for-linux/issues/375
Try 18.03

Resources