How can I investigate the root cause of a network is unreachable in Docker? I'm using Docker in swarm mode with docker-compose. My logs look like this they are full with this.
Jun 05 12:23:22 myServer dockerd[6151]: time="2019-06-05T12:23:22.996311465+02:00" level=warning msg="[resolver] connect failed: dial udp 10.58.194.11:53: connect: network is unreachable"
Jun 05 12:23:22 myServer dockerd[6151]: time="2019-06-05T12:23:22.996246500+02:00" level=warning msg="[resolver] connect failed: dial udp 10.58.194.16:53: connect: network is unreachable"
Jun 05 12:23:22 myServer dockerd[6151]: time="2019-06-05T12:23:22.996342243+02:00" level=warning msg="[resolver] connect failed: dial udp 10.58.194.11:53: connect: network is unreachable"
RedHat 7.5 Linux server
Docker version 18.02.0-ce, build fc4de44
docker-compose version 1.19.0, build 9e633ef
docker-compose.yml version: version: "3.4"
Docker info:
Containers: 121
Running: 115
Paused: 0
Stopped: 6
Images: 752
Server Version: 18.02.0-ce
Storage Driver: overlay
Backing Filesystem: xfs
Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: v7bi5f4cnbzmq4hrnbw6xlev8
Is Manager: true
ClusterID: 5wpoqhzk0vphn1grirb7y34jk
Managers: 1
Nodes: 1
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.49.0.242
Manager Addresses:
10.49.0.242:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9b55aab90508bd389d7654c4baf173a981477d55
runc version: 9f9c96235cc97674e935002fc3d78361b696a69e
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-862.14.4.el7.x86_64
Operating System: Red Hat Linux
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 47GiB
Name: myServer
ID: V5I2:SAMK:WAEE:TLGM:ZA7M:5C4D:A2DK:TQVA:462K:DA7M:RTAT:QDSF
Docker Root Dir: /net/volume/fs0/dckrdata
Debug Mode (client): false
Debug Mode (server): false
HTTP Proxy: http://localhost:3128
HTTPS Proxy: http://localhost:3128
No Proxy: .myApp.com,localhost,127.0.0.1
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
docker.myApp.com
registry:5000
127.0.0.0/8
Live Restore Enabled: false
Related
I am trying to learn docker so i installed official docker application from it's site and after installing i installed kitematic.I am using windows 10:
Client:
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 19.03.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.184-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.952GiB
Name: docker-desktop
ID: VABX:P63L:ONMD:575Z:O4HT:WEZJ:RBNK:VB3G:4DC7:LI7T:YGX6:E4JL
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 29
Goroutines: 44
System Time: 2019-08-21T11:22:46.4753036Z
EventsListeners: 2
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
PS D:\Software\Windows\Docker\Kitematic-Windows>
After running kitamatic and search busybox and clicked on create after a few second i got this error:
(HTTP code 500) server error - Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.65.1:53: read udp 192.168.65.3:57485->192.168.65.1:53: i/o timeout
to test i used terminal to create container :
PS D:\Software\Windows\Docker\Kitematic-Windows> docker run busybox
Unable to find image 'busybox:latest' locally
C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: Get https://registry-1.docker.io/v2/library/busybox/manifests/latest: Get https://auth.docker.io/token?account=731364&scope=repository%3Alibrary%2Fbusybox%3Apull&service=registry.docker.io: dial tcp: lookup auth.docker.io on 192.168.65.1:53: read udp 192.168.65.3:53973->192.168.65.1:53: i/o timeout.
See 'C:\Program Files\Docker\Docker\Resources\bin\docker.exe run --help'.
PS D:\Software\Windows\Docker\Kitematic-Windows>
What's happen ?
Your machine is using 192.168.65.1 as DNS server to resolve the IP for registry-1.docker.io.
In a lot of places, it's suggested to go in the settings and change the DNS from Automatic to Fixed: 8.8.8.8
I would like to test some docker swarm features and for that I have a windows PC and a mac book pro, both in my private Network.
I installed Docker for Windows (Windows 10 pro, using linux containers) and also Docker for mac.
Then I started both of them and also configured my router to allow the ports they need for TCP and UDP:
Port 2377 TCP for node communication
Port 7946 TCP/UDP for container network discovery.
Port 4789 UDP for the container ingress network.
Also I deactivated the firewall both on my pc and on my mac.
Then I ran docker swarm init on my macbook, which gave me a join token.
On my windows PC I entered that join command in the console and....... it failed!
I got an error message that ends with "... connection refused".
So, can you give me some advise or links to how to properly connect to local machines via docker swarm? I would LOVE to test it and use it for local development and testing of my apps. thanks!
Docker Info from Mac
$ docker info
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 185
Server Version: 18.03.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: v3fhiinezmdbbn98l0s6bgqzo
Is Manager: true
ClusterID: o9mcdlgtq37t5r86ganupstez
Managers: 1
Nodes: 1
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 192.168.65.3
Manager Addresses:
192.168.65.3:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.87-linuxkit-aufs
Operating System: Docker for Mac
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 4.095GiB
Name: linuxkit-025000000001
ID: 2D57:Q3QP:6UZ2:S6JV:WXLG:JN4H:TR6G:V3C3:P6ZP:2ENA:L7ES:OIJD
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
HTTP Proxy: docker.for.mac.http.internal:3128
HTTPS Proxy: docker.for.mac.http.internal:3129
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Docker Info from Windows
$ docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 18.09.2
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 09c8266bf2fcf9519a651b04ae54c967b9ab86ec
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.125-linuxkit
Operating System: Docker for Windows
OSType: linux
Architecture: x86_64
CPUs: 3
Total Memory: 7.768GiB
Name: linuxkit-00155d674805
ID: S7LD:PA6I:QGZR:YFQH:BR62:JS5C:DZLS:C6O3:RZUL:7ZXE:PRI6:HPRD
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 22
Goroutines: 46
System Time: 2019-04-11T13:28:11.3484452Z
EventsListeners: 1
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
Docker swarm join command output
$ docker swarm join --token SWMTKN-1-5rp7ownwv3ob27vl52ogo8z6d3mbxasdfasdfsadfkrf8hqjk1b5-bi2p5u7i7blk5wepw389sba0w 192.168.x.x:2377
Error response from daemon: rpc error: code = Unavailable desc = all
SubConns are in TransientFailure, latest connection error:
connection error:
desc = "transport: Error while dialing dial tcp 192.168.x.x:2377:
connect: connection refused"
The problem is that netiher docker Desktop for Mac nor for Windows with Linux containers are "true" dockers. Both are using virtual machines with Linux os where true docker engine works.
If I'm correct, 192.162.65.3 is not the IP of your Mac but the IP of vm within some virtual mac network.
Basing on this article https://docs.docker.com/docker-for-mac/docker-toolbox/ and this sentence "Also note that Docker Desktop for Mac can’t route traffic to containers, so you can’t directly access an exposed port on a running container from the hosting machine." Connecting Mac and Windows on Linux containers might not be easy.
I'd recommend for testing either get some cloud VMs or on Windows you can use docker-machine command to spawn multiple Linux VMs on which you can setup local swarm to test features you wish.
Steps to reproduce the issue:
I am creating 5 node docker cluster AWS instances
Terminate the leader(Primary Manager) from AWS console
docker node demote (primary-node) from the terminal
docker node rm (primary-node)
docker node promote (worker-node) / docker node update (worker-node) --role manager
After Step 5 ,
Status of the worker node is down
Availability is active.
Status of the worker node is always down
I tried
docker node update --availability active (worker-node)
But it doesnt help
Output of docker version:
Docker version 18.03.1-ce, build 9ee9f40
Output of docker info:
Containers: 5
Running: 0
Paused: 0
Stopped: 5
Images: 18
Server Version: 18.03.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: pw2cgzi62tr1g5yn42gdue9sd
Is Manager: true
ClusterID: 4l9ngpfnlqov063np7efi9idw
Managers: 2
Nodes: 4
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 172.31.4.129
Manager Addresses:
172.31.4.129:2377
172.31.6.143:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-1060-aws
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 990.7MiB
Name: ip-172-31-4-129
ID: CBXR:P6YF:ICCQ:R7LT:XM4M:BBMD:N4FN:ZPRI:3VOC:FO54:Y7I6:6LHK
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Fix is as below:
1) When the primary leader leaves the cluster, swarm will chose a new leader.
2) Generate manager token of the New Primary leader
3) SSH into worker . Docker swarm leave . Docker swarm join with manager token
I have 2 Docker Hosts with identical Docker Version and Configs.
Since yesterday i have a Problem with the Overlay Networks on one of the Hosts.
What i did:
docker network create -d overlay --attachable test_network
docker run --rm -it --network="test_network" --name test.DNS.name bash
wget test.DNS.name
On One Host the response is as expected:
bash-4.4# wget test.DNS.name
Connecting to test.DNS.name (10.0.1.6:80)
wget: can't connect to remote host (10.0.1.6): Connection refused
On the other the response is:
bash-4.4# wget test.DNS.name
wget: bad address 'test.DNS.name'
I have no idea where this could come from.
Every Idea is welcome
There the output from docker info (identical on both machines):
Containers: 3
Running: 2
Paused: 0
Stopped: 1
Images: 156
Server Version: 18.03.0-ce
Storage Driver: devicemapper
Pool Name: vg01-docker--pool
Pool Blocksize: 524.3kB
Base Device Size: 10.74GB
Backing Filesystem: xfs
Udev Sync Supported: true
Data Space Used: 22.89GB
Data Space Total: 42.95GB
Data Space Available: 20.06GB
Metadata Space Used: 6.922MB
Metadata Space Total: 5.583GB
Metadata Space Available: 5.576GB
Thin Pool Minimum Free Space: 4.295GB
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: m99xj3rhtk8ymabv1k78v4pur
Is Manager: true
ClusterID: of2g9dy3xbwj2jye6potfq5dq
Managers: 1
Nodes: 1
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.16.21.145
Manager Addresses:
10.16.21.145:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-693.21.1.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.4 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 62.74GiB
Name: gtunxlvd04346
ID: QPYZ:ZYLJ:DNXO:VCRC:Y2CG:LOEN:WSKN:52X5:JPNX:FLSB:GPIR:FY3U
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
thanks Constantin for your suggestion, we restartet the Host with the Problem and now it is working as expected.
So this Problem can be solved with a restart of the Host.
It would still be interesting to find the cause of this, but i think that if there is no relation between researching this issue and the chance this happens again.
I have docker setup as 1 Manager and 1 Worker. Both node are separate machine within a same network.
Initialized docker swarm in manager node and connected another PC to swarm using the docker swarm join-token worker code generated by manager node.
docker info Manager Node
Containers: 16
Running: 5
Paused: 0
Stopped: 11
Images: 303
Server Version: 18.03.1-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 572
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: v5out80i284bavbkhrny82non
Is Manager: true
ClusterID: 2h6jhemo4ch03zzk9dm8hkn97
Managers: 1
Nodes: 2
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.0.0.1
Manager Addresses:
10.0.0.1:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.13.0-37-generic
Operating System: KDE neon Developer Edition
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.691GiB
Name: wannamit
ID: KR2B:Q2E6:GAPR:HY6X:PYZQ:KUMU:DXCE:7YKI:E5MM:RRHO:BBWG:GM6S
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: amithp
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
docker info Worker Node
Containers: 4
Running: 4
Paused: 0
Stopped: 0
Images: 4
Server Version: 18.03.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: vvusfvjpenc9ymsotj4bcs25c
Is Manager: false
Node Address: 192.168.86.38
Manager Addresses:
10.0.0.1:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-22-generic
Operating System: Ubuntu 18.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.093GiB
Name: ubuntu
ID: SKCQ:JZGO:VUHX:HZN5:JD4H:4KPM:5RXK:DWG2:A7E6:WU4T:VQ5N:YHQB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
A Simple is deployed with 5 replication.
version: "3.2"
services:
webapp:
image: amithp/pyapp:latest
deploy:
replicas: 5
restart_policy:
condition: on-failure
resources:
limits:
cpus: "0.1"
memory: 50M
ports:
- "28888:28888"
networks:
- frontend-network
redis:
image: redis
command: redis-server --appendonly yes
deploy:
restart_policy:
condition: on-failure
networks:
- frontend-network
networks:
frontend-network:
external:
name: frontend-network
Deployment is a success. Mostly 2 is deployed in Manager node, 3 other and Redis is deployed over worker node. The docker image is flask app that shows total view count and origin IP from app is being served.
Hello world!
Hostname: 351d83b03555
HostIP: 10.0.0.28
Visits: cannot connect to Redis, counter disabled
Now if I visit localhost:2888 from manager node. It cannot connect to redis and only loops over 2 different IPs. I cross-checked those IP and they are from manager node. Also, I identified IP of containers in worker node and tried to ping from manager node's container, response cannot connect to host (no ping reply).
Am I doing something wrong or did I miss something?