How to properly connect with worker node in docker? - docker

I have docker setup as 1 Manager and 1 Worker. Both node are separate machine within a same network.
Initialized docker swarm in manager node and connected another PC to swarm using the docker swarm join-token worker code generated by manager node.
docker info Manager Node
Containers: 16
Running: 5
Paused: 0
Stopped: 11
Images: 303
Server Version: 18.03.1-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 572
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: v5out80i284bavbkhrny82non
Is Manager: true
ClusterID: 2h6jhemo4ch03zzk9dm8hkn97
Managers: 1
Nodes: 2
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.0.0.1
Manager Addresses:
10.0.0.1:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.13.0-37-generic
Operating System: KDE neon Developer Edition
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.691GiB
Name: wannamit
ID: KR2B:Q2E6:GAPR:HY6X:PYZQ:KUMU:DXCE:7YKI:E5MM:RRHO:BBWG:GM6S
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: amithp
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
docker info Worker Node
Containers: 4
Running: 4
Paused: 0
Stopped: 0
Images: 4
Server Version: 18.03.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: vvusfvjpenc9ymsotj4bcs25c
Is Manager: false
Node Address: 192.168.86.38
Manager Addresses:
10.0.0.1:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-22-generic
Operating System: Ubuntu 18.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.093GiB
Name: ubuntu
ID: SKCQ:JZGO:VUHX:HZN5:JD4H:4KPM:5RXK:DWG2:A7E6:WU4T:VQ5N:YHQB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
A Simple is deployed with 5 replication.
version: "3.2"
services:
webapp:
image: amithp/pyapp:latest
deploy:
replicas: 5
restart_policy:
condition: on-failure
resources:
limits:
cpus: "0.1"
memory: 50M
ports:
- "28888:28888"
networks:
- frontend-network
redis:
image: redis
command: redis-server --appendonly yes
deploy:
restart_policy:
condition: on-failure
networks:
- frontend-network
networks:
frontend-network:
external:
name: frontend-network
Deployment is a success. Mostly 2 is deployed in Manager node, 3 other and Redis is deployed over worker node. The docker image is flask app that shows total view count and origin IP from app is being served.
Hello world!
Hostname: 351d83b03555
HostIP: 10.0.0.28
Visits: cannot connect to Redis, counter disabled
Now if I visit localhost:2888 from manager node. It cannot connect to redis and only loops over 2 different IPs. I cross-checked those IP and they are from manager node. Also, I identified IP of containers in worker node and tried to ping from manager node's container, response cannot connect to host (no ping reply).
Am I doing something wrong or did I miss something?

Related

Docker-compose command not working on Windows 10

I have Docker for Windows 10 and always and it has worked well.
But today not working anymore. If I give a docker-compose up nothing happens, it just goes to the bottom line.
The commands docker-compose up, docker-compose build, docker-compose -v, nothing happens, no error, just goes to the bottom line.
I uninstalled docker and I disabled Hyper-V, I installed all again and it did not work.
One example of the docker-compose.yml
version: '3'
volumes:
data:
services:
db:
image: mysql:5.7
ports:
- "3306:3306"
volumes:
- data:/var/lib/mysql
environment:
- MYSQL_ROOT_PASSWORD=password
- MYSQL_DATABASE=db
Thanks!
Update:
I installed a version previous and I have the same thing.
Update2:
I downgrade to version 17.12.0-ce-win47 and it's works. More people with the same problem here.
https://github.com/docker/for-win/issues/2699
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 18.06.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.93-linuxkit-aufs
Operating System: Docker for Windows
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.934GiB
Name: linuxkit-00155d0f1009
ID: FTO7:ABFI:2B7X:5KYO:R4YA:R4QI:62LR:GYSA:QOWS:A4DO:RKOK:45WR
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 22
Goroutines: 46
System Time: 2018-10-10T00:45:47.3809481Z
EventsListeners: 1
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
For other people with the same problem, I downgrade to version 17.12.0-ce-win47 and it works.
Not is necessarily a solution, but I can work.

docker swarm container memory limit does not work

I'm experimenting with forcing a container to use more memory than it's allowed but I can't get it to work. The container is part of a stack defined with docker compose and it's deployed to docker in swarm mode.
Docker is allowing the container to go way above the 50M limit I've set. I was expecting docker to kill the container, throw an error, etc.
Can anyone help me on why Docker does not enforce the memory limit here?
The container in docker-compose.yml is defined to have a memory limit of 50M, and then I have setup a very simple PHP test which will try to allocate 200M. I've defined PHP mem limit to 128M.
This is my docker-compose.yml
version: "3"
services:
nginx:
image: nginx:latest
restart: unless-stopped
volumes:
- ./deploy/nginx/nginx.conf:/etc/nginx/nginx.conf
- ./public:/usr/share/nginx/html
ports:
- "8180:80"
links:
- app
app:
image: 127.0.0.1:5000/wpdemo
build:
context: .
dockerfile: Dockerfile-app
restart: unless-stopped
volumes:
- .:/var/www/html
links:
- mysql
deploy:
resources:
limits:
cpus: '0.50'
memory: 50M
reservations:
cpus: '0.25'
memory: 20M
mysql:
image: mysql:5.7
restart: unless-stopped
ports:
- "13306:3306"
environment:
MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
volumes:
- ~/docker/volumes/mysql:/var/lib/mysql
Instead of docker killing the container, it allows it to take as much memory as it wants and PHP eventually stops the process throwing the error below:
"PHP message: PHP Fatal error: Allowed memory size of 125829120 bytes exhausted (tried to allocate 67108872 bytes) in /var/www/html/public/index.php on line 4"
I'm using Ubuntu 18.04.
uname -a
Linux 4.18.10-041810-generic #201809260332 SMP Wed Sep 26 07:34:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Docker version 18.06.1-ce, build e68fc7a
docker-compose version 1.17.1, build unknown docker-py version: 2.5.1
CPython version: 2.7.15rc1 OpenSSL version: OpenSSL 1.1.0g 2 Nov 2017
This is the output of "docker stats" on the app container:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
679c8495ac1d stackdemo_app.1.hr3ufwlskhdafre39aqrshxyu 0.00% 43.81MiB / 50MiB 87.62% 106kB / 389kB 2.05GB / 10.6GB 5
This is the output of "docker info":
Containers: 36
Running: 5
Paused: 0
Stopped: 31
Images: 450
Server Version: 18.06.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: wnegv5lp41wfs3epfrua489or
Is Manager: true
ClusterID: hq7o176yffjglxzb9pu3fiomr
Managers: 1
Nodes: 1
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 192.168.1.120
Manager Addresses:
192.168.1.120:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.18.10-041810-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.49GiB
Name: rafxps15
ID: QEX7:FEB3:J76L:DCAQ:SO4S:SWVE:4XPI:PI6R:YM4C:MV4I:C3PM:FLOQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
As you said in comment, swap is enabled on host but swap limit in cgroups does not supported yet.
According to this enabling swap limit support. Note that reboot of system is essential.
At last, —-memory-swap flag should be set. If you want to prevent your PHP app accessing swap, you should set it with the same value of —-memory. More details about memory swap settings.

Docker worker node promote failing

Steps to reproduce the issue:
I am creating 5 node docker cluster AWS instances
Terminate the leader(Primary Manager) from AWS console
docker node demote (primary-node) from the terminal
docker node rm (primary-node)
docker node promote (worker-node) / docker node update (worker-node) --role manager
After Step 5 ,
Status of the worker node is down
Availability is active.
Status of the worker node is always down
I tried
docker node update --availability active (worker-node)
But it doesnt help
Output of docker version:
Docker version 18.03.1-ce, build 9ee9f40
Output of docker info:
Containers: 5
Running: 0
Paused: 0
Stopped: 5
Images: 18
Server Version: 18.03.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: pw2cgzi62tr1g5yn42gdue9sd
Is Manager: true
ClusterID: 4l9ngpfnlqov063np7efi9idw
Managers: 2
Nodes: 4
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 172.31.4.129
Manager Addresses:
172.31.4.129:2377
172.31.6.143:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-1060-aws
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 990.7MiB
Name: ip-172-31-4-129
ID: CBXR:P6YF:ICCQ:R7LT:XM4M:BBMD:N4FN:ZPRI:3VOC:FO54:Y7I6:6LHK
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Fix is as below:
1) When the primary leader leaves the cluster, swarm will chose a new leader.
2) Generate manager token of the New Primary leader
3) SSH into worker . Docker swarm leave . Docker swarm join with manager token

Docker overlay DNS cant resolve Container Name

I have 2 Docker Hosts with identical Docker Version and Configs.
Since yesterday i have a Problem with the Overlay Networks on one of the Hosts.
What i did:
docker network create -d overlay --attachable test_network
docker run --rm -it --network="test_network" --name test.DNS.name bash
wget test.DNS.name
On One Host the response is as expected:
bash-4.4# wget test.DNS.name
Connecting to test.DNS.name (10.0.1.6:80)
wget: can't connect to remote host (10.0.1.6): Connection refused
On the other the response is:
bash-4.4# wget test.DNS.name
wget: bad address 'test.DNS.name'
I have no idea where this could come from.
Every Idea is welcome
There the output from docker info (identical on both machines):
Containers: 3
Running: 2
Paused: 0
Stopped: 1
Images: 156
Server Version: 18.03.0-ce
Storage Driver: devicemapper
Pool Name: vg01-docker--pool
Pool Blocksize: 524.3kB
Base Device Size: 10.74GB
Backing Filesystem: xfs
Udev Sync Supported: true
Data Space Used: 22.89GB
Data Space Total: 42.95GB
Data Space Available: 20.06GB
Metadata Space Used: 6.922MB
Metadata Space Total: 5.583GB
Metadata Space Available: 5.576GB
Thin Pool Minimum Free Space: 4.295GB
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: m99xj3rhtk8ymabv1k78v4pur
Is Manager: true
ClusterID: of2g9dy3xbwj2jye6potfq5dq
Managers: 1
Nodes: 1
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.16.21.145
Manager Addresses:
10.16.21.145:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-693.21.1.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.4 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 62.74GiB
Name: gtunxlvd04346
ID: QPYZ:ZYLJ:DNXO:VCRC:Y2CG:LOEN:WSKN:52X5:JPNX:FLSB:GPIR:FY3U
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
thanks Constantin for your suggestion, we restartet the Host with the Problem and now it is working as expected.
So this Problem can be solved with a restart of the Host.
It would still be interesting to find the cause of this, but i think that if there is no relation between researching this issue and the chance this happens again.

Error response when installing Shipyard: Bind for 0.0.0.0:2375 failed: port is already allocated

I meet the problem when running the deploy sh of Shipyard. The error is
docker: Error response from daemon: driver failed programming external connectivity on endpoint shipyard-proxy (): Bind for 0.0.0.0:2375 failed: port is already allocated.
Then I use netstat -ntple command to find out who is listening port 2375.
The result is dockerd itself.
So I wonder how to overcome this problem in order to finish the installation of Shipyard.
And here is the docker ps
My docker info result is showed below
Containers: 8
Running: 4
Paused: 0
Stopped: 4
Images: 21
Server Version: 17.12.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9b55aab90508bd389d7654c4baf173a981477d55
runc version: 9f9c96235cc97674e935002fc3d78361b696a69e
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.13.0-37-generic
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 962.8MiB
Name: flora-virtual-machine
ID: J765:X2HP:UL4S:2SCV:7CZK:LG63:4H5V:QFDF:RUM5:2F2Y:XC6I:Y3AE
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Registry Mirrors:
http://4d7846d0.m.daocloud.io/
Live Restore Enabled: false

Resources