docker swarm container memory limit does not work - docker

I'm experimenting with forcing a container to use more memory than it's allowed but I can't get it to work. The container is part of a stack defined with docker compose and it's deployed to docker in swarm mode.
Docker is allowing the container to go way above the 50M limit I've set. I was expecting docker to kill the container, throw an error, etc.
Can anyone help me on why Docker does not enforce the memory limit here?
The container in docker-compose.yml is defined to have a memory limit of 50M, and then I have setup a very simple PHP test which will try to allocate 200M. I've defined PHP mem limit to 128M.
This is my docker-compose.yml
version: "3"
services:
nginx:
image: nginx:latest
restart: unless-stopped
volumes:
- ./deploy/nginx/nginx.conf:/etc/nginx/nginx.conf
- ./public:/usr/share/nginx/html
ports:
- "8180:80"
links:
- app
app:
image: 127.0.0.1:5000/wpdemo
build:
context: .
dockerfile: Dockerfile-app
restart: unless-stopped
volumes:
- .:/var/www/html
links:
- mysql
deploy:
resources:
limits:
cpus: '0.50'
memory: 50M
reservations:
cpus: '0.25'
memory: 20M
mysql:
image: mysql:5.7
restart: unless-stopped
ports:
- "13306:3306"
environment:
MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
volumes:
- ~/docker/volumes/mysql:/var/lib/mysql
Instead of docker killing the container, it allows it to take as much memory as it wants and PHP eventually stops the process throwing the error below:
"PHP message: PHP Fatal error: Allowed memory size of 125829120 bytes exhausted (tried to allocate 67108872 bytes) in /var/www/html/public/index.php on line 4"
I'm using Ubuntu 18.04.
uname -a
Linux 4.18.10-041810-generic #201809260332 SMP Wed Sep 26 07:34:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Docker version 18.06.1-ce, build e68fc7a
docker-compose version 1.17.1, build unknown docker-py version: 2.5.1
CPython version: 2.7.15rc1 OpenSSL version: OpenSSL 1.1.0g 2 Nov 2017
This is the output of "docker stats" on the app container:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
679c8495ac1d stackdemo_app.1.hr3ufwlskhdafre39aqrshxyu 0.00% 43.81MiB / 50MiB 87.62% 106kB / 389kB 2.05GB / 10.6GB 5
This is the output of "docker info":
Containers: 36
Running: 5
Paused: 0
Stopped: 31
Images: 450
Server Version: 18.06.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: wnegv5lp41wfs3epfrua489or
Is Manager: true
ClusterID: hq7o176yffjglxzb9pu3fiomr
Managers: 1
Nodes: 1
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 192.168.1.120
Manager Addresses:
192.168.1.120:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.18.10-041810-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.49GiB
Name: rafxps15
ID: QEX7:FEB3:J76L:DCAQ:SO4S:SWVE:4XPI:PI6R:YM4C:MV4I:C3PM:FLOQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support

As you said in comment, swap is enabled on host but swap limit in cgroups does not supported yet.
According to this enabling swap limit support. Note that reboot of system is essential.
At last, —-memory-swap flag should be set. If you want to prevent your PHP app accessing swap, you should set it with the same value of —-memory. More details about memory swap settings.

Related

Docker-compose command not working on Windows 10

I have Docker for Windows 10 and always and it has worked well.
But today not working anymore. If I give a docker-compose up nothing happens, it just goes to the bottom line.
The commands docker-compose up, docker-compose build, docker-compose -v, nothing happens, no error, just goes to the bottom line.
I uninstalled docker and I disabled Hyper-V, I installed all again and it did not work.
One example of the docker-compose.yml
version: '3'
volumes:
data:
services:
db:
image: mysql:5.7
ports:
- "3306:3306"
volumes:
- data:/var/lib/mysql
environment:
- MYSQL_ROOT_PASSWORD=password
- MYSQL_DATABASE=db
Thanks!
Update:
I installed a version previous and I have the same thing.
Update2:
I downgrade to version 17.12.0-ce-win47 and it's works. More people with the same problem here.
https://github.com/docker/for-win/issues/2699
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 18.06.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.93-linuxkit-aufs
Operating System: Docker for Windows
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.934GiB
Name: linuxkit-00155d0f1009
ID: FTO7:ABFI:2B7X:5KYO:R4YA:R4QI:62LR:GYSA:QOWS:A4DO:RKOK:45WR
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 22
Goroutines: 46
System Time: 2018-10-10T00:45:47.3809481Z
EventsListeners: 1
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
For other people with the same problem, I downgrade to version 17.12.0-ce-win47 and it works.
Not is necessarily a solution, but I can work.

"cAdvisor" is not allowing other containers to be removed

I have installed "cAdvisor" to monitor my containers on the host, now whenever I try to stop and remove any other containers, cAdvisor is not allowing to remove the other containers.
`# docker ps -a | grep -i api
790ddf58f54a container/container-abc "/bin/sh -c 'sh -c..." 42 hours ago Dead`
`# docker rm 790ddf58f54a
Error response from daemon: Unable to remove filesystem for
790ddf58f54acf041b3e33bc040ea035d43be92315c7d970f411ad56a855e627: remove /var/lib/docker/containers/790ddf58f54acf041b3e33bc040ea035d43be92315c7d970f411ad56a855e627/shm: device or resource busy`
`# docker rm 790ddf58f54a
Error response from daemon: Unable to remove filesystem for
790ddf58f54acf041b3e33bc040ea035d43be92315c7d970f411ad56a855e627: remove` `/var/lib/docker/containers/790ddf58f54acf041b3e33bc040ea035d43be92315c7d970f411ad56a855e627/shm: device or resource busy`
When I stop the cAdvisor, it allows me to remove the containers.
`# docker ps | grep -i cadvisor
b54e4acb3f36 google/cadvisor "/usr/bin/cadvisor..." 21 hours ago Up 21 hours 0.0.0.0:9911->8080/tcp cadvisor`
`# docker stop b54e4acb3f36
b54e4acb3f36`
`# docker rm 790ddf58f54a
790ddf58f54a
I reffered to this github issue, but of no help https://github.com/moby/moby/issues/34198. Also searched further on this, is there any permanent fix for this issue.
1) container/container-abc is being generated using below command.
# docker run -d --restart=on-failure:5 --name=container-abc -p 15200:15200 -p 15201:15201 container-abc-image
2) I am using docker-compose for cadvisor, below is the docker compose content.
services:
cadvisor:
image: google/cadvisor
container_name: cadvisor
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
ports:
- 9911:8080
privileged: true
restart: always
3) Below is the docker info command.
# docker info
Containers: 38
Running: 24
Paused: 0
Stopped: 14
Images: 310
Server Version: 1.13.1
Storage Driver: overlay
Backing Filesystem: extfs
Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
apparmor
Kernel Version: 3.12.74-60.64.85-default
Operating System: SUSE Linux Enterprise Server 12 SP1
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 31.27 GiB
ID: BTZ2:KQZM:VGL5:DN7P:LKEB:JMDY:57N6:JUC2:LIBA:UZWA:EU3T:CHWP
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 171
Goroutines: 132
System Time: 2018-07-16T03:58:55.156080332-07:00
EventsListeners: 0
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
WARNING: No kernel memory limit support
Experimental: false
Insecure Registries:
172.24.227.60:8090
127.0.0.0/8
Live Restore Enabled: false
Thanks in advance.

How to properly connect with worker node in docker?

I have docker setup as 1 Manager and 1 Worker. Both node are separate machine within a same network.
Initialized docker swarm in manager node and connected another PC to swarm using the docker swarm join-token worker code generated by manager node.
docker info Manager Node
Containers: 16
Running: 5
Paused: 0
Stopped: 11
Images: 303
Server Version: 18.03.1-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 572
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: v5out80i284bavbkhrny82non
Is Manager: true
ClusterID: 2h6jhemo4ch03zzk9dm8hkn97
Managers: 1
Nodes: 2
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.0.0.1
Manager Addresses:
10.0.0.1:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.13.0-37-generic
Operating System: KDE neon Developer Edition
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.691GiB
Name: wannamit
ID: KR2B:Q2E6:GAPR:HY6X:PYZQ:KUMU:DXCE:7YKI:E5MM:RRHO:BBWG:GM6S
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: amithp
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
docker info Worker Node
Containers: 4
Running: 4
Paused: 0
Stopped: 0
Images: 4
Server Version: 18.03.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: vvusfvjpenc9ymsotj4bcs25c
Is Manager: false
Node Address: 192.168.86.38
Manager Addresses:
10.0.0.1:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-22-generic
Operating System: Ubuntu 18.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.093GiB
Name: ubuntu
ID: SKCQ:JZGO:VUHX:HZN5:JD4H:4KPM:5RXK:DWG2:A7E6:WU4T:VQ5N:YHQB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
A Simple is deployed with 5 replication.
version: "3.2"
services:
webapp:
image: amithp/pyapp:latest
deploy:
replicas: 5
restart_policy:
condition: on-failure
resources:
limits:
cpus: "0.1"
memory: 50M
ports:
- "28888:28888"
networks:
- frontend-network
redis:
image: redis
command: redis-server --appendonly yes
deploy:
restart_policy:
condition: on-failure
networks:
- frontend-network
networks:
frontend-network:
external:
name: frontend-network
Deployment is a success. Mostly 2 is deployed in Manager node, 3 other and Redis is deployed over worker node. The docker image is flask app that shows total view count and origin IP from app is being served.
Hello world!
Hostname: 351d83b03555
HostIP: 10.0.0.28
Visits: cannot connect to Redis, counter disabled
Now if I visit localhost:2888 from manager node. It cannot connect to redis and only loops over 2 different IPs. I cross-checked those IP and they are from manager node. Also, I identified IP of containers in worker node and tried to ping from manager node's container, response cannot connect to host (no ping reply).
Am I doing something wrong or did I miss something?

Docker swarm can't resolve service name on other nodes

I had a small working docker swarm on google cloud platform.
There are just two nodes, one with nginx and php, the other one with mysql.
Right now it seems that from master node I can't connect to the mysql on the worker node.
SQLSTATE[HY000] [2002] php_network_getaddresses: getaddrinfo failed: Name or service not known
Same problem also with ping from a shell inside the container.
I've used --advertise-addr flag when init the swarm:
docker swarm init --advertise-addr 10.156.0.3
Then I've successfully join the swarm from the 2nd node:
docker swarm join --token my-token 10.156.0.3:2377
Also the deploy is successful
docker stack deploy --compose-file docker-compose.yml test
Creating network test_default
Creating service test_mysql
Creating service test_web
Creating service test_app
(in docker-compose.yml there is no network definition, I'm using the docker default)
Nodes:
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
oz1ebgrp1a68brxi0nd1gdr2k mysql-001 Ready Active 18.03.1-ce
ndy11zyxi0wym8mjmgh8op1ni * app-001 Ready Active Leader 18.03.1-ce
docker stack ps test
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
9afwjgtpy8lc test_app.1 127.0.0.1:5000/app:latest app-001 Running Running 8 minutes ago
mgajupmcai0t test_web.1 127.0.0.1:5000/web:latest app-001 Running Running 8 minutes ago
s17jvkukahl7 test_mysql.1 mysql:5.7 mysql-001 Running Running 8 minutes ago
docker networks:
NETWORK ID NAME DRIVER SCOPE
9084b39892f4 bridge bridge local
ofqtewx039fl test_default overlay swarm
5cc9d4554bea docker_gwbridge bridge local
97fbd06a23b5 host host local
x8f408klk2ms ingress overlay swarm
ca1b849ea73a none null local
Here is my docker info
Containers: 12
Running: 3
Paused: 0
Stopped: 9
Images: 35
Server Version: 18.03.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: ndy11zyxi0wym8mjmgh8op1ni
Is Manager: true
ClusterID: q23l1v6dav3u4anqqu51nwx0r
Managers: 1
Nodes: 2
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
Total Memory: 14.09GiB
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.156.0.3
Manager Addresses:
10.156.0.3:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.13.0-1019-gcp
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 14.09GiB
Name: app-001
ID: IWKK:NWRJ:HKAQ:3JSQ:7H3L:2WXC:IIJ7:OEKB:4ARR:T7FY:VAWR:HOPL
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
This swarm was working fine few weeks ago. I didn't need this application for few weeks so I've turned off all the machines. Meanwhile swarm-node.crt expired and so today when I've turned on the machine I had to remove the service and the swarm and recreate it from scratch. The result is that I can't connect from container on one node to container on the other node.
Any help will be appreciated.
UPDATE:
here is docker-compose.yml
version: '3'
services:
web:
image: 127.0.0.1:5000/web
build:
context: ./web
volumes:
- ./test:/var/www
build:
ports:
- 80:80
links:
- app
deploy:
mode: replicated
replicas: 1
placement:
constraints:
- node.hostname == app-001
app:
image: 127.0.0.1:5000/app
build:
context: ./app
volumes:
- ./test:/var/www
depends_on:
- mysql
deploy:
mode: replicated
replicas: 1
placement:
constraints:
- node.hostname == app-001
mysql:
image: mysql:5.7
volumes:
- /mnt/disks/ssd-001/mysql:/var/lib/mysql
- /mnt/disks/buckets/common-storage-001/backup/mysql:/backup
environment:
- "MYSQL_DATABASE=test"
- "MYSQL_USER=test"
- "MYSQL_PASSWORD=*****"
- "MYSQL_ROOT_PASSWORD=*****"
command: mysqld --key-buffer-size=32M --max-allowed-packet=16M --myisam-recover-options=FORCE,BACKUP --tmp-table-size=32M --query-cache-type=0 --query-cache-size=0 --max-heap-table-size=32M --max-connections=500 --thread-cache-size=50 --innodb-flush-method=O_DIRECT --innodb-log-file-size=512M --innodb-buffer-pool-size=16G --open-files-limit=65535
deploy:
mode: replicated
replicas: 1
placement:
constraints:
- node.hostname == mysql-001

Unable to connect to containers of a swarm in docker-in-docker

I have been playing around with docker-in-docker (dind) setups and am running into a weird problem.
If I run a docker container separately inside dind and expose a port then I could connect to the port without any problems. For example, using the docker swarm visualizer inside dind:
/home/dockremap # docker run -d -p 8080:8080 dockersamples/visualizer:stable
/home/dockremap # wget localhost:8080
Connecting to localhost:8080 (127.0.0.1:8080)
index.html 100% |*********************** ....
However, if I run the same inside a swarm by deploying from a compose file it doesn't work.
Here is what my compose file looks like:
version: "3"
services:
visualizer:
image: dockersamples/visualizer:stable
ports:
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
deploy:
placement:
constraints: [node.role == manager]
networks:
- webnet
networks:
webnet:
and the commands I run:
/home/dockremap # docker swarm init
/home/dockremap # docker stack deploy -c compose.yaml test
now when I do wget I get connection refused error:
/home/dockremap # wget localhost:8080
Connecting to localhost:8080 (127.0.0.1:8080)
wget: can't connect to remote host (127.0.0.1): Connection refused
Should doing this sort of thing in dind be able to work by default, or is there something I need to configure? I am using docker 17.03.1-ce on Windows and here is what I get when I run docker info in dind:
Containers: 2
Running: 1
Paused: 0
Stopped: 1
Images: 1
Server Version: 17.05.0-ce
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: active
NodeID: wz2r6iuyqztg3ivyk9fwsn976
Is Manager: true
ClusterID: mshadtrs0b1oayva2vrquf67d
Managers: 1
Nodes: 1
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Node Address: 172.17.0.2
Manager Addresses:
172.17.0.2:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.4.59-boot2docker
Operating System: Alpine Linux v3.5 (containerized)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 987.1MiB
Name: 7e480e7313ae
ID: EL7P:NI2I:TOR4:I7IW:DPAB:WKYU:6A6J:NCC7:3K3E:6YVH:PYVB:2L2W
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Resources