I have installed "cAdvisor" to monitor my containers on the host, now whenever I try to stop and remove any other containers, cAdvisor is not allowing to remove the other containers.
`# docker ps -a | grep -i api
790ddf58f54a container/container-abc "/bin/sh -c 'sh -c..." 42 hours ago Dead`
`# docker rm 790ddf58f54a
Error response from daemon: Unable to remove filesystem for
790ddf58f54acf041b3e33bc040ea035d43be92315c7d970f411ad56a855e627: remove /var/lib/docker/containers/790ddf58f54acf041b3e33bc040ea035d43be92315c7d970f411ad56a855e627/shm: device or resource busy`
`# docker rm 790ddf58f54a
Error response from daemon: Unable to remove filesystem for
790ddf58f54acf041b3e33bc040ea035d43be92315c7d970f411ad56a855e627: remove` `/var/lib/docker/containers/790ddf58f54acf041b3e33bc040ea035d43be92315c7d970f411ad56a855e627/shm: device or resource busy`
When I stop the cAdvisor, it allows me to remove the containers.
`# docker ps | grep -i cadvisor
b54e4acb3f36 google/cadvisor "/usr/bin/cadvisor..." 21 hours ago Up 21 hours 0.0.0.0:9911->8080/tcp cadvisor`
`# docker stop b54e4acb3f36
b54e4acb3f36`
`# docker rm 790ddf58f54a
790ddf58f54a
I reffered to this github issue, but of no help https://github.com/moby/moby/issues/34198. Also searched further on this, is there any permanent fix for this issue.
1) container/container-abc is being generated using below command.
# docker run -d --restart=on-failure:5 --name=container-abc -p 15200:15200 -p 15201:15201 container-abc-image
2) I am using docker-compose for cadvisor, below is the docker compose content.
services:
cadvisor:
image: google/cadvisor
container_name: cadvisor
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
ports:
- 9911:8080
privileged: true
restart: always
3) Below is the docker info command.
# docker info
Containers: 38
Running: 24
Paused: 0
Stopped: 14
Images: 310
Server Version: 1.13.1
Storage Driver: overlay
Backing Filesystem: extfs
Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
apparmor
Kernel Version: 3.12.74-60.64.85-default
Operating System: SUSE Linux Enterprise Server 12 SP1
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 31.27 GiB
ID: BTZ2:KQZM:VGL5:DN7P:LKEB:JMDY:57N6:JUC2:LIBA:UZWA:EU3T:CHWP
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 171
Goroutines: 132
System Time: 2018-07-16T03:58:55.156080332-07:00
EventsListeners: 0
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
WARNING: No kernel memory limit support
Experimental: false
Insecure Registries:
172.24.227.60:8090
127.0.0.0/8
Live Restore Enabled: false
Thanks in advance.
Related
Has anyone seen / resolved the below:
I have a jenkins slave with rootless dind configured, all docker commands work except docker run, details and error below:
Error:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:319: getting the final child's pid from pipe caused \"EOF\"": unknown.
ERRO[0004] error waiting for container: context canceled
config and versions:
uname -a
Linux jnlp-5n7x4 4.4.0-1092-aws #103-Ubuntu SMP Tue Aug 27 10:21:48 UTC 2019 x86_64 Linux
docker info:
Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 1
Server Version: 19.03.8
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: none
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
seccomp
Profile: default
rootless
Kernel Version: 4.4.0-1092-aws
Operating System: Alpine Linux v3.11 (containerized)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.812GiB
Name: jnlp-5n7x4
ID: X54B:QFRO:NKMQ:YJMW:NEVU:QU2A:VDHC:RJBI:M3YQ:KUU6:C4N7:IXNN
Docker Root Dir: /home/jenkins/.local/share/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
Thanks in advance
I was able to resolve the issue by using a different default runtime for docker.
#update default runtime
RUN wget -O crun https://github.com/containers/crun/releases/download/0.13/crun-0.13-static-x86_64 \
&& cp crun /usr/local/bin \
&& chmod a+x /usr/local/bin/crun \
&& chown -R rootless:rootless /usr/local/bin/crun
the service is then started with supervisor, config file shown:
[program:docker]
command=/home/rootless/bin/dockerd-rootless.sh --experimental --default-
runtime crun --add-runtime crun=/usr/local/bin/crun --storage-driver vfs
autorestart=true
user=rootless
detailed discussion here:
https://github.com/moby/moby/issues/40068
I'm experimenting with forcing a container to use more memory than it's allowed but I can't get it to work. The container is part of a stack defined with docker compose and it's deployed to docker in swarm mode.
Docker is allowing the container to go way above the 50M limit I've set. I was expecting docker to kill the container, throw an error, etc.
Can anyone help me on why Docker does not enforce the memory limit here?
The container in docker-compose.yml is defined to have a memory limit of 50M, and then I have setup a very simple PHP test which will try to allocate 200M. I've defined PHP mem limit to 128M.
This is my docker-compose.yml
version: "3"
services:
nginx:
image: nginx:latest
restart: unless-stopped
volumes:
- ./deploy/nginx/nginx.conf:/etc/nginx/nginx.conf
- ./public:/usr/share/nginx/html
ports:
- "8180:80"
links:
- app
app:
image: 127.0.0.1:5000/wpdemo
build:
context: .
dockerfile: Dockerfile-app
restart: unless-stopped
volumes:
- .:/var/www/html
links:
- mysql
deploy:
resources:
limits:
cpus: '0.50'
memory: 50M
reservations:
cpus: '0.25'
memory: 20M
mysql:
image: mysql:5.7
restart: unless-stopped
ports:
- "13306:3306"
environment:
MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
volumes:
- ~/docker/volumes/mysql:/var/lib/mysql
Instead of docker killing the container, it allows it to take as much memory as it wants and PHP eventually stops the process throwing the error below:
"PHP message: PHP Fatal error: Allowed memory size of 125829120 bytes exhausted (tried to allocate 67108872 bytes) in /var/www/html/public/index.php on line 4"
I'm using Ubuntu 18.04.
uname -a
Linux 4.18.10-041810-generic #201809260332 SMP Wed Sep 26 07:34:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Docker version 18.06.1-ce, build e68fc7a
docker-compose version 1.17.1, build unknown docker-py version: 2.5.1
CPython version: 2.7.15rc1 OpenSSL version: OpenSSL 1.1.0g 2 Nov 2017
This is the output of "docker stats" on the app container:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
679c8495ac1d stackdemo_app.1.hr3ufwlskhdafre39aqrshxyu 0.00% 43.81MiB / 50MiB 87.62% 106kB / 389kB 2.05GB / 10.6GB 5
This is the output of "docker info":
Containers: 36
Running: 5
Paused: 0
Stopped: 31
Images: 450
Server Version: 18.06.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: wnegv5lp41wfs3epfrua489or
Is Manager: true
ClusterID: hq7o176yffjglxzb9pu3fiomr
Managers: 1
Nodes: 1
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 192.168.1.120
Manager Addresses:
192.168.1.120:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.18.10-041810-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.49GiB
Name: rafxps15
ID: QEX7:FEB3:J76L:DCAQ:SO4S:SWVE:4XPI:PI6R:YM4C:MV4I:C3PM:FLOQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
As you said in comment, swap is enabled on host but swap limit in cgroups does not supported yet.
According to this enabling swap limit support. Note that reboot of system is essential.
At last, —-memory-swap flag should be set. If you want to prevent your PHP app accessing swap, you should set it with the same value of —-memory. More details about memory swap settings.
Command of running service:
docker service create -d \
-p 8080:8080 \
--mount type=bind,source=/etc/localtime,target=/etc/localtime \
--mount type=bind,source=/etc/timezone,target=/etc/timezone \
--mount type=bind,source=/home/test/docker/manager,target=/root \
--network test-network \
--workdir /root \
--name test-manager \
--replicas 2 \
--limit-cpu 2 \
--limit-memory 4G \
java:8 java -Dspring.profiles.active=$PROFILE -jar -Xms512m -Xmx4096m /root/target/test-manager.jar
After service started, I tested service with curl 192.168.2.48:8080/info, 50% of the requests are not working. entered containers with docker exec -it xxx bash, used curl 10.0.1.6:8080/info and curl 10.0.1.7:8080/info, found all result was ok.
But if I restart the above service several times, Sometimes, all requests are working completely.
Network check
nc -vuz 192.168.2.48 4789
nc -vz 192.168.2.48 2377
nc -vuz 192.168.2.48 7946
nc -vz 192.168.2.48 7946
All are succeeded.
Results of docker info:
Containers: 4
Running: 3
Paused: 0
Stopped: 1
Images: 25
Server Version: 17.06.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 102
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: tdc32kn0n6bwcz32ljvprcmq0
Is Manager: true
ClusterID: hdyakushxu1c6rsk2cml7b0l3
Managers: 2
Nodes: 2
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Root Rotation In Progress: false
Node Address: 192.168.2.47
Manager Addresses:
192.168.2.47:2377
192.168.2.48:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-92-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.78GiB
Name: ubuntu-qgsp01
ID: RP4U:E3PW:AU5R:BLD2:2QDL:DA25:GY2P:YV67:IR2F:GEBZ:XVX3:XC72
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Registry Mirrors:
https://20agqwyc.mirror.aliyuncs.com/
Live Restore Enabled: false
I suspect the problem caused by network load balance or vip, a node cannot be reached, so the service was hanged. But if I ping another in a container, I find the network is working. I feel puzzled.
This problem has been bothering me for a long time，hope someone can help me.
I have a Julia program which loads 8 GB of data. It is working fine in my local machine.
But when I try it in docker container it is not loading data and giving bus error. It is working fine with small data like 20 MB in docker container.
Dockerfile
FROM ubuntu:16.04
WORKDIR /julia
RUN apt-get -y update
RUN apt-get -y install unzip
RUN apt-get -y install cmake
RUN apt-get -y install clang
RUN apt-get -y install wget
RUN cd /tmp/
RUN wget "https://julialang.s3.amazonaws.com/bin/linux/x64/0.5/julia-0.5.0-linux-x86_64.tar.gz"
RUN tar -xzvf julia-0.5.0-linux-x86_64.tar.gz
RUN mv julia-3c9d75391c/ ~/julia
ENV PATH="/root/julia/bin:${PATH}"
RUN julia --eval 'Pkg.add("JSON")'
RUN julia --eval 'Pkg.add("HttpServer")'
RUN julia --eval 'Pkg.add("URIParser")'
RUN julia --eval 'Pkg.clone("https://github.com/deep-compute/AdaGram.jl.git")'
RUN julia --eval 'Pkg.build("AdaGram")'
CMD ["julia", "/tmp/adagram_server.jl", "80", "/julia/full.embed"]
docker-compose.yml
version: "3.1"
services:
julia:
image: ramidavalapati/julia:v-1
volumes:
- /home/ram/adagram_data/40MBfull.embed:/julia/full.embed
ports:
- 8080:80
command: ["sleep", "1h"]
Next I am doing
sudo docker-compose up
sudo docker exec -it $(sudo docker-compose ps -q julia) bash
root#3748d5958f77:/julia# julia
julia> using AdaGram
julia> AdaGram.load_model("/julia/full.embed")
Error
signal (7): Bus error
while loading no file, in expression starting on line 0
macro expansion at ./cartesian.jl:62 [inlined]
macro expansion at ./multidimensional.jl:429 [inlined]
_unsafe_batchsetindex! at ./multidimensional.jl:421
_setindex! at ./multidimensional.jl:370 [inlined]
setindex! at ./abstractarray.jl:832 [inlined]
#9 at /root/.julia/v0.5/AdaGram/src/AdaGram.jl:64
#600 at ./multi.jl:1030
run_work_thunk at ./multi.jl:1001
run_work_thunk at ./multi.jl:1010 [inlined]
#597 at ./event.jl:68
unknown function (ip: 0x7fe1822db16f)
jl_call_method_internal at /home/centos/buildbot/slave/package_tarball64/build/src/julia_internal.h:189 [inlined]
jl_apply_generic at /home/centos/buildbot/slave/package_tarball64/build/src/gf.c:1942
jl_apply at /home/centos/buildbot/slave/package_tarball64/build/src/julia.h:1392 [inlined]
start_task at /home/centos/buildbot/slave/package_tarball64/build/src/task.c:253
unknown function (ip: 0xffffffffffffffff)
Allocations: 9661042 (Pool: 9659980; Big: 1062); GC: 16
Bus error (core dumped)
Docker version
Client:
Version: 17.09.0-ce
API version: 1.32
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:42:18 2017
OS/Arch: linux/amd64
Server:
Version: 17.09.0-ce
API version: 1.32 (minimum version 1.12)
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:40:56 2017
OS/Arch: linux/amd64
Experimental: false
Docker info
Containers: 24
Running: 0
Paused: 0
Stopped: 24
Images: 24
Server Version: 17.09.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: jlkmigmtyjhz6yvi3zuvkobu7
Is Manager: true
ClusterID: rqt03ulgvvnym235m1qm8vd17
Managers: 4
Nodes: 15
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: X.X.X.X
Manager Addresses:
X.X.X.X:2377
X.X.X.X:2377
X.X.X.X:2377
X.X.X.X:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.10.0-35-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 251.8GiB
Name: ram
ID: 3OGG:275C:Q3IW:O4HX:DPLP:DPI3:5TIT:AG5J:EDMK:7NK3:L4UZ:BTQH
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: ramidavalapati
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Thanks in advance for any kind of help
link to 40MBfull.embed file
The problem is, there is no sufficient shared memory in docker container(default is 64 MB).
The problem was solved by giving the option --shm-size while running docker image.
Run : sudo docker run --shm-size 1G sample:latest
docker-compose file
version: "3.1"
services:
julia:
image: ramidavalapati/julia:v-1
shm_size: 1g
volumes:
- /home/ram/adagram_data/40MBfull.embed:/julia/full.embed
ports:
- 8080:80
command: ["sleep", "1h"]
If we want to work in swarm mode, we need to refer shared memory in volume section.
version: "3.3"
services:
julia:
image: ramidavalapati/julia:v-1
volumes:
- /home/ram/adagram_data/40MBfull.embed:/julia/full.embed
- /dev/shm:/dev/shm
ports:
- 8080:80
command: ["sleep", "1h"]
Here container uses shared memory of the host in which it is running.
I have been playing around with docker-in-docker (dind) setups and am running into a weird problem.
If I run a docker container separately inside dind and expose a port then I could connect to the port without any problems. For example, using the docker swarm visualizer inside dind:
/home/dockremap # docker run -d -p 8080:8080 dockersamples/visualizer:stable
/home/dockremap # wget localhost:8080
Connecting to localhost:8080 (127.0.0.1:8080)
index.html 100% |*********************** ....
However, if I run the same inside a swarm by deploying from a compose file it doesn't work.
Here is what my compose file looks like:
version: "3"
services:
visualizer:
image: dockersamples/visualizer:stable
ports:
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
deploy:
placement:
constraints: [node.role == manager]
networks:
- webnet
networks:
webnet:
and the commands I run:
/home/dockremap # docker swarm init
/home/dockremap # docker stack deploy -c compose.yaml test
now when I do wget I get connection refused error:
/home/dockremap # wget localhost:8080
Connecting to localhost:8080 (127.0.0.1:8080)
wget: can't connect to remote host (127.0.0.1): Connection refused
Should doing this sort of thing in dind be able to work by default, or is there something I need to configure? I am using docker 17.03.1-ce on Windows and here is what I get when I run docker info in dind:
Containers: 2
Running: 1
Paused: 0
Stopped: 1
Images: 1
Server Version: 17.05.0-ce
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: active
NodeID: wz2r6iuyqztg3ivyk9fwsn976
Is Manager: true
ClusterID: mshadtrs0b1oayva2vrquf67d
Managers: 1
Nodes: 1
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Node Address: 172.17.0.2
Manager Addresses:
172.17.0.2:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.4.59-boot2docker
Operating System: Alpine Linux v3.5 (containerized)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 987.1MiB
Name: 7e480e7313ae
ID: EL7P:NI2I:TOR4:I7IW:DPAB:WKYU:6A6J:NCC7:3K3E:6YVH:PYVB:2L2W
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled