I am trying to start docker containers, Seems like it has failed. After Analyzing DockerD logs.
ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Mar 13 21:37:58 ubuntu-nuc2 dockerd[692]: time="2020-03-13T21:37:58.277230495Z" level=error msg="stream copy error: reading from a closed fifo"
Mar 13 21:37:58 ubuntu-nuc2 dockerd[692]: time="2020-03-13T21:37:58.277625568Z" level=error msg="stream copy error: reading from a closed fifo"
Mar 13 21:37:58 ubuntu-nuc2 dockerd[692]: time="2020-03-13T21:37:58.364681832Z" level=error msg="54dd3a9bfcbcbcdddcc82350e8045985d8c8495ac465e7e335c3a91b75bdf30b cleanup: failed to delete container from containerd: no such container"
Mar 13 21:37:58 ubuntu-nuc2 dockerd[692]: time="2020-03-13T21:37:58.364824367Z" level=error msg="Handler for POST /v1.25/containers/54dd3a9bfcbcbcdddcc82350e8045985d8c8495ac465e7e335c3a91b75bdf30b/start returned error: OCI runtime create failed: container_linux.go:349: starting container process caused \"process_linux.go:449: container init caused \\\"rootfs_linux.go:109: jailing process inside rootfs caused \\\\\\\"pivot_root invalid argument\\\\\\\"\\\"\": unknown"
Mar 13 21:37:59 ubuntu-nuc2 dockerd[692]: time="2020-03-13T21:37:59.159258456Z" level=error msg="8292a9a1eb9fccfe8848c57f3aaaf2c52c557c5b55b606112cdf66148fbba6d7 cleanup: failed to delete container from containerd: no such container"
Mar 13 21:37:59 ubuntu-nuc2 dockerd[692]: time="2020-03-13T21:37:59.159376298Z" level=error msg="Handler for POST /v1.25/containers/8292a9a1eb9fccfe8848c57f3aaaf2c52c557c5b55b606112cdf66148fbba6d7/start returned error: cgroups: cgroup mountpoint does not exist: unknown"
Mar 13 22:14:22 ubuntu-nuc2 dockerd[692]: time="2020-03-13T22:14:22.094213684Z" level=error msg="Error setting up exec command in container rwo_gluster-server_1: Container 54dd3a9bfcbcbcdddcc82350e8045985d8c8495ac465e7e335c3a91b75bdf30b is not running"
-- Reboot --
Mar 13 22:24:23 ubuntu-nuc2 systemd[1]: Starting Docker Application Container Engine...
I went through several threads for help and got this.
export DOCKER_RAMDISK=true
But not able to figure out what is happening, Why Containers failed due to pivot_root error?
Docker-compose.yml:
gluster-server:
image: edge/glusterfs-rest:7
entrypoint: dumb-init glusterd --no-daemon --pid-file=/var/run/glusterd.pid
restart: unless-stopped
privileged: true
ipc: host
network_mode: host
# pid: host
userns_mode: host
volumes:
- gluster_database_vol:/var/lib/glusterd
- docker_pid_vol:/var/run
- /mnt:/mnt:shared
- /dev:/dev
- /opt:/opt
- /sys/fs/cgroup:/sys/fs/cgroup:ro
- /var/log/glusterfs:/var/log/glusterfs:shared
- /etc/ssl:/etc/ssl
depends_on:
- app-docker
- gluster-rest
- gluster-security-enabler
Related
I'm trying to pull an image off GCR using docker, using:
docker pull eu.gcr.io/<project>/<image>:<tag>
However I keep getting the error:
failed to register layer: ApplyLayer exit status 1 stdout: stderr: Error cleaning up after pivot: remove /.pivot_root347438920: device or resource busy
After receiving the error all calls to docker returns:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
The output of systemctl status docker is:
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2020-08-18 10:38:26 CEST; 14min ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 985 (dockerd)
Tasks: 42
Memory: 175.0M
CGroup: /system.slice/docker.service
└─985 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Aug 18 10:38:25 <my_name> dockerd[985]: time="2020-08-18T10:38:25.329780734+02:00" level=info msg="Loading containers: start."
Aug 18 10:38:26 <my_name> dockerd[985]: time="2020-08-18T10:38:26.093178325+02:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Aug 18 10:38:26 <my_name> dockerd[985]: time="2020-08-18T10:38:26.219186657+02:00" level=info msg="Loading containers: done."
Aug 18 10:38:26 <my_name> dockerd[985]: time="2020-08-18T10:38:26.333584778+02:00" level=info msg="Docker daemon" commit=48a66213fe graphdriver(s)=devicemapper version=19.03.12-ce
Aug 18 10:38:26 <my_name> dockerd[985]: time="2020-08-18T10:38:26.334733161+02:00" level=info msg="Daemon has completed initialization" Aug 18 10:38:26 emil-pc dockerd[985]: time="2020-08-18T10:38:26.389239911+02:00" level=info msg="API listen on /run/docker.sock"
Aug 18 10:38:26 <my_name> systemd[1]: Started Docker Application Container Engine.
Aug 18 10:46:05 <my_name> dockerd[985]: time="2020-08-18T10:46:05.342092783+02:00" level=info msg="Attempting next endpoint for pull after error: failed to register layer: ApplyLayer exit status 1 stdout: stderr: Error cleaning up after pivot: remove /.pivot_root557795275: device or resource busy"
I've tried restarting docker but to no avail. Any help is much appriciated.
we have docker-compose.yml that contain configuration for Kafka , zookeeper and schema registry
when we start the docker compose we get the following errors
docker-compose up -d
Starting kafka-docker-final_zookeeper3_1 ... error
ERROR: for kafka-docker-final_zookeeper3_1 Cannot start service zookeeper3: network dd321821f3cb4a715c31e04b32bff2cf206c85ed5581b01b1c6a94ffa45f330e not found
ERROR: for zookeeper3 Cannot start service zookeeper3: network dd321821f3cb4a715c31e04b32bff2cf206c85ed5581b01b1c6a94ffa45f330e not found
ERROR: Encountered errors while bringing up the project.
and
systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2020-03-19 07:57:29 UTC; 1h 55min ago
Docs: https://docs.docker.com
Main PID: 12105 (dockerd)
Tasks: 30
Memory: 654.6M
CGroup: /system.slice/docker.service
└─12105 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Mar 19 07:57:29 master3 dockerd[12105]: time="2020-03-19T07:57:29.610005717Z" level=info msg="Daemon has completed initialization"
Mar 19 07:57:29 master3 dockerd[12105]: time="2020-03-19T07:57:29.631338594Z" level=info msg="API listen on /var/run/docker.sock"
Mar 19 07:57:29 master3 systemd[1]: Started Docker Application Container Engine.
Mar 19 07:58:12 master3 dockerd[12105]: time="2020-03-19T07:58:12.352833676Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: net/http: re...ng headers)"
Mar 19 07:58:12 master3 dockerd[12105]: time="2020-03-19T07:58:12.352916724Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/...ng headers)"
Mar 19 07:58:12 master3 dockerd[12105]: time="2020-03-19T07:58:12.353019409Z" level=error msg="Handler for POST /v1.22/images/create returned error: Get https://registry-1.do...ng headers)"
Mar 19 08:03:47 master3 dockerd[12105]: time="2020-03-19T08:03:47.255058871Z" level=warning msg="error locating sandbox id 20ce3c5b6383ad92dae848c3de1d91bbfff9306ca86fdc90fae...c not found"
Mar 19 08:03:47 master3 dockerd[12105]: time="2020-03-19T08:03:47.263976715Z" level=error msg="ef808aa411ae0aaef0920397c77b6d9a327bdd1651877402fe1fc142a513af8a cleanup: faile...h container"
Mar 19 09:50:43 master3 dockerd[12105]: time="2020-03-19T09:50:43.920457464Z" level=warning msg="error locating sandbox id 20ce3c5b6383ad92dae848c3de1d91bbfff9306ca86fdc90fae...c not found"
Mar 19 09:50:43 master3 dockerd[12105]: time="2020-03-19T09:50:43.927744636Z" level=error msg="ef808aa411ae0aaef0920397c77b6d9a327bdd1651877402fe1fc142a513af8a cleanup: faile...h container"
Hint: Some lines were ellipsized, use -l to show in full.
regarding to
Cannot start service zookeeper3: network dd321821f3cb4a715c31e04b32bff2cf206c85ed5581b01b1c6a94ffa45f330e not found
Cannot start service zookeeper3: network dd321821f3cb4a715c31e04b32bff2cf206c85ed5581b01b1c6a94ffa45f330e not found
how to fix this issue?
docker container ls -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6c729cb0bb2c confluentinc/cp-schema-registry:latest "/etc/confluent/dock…" 3 months ago Exited (255) 2 hours ago 0.0.0.0:8081->8081/tcp kafka-docker-schemaregistry_1
ef808aa411ae confluentinc/cp-zookeeper:latest "/etc/confluent/dock…" 3 months ago Exited (255) 2 hours ago
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
docker network ls
NETWORK ID NAME DRIVER SCOPE
e5566ab8ca6d bridge bridge local
2467d9664593 host host local
c509e32d0d67 kafka-docker-default bridge local
08966157382c none null local
we fixed the issue by the following procedure
# docker container ls -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6c729cb0bb2c confluentinc/cp-schema-registry:latest "/etc/confluent/dock…" 3 months ago Exited (255) 5 hours ago 0.0.0.0:8081->8081/tcp kafka-docker-schemaregistry_1
ef808aa411ae confluentinc/cp-zookeeper:latest "/etc/confluent/dock…" 3 months ago Exited (255) 5 hours ago kafka-docker-zookeeper3_1
# docker container rm 6c729cb0bb2c
# docker container rm ef808aa411ae
systemctl stop docker
systemctl start docker
docker-compose up -d
Creating kafka-docker-zookeeper3_1 ... done
Creating kafka-docker-kafka3_1 ... done
Creating kafka-docker-schemaregistry_1 ... done
docker-compose ps
Name Command State Ports
------------------------------------------------------------------------------------------------------------------------------------------------
kafka-docker-kafka3_1 /etc/confluent/docker/run Up 0.0.0.0:9092->9092/tcp
kafka-docker-schemaregistry_1 /etc/confluent/docker/run Up 0.0.0.0:8081->8081/tcp
kafka-docker-zookeeper3_1 /etc/confluent/docker/run Up 0.0.0.0:2181->2181/tcp, 0.0.0.0:2888->2888/tcp, 0.0.0.0:3888->3888/tcp
I had the same problem and for me it was enough to:
sudo docker-compose down
sudo docker-compose up
When I start docker:
service docker start
Shows this log:
Sep 15 02:05:16 iZuf63refzweg1d9dh94t9Z dockerd[13274]: time="2019-09-15T02:05:16.969535655+08:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Sep 15 02:05:16 iZuf63refzweg1d9dh94t9Z dockerd[13274]: time="2019-09-15T02:05:16.970807372+08:00" level=info msg="Loading containers: start."
Sep 15 02:05:16 iZuf63refzweg1d9dh94t9Z dockerd[13274]: time="2019-09-15T02:05:16.972484550+08:00" level=error msg="Failed to load container mount 022183ecc93800ee6af20bfa8a59ec3746568c73abbea29e3d305c46b468990c: mount does not exist"
Sep 15 02:05:16 iZuf63refzweg1d9dh94t9Z dockerd[13274]: time="2019-09-15T02:05:16.972792385+08:00" level=error msg="Failed to load container mount 05073c615ef179888e689414348f026116a206987ae4e66cff04fa85435e1e70: mount does not exist"
Sep 15 02:05:16 iZuf63refzweg1d9dh94t9Z dockerd[13274]: time="2019-09-15T02:05:16.973235355+08:00" level=error msg="Failed to load container mount 0857413708fdc210726d664b3c312fc65ff40448db177d9d2af545a13f91e48c: mount does not exist"
Sep 15 02:05:16 iZuf63refzweg1d9dh94t9Z dockerd[13274]: time="2019-09-15T02:05:16.973572688+08:00" level=error msg="Failed to load container mount 117c3e16388063b1741fb584f10bb2f7e09508c844b661e3cb628a836094f011: mount does not exist"
Sep 15 02:05:16 iZuf63refzweg1d9dh94t9Z dockerd[13274]: time="2019-09-15T02:05:16.973619897+08:00" level=error msg="Failed to load container 16831d8345c6ade3f96544a2bd95020841933480b4a9c96ae5d5f36a17eb310e: open /var/lib/docker/containers/16831d8345c6ade3f96544a2bd95020841933480b4a9c96ae5d5f36a17eb310e/config.v2.json: no such file or directory"
Sep 15 02:05:17 iZuf63refzweg1d9dh94t9Z dockerd[13274]: time="2019-09-15T02:05:17.167060906+08:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
docker info shows root path:
Docker Root Dir: /var/lib/docker
and container shows:
[root#iZuf63refzweg1d9dh94t9Z containers]# ls
022183ecc93800ee6af20bfa8a59ec3746568c73abbea29e3d305c46b468990c 117c3e16388063b1741fb584f10bb2f7e09508c844b661e3cb628a836094f011
05073c615ef179888e689414348f026116a206987ae4e66cff04fa85435e1e70 16831d8345c6ade3f96544a2bd95020841933480b4a9c96ae5d5f36a17eb310e
0857413708fdc210726d664b3c312fc65ff40448db177d9d2af545a13f91e48c
why the docker still do not find container?
Suddently my docker daemon stop and never turned on again. I'm running docker on a Linux raspberrypi 4.1.13-v7+. It worked before until last week when my docker service suddenly stop working and I don't have a clue why.
My docker version is:
raspberrypi:~ $ docker version
Client:
Version: 1.10.3
API version: 1.22
Go version: go1.4.3
Git commit: 20f81dd
Built: Thu Mar 10 22:23:48 2016
OS/Arch: linux/arm
An error occurred trying to connect: Get http:///var/run/docker.sock/v1.22/version: read unix /var/run/docker.sock: connection reset by peer
Socket is ok:
● docker.socket - Docker Socket for the API
Loaded: loaded (/lib/systemd/system/docker.socket; disabled)
Active: active (listening) since Sat 2018-03-17 00:42:46 UTC; 6s ago
Listen: /var/run/docker.sock (Stream)
Looking to my service status you can see the following log:
docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled)
Active: failed (Result: start-limit) since Sat 2018-03-17 00:05:52 UTC; 4min 55s ago
Docs: https://docs.docker.com
Process: 2891 ExecStart=/usr/bin/docker daemon -H fd:// $DOCKER_OPTS (code=exited, status=1/FAILURE)
Main PID: 2891 (code=exited, status=1/FAILURE)
Mar 17 00:05:52 raspberrypi docker[2891]: time="2018-03-17T00:05:52.743474604Z" level=debug msg="ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.1)"
Mar 17 00:05:52 raspberrypi docker[2891]: time="2018-03-17T00:05:52.758090386Z" level=debug msg="ReleasePool(LocalDefault/172.17.0.0/16)"
Mar 17 00:05:52 raspberrypi docker[2891]: time="2018-03-17T00:05:52.772819345Z" level=debug msg="Cleaning up old shm/mqueue mounts: start."
Mar 17 00:05:52 raspberrypi docker[2891]: time="2018-03-17T00:05:52.773269239Z" level=fatal msg="Error starting daemon: Error initializing network controller: Error creating default \"bridge\" network: package not installed"
Mar 17 00:05:52 raspberrypi systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Mar 17 00:05:52 raspberrypi systemd[1]: Failed to start Docker Application Container Engine.
I already tried this solution but for me it didn't work.
How can I make my docker service start again? It seems that a package is not installed but I tried to:
raspberrypi:~ $ modprobe bridge
modprobe: FATAL: Module bridge not found.
I've installed Docker on Centos 7 using the yum process. Below is the output for systemctl status docker.service -l:
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2018-01-02 14:11:09 GMT; 1h 15min ago
Docs: https://docs.docker.com
Main PID: 136537 (dockerd)
Memory: 71.3M
CGroup: /system.slice/docker.service
├─136537 /usr/bin/dockerd
└─136556 docker-containerd --config /var/run/docker/containerd/containerd.toml
Jan 02 15:00:33 colossus dockerd[136537]: time="2018-01-02T15:00:33.157175048Z" level=error msg="c60a6261b7381f5739fa11e6df8b4219d8db94e766c6e4c8c4c2d45a44ee03a5 cleanup: failed to delete container from containerd: no such container"
Jan 02 15:00:33 colossus dockerd[136537]: time="2018-01-02T15:00:33.157230927Z" level=error msg="Handler for POST /v1.35/containers/c60a6261b7381f5739fa11e6df8b4219d8db94e766c6e4c8c4c2d45a44ee03a5/start returned error: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/c60a6261b7381f5739fa11e6df8b4219d8db94e766c6e4c8c4c2d45a44ee03a5/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44.837926048Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f/shim.sock" debug=false module="containerd/tasks" pid=145298
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44Z" level=info msg="shim reaped" id=3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f module="containerd/tasks"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44.899560066Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44.899585062Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44.921068216Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/delete type="*events.ContainerDelete"
Jan 02 15:09:45 colossus dockerd[136537]: time="2018-01-02T15:09:45.309201262Z" level=error msg="3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f cleanup: failed to delete container from containerd: no such container"
Jan 02 15:09:45 colossus dockerd[136537]: time="2018-01-02T15:09:45.309254679Z" level=error msg="Handler for POST /v1.35/containers/3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f/start returned error: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown"
The error i get (sudo docker run hello-world) is:
docker: Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown.
ERRO[0001] error waiting for container: context canceled
docker --version
Docker version 17.12.0-ce, build c97c6d6
and here is the docker info
Containers: 27
Running: 0
Paused: 0
Stopped: 27
Images: 47
Server Version: 17.12.0-ce
Storage Driver: devicemapper
Pool Name: docker-253:0-135160107-pool
Pool Blocksize: 65.54kB
Base Device Size: 10.74GB
Backing Filesystem: xfs
Udev Sync Supported: true
Data file: /dev/loop0
Metadata file: /dev/loop1
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 14.4GB
Data Space Total: 107.4GB
Data Space Available: 25.82GB
Metadata Space Used: 12.6MB
Metadata Space Total: 2.147GB
Metadata Space Available: 2.135GB
Thin Pool Minimum Free Space: 10.74GB
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: N/A (expected: b2567b37d7b75eb4cf325b77297b140ea686ce8f)
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-327.36.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 88
Total Memory: 125.6GiB
Name: colossus
ID: HV7I:KP2J:N6SQ:SAEP:MWSO:O5T3:Y2PE:L6TL:2D6L:IHVD:SQ3E:TM5P
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
You most likely have an old package installed that is breaking runc. Make sure to upgrade your operating system to the current release (for CentOS, this is 1708, or the RHEL equivalent of 7.4). Others have reported similar issues related to packages being out of date.
I had 2 systems with identical CentOS 7.2 versions and kernel. Had this problem with one but not the other. Comparing package versions showed a difference in the version of containerd.io -- I downgraded that on the non-working version and it worked:
$ sudo yum downgrade containerd.io-1.2.0-3.el7
Removed:
containerd.io.x86_64 0:1.2.4-3.1.el7
Installed:
containerd.io.x86_64 0:1.2.0-3.el7