When I want to rerun container with another volumes or update image.
I stop and try to remove container, but often geterror on rm command
# docker rm containername
Error response from daemon: Driver devicemapper failed to remove root filesystem dbe6....f91f: Device is Busy
I need to restart docker daemon to remove container.
~ # docker version root#CentOS-72-64-minimal
Client:
Version: 1.12.5
API version: 1.24
Go version: go1.6.4
Git commit: 7392c3b
Built: Fri Dec 16 02:23:59 2016
OS/Arch: linux/amd64
Server:
Version: 1.12.5
API version: 1.24
Go version: go1.6.4
Git commit: 7392c3b
Built: Fri Dec 16 02:23:59 2016
OS/Arch: linux/amd64
------------------------------------------------------------
~ # docker info root#CentOS-72-64-minimal
Containers: 40
Running: 11
Paused: 0
Stopped: 29
Images: 32
Server Version: 1.12.5
Storage Driver: devicemapper
Pool Name: docker-8:3-28705145-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 14.83 GB
Data Space Total: 107.4 GB
Data Space Available: 92.54 GB
Metadata Space Used: 21.15 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.126 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.135-RHEL7 (2016-09-28)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge overlay host null
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-514.2.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 30.96 GiB
Name: CentOS-72-64-minimal
ID: SMTY:72HJ:5QIS:AT63:6GPI:U2UQ:KUYY:C7M6:UIOY:37AR:JS53:JAGA
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8
I've been experiencing this issue quite a bit on a Red Hat host. The fix according to the reported issue on this is to upgrade to a newer kernel. As a workaround for places where that's not an option, I've been using docker rm -f ... which still throws the error but the container does get cleaned up. Much quicker and less intrusive than a restart of the daemon.
I experienced the same problem, service docker restart almost always fixes the issue.(restarting docker service)
More information about this issue availbale here.
Related
I'm getting this error when pulling some docker images (but not all):
failed to register layer: Error processing tar file(exit status 1): operation not permitted
For example: docker pull nginx works, but not docker pull redis.
I get the same result wether i run the command with a user that is part of the docker group, using sudo or as root.
If i run dockerd in debug mode i see this in the logs:
DEBU[0025] Downloaded 5233d9aed181 to tempfile /var/lib/docker/tmp/GetImageBlob023191751
DEBU[0025] Applying tar in /var/lib/docker/overlay2/e5290b8c50d601918458c912d937a4f6d4801ecaa90afb3b729a5dc0fc405afc/diff
DEBU[0027] Applied tar sha256:16ada34affd41b053ca08a51a3ca92a1a63379c1b04e5bbe59ef27c9af98e5c6 to e5290b8c50d601918458c912d937a4f6d4801ecaa90afb3b729a5dc0fc405afc, size: 79185732
(...)
DEBU[0029] Applying tar in /var/lib/docker/overlay2/c5c0cfb9907a591dc57b1b7ba0e99ae48d0d7309d96d80861d499504af94b21d/diff
DEBU[0029] Cleaning up layer c5c0cfb9907a591dc57b1b7ba0e99ae48d0d7309d96d80861d499504af94b21d: Error processing tar file(exit status 1): operation not permitted
INFO[0029] Attempting next endpoint for pull after error: failed to register layer: Error processing tar file(exit status 1): operation not permitted
INFO[0029] Layer sha256:938f1cd4eae26ed4fc51c37fa2f7b358418b6bd59c906119e0816ff74a934052 cleaned up
(...)
If i run watch -n 0 "sudo ls -lt /var/lib/docker/overlay2/" while the image is pulling, i can see new folders appearing (and disappearing after it fails) and the permissions on /var/lib/docker/overlay2/ are root:root:700 so i don't think it's exactly a permission issue.
Here are some detail about the environment:
I have a proxmox running the LXC container where i'm having the issue.
The container itself is running Debian 8.
And here are the various versions:
$> uname -a
Linux [redacted-hostname] 4.10.15-1-pve #1 SMP PVE 4.10.15-15 (Fri, 23 Jun 2017 08:57:55 +0200) x86_64 GNU/Linux
$> docker version
Client:
Version: 17.06.0-ce
API version: 1.30
Go version: go1.8.3
Git commit: 02c1d87
Built: Fri Jun 23 21:20:04 2017
OS/Arch: linux/amd64
Server:
Version: 17.06.0-ce
API version: 1.30 (minimum version 1.12)
Go version: go1.8.3
Git commit: 02c1d87
Built: Fri Jun 23 21:18:59 2017
OS/Arch: linux/amd64
Experimental: false
$>docker info
Containers: 20
Running: 0
Paused: 0
Stopped: 20
Images: 28
Server Version: 17.06.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Kernel Version: 4.10.15-1-pve
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.906GiB
Name: resumed-dev
ID: EBJ6:AFVS:L3RC:ZEE7:A6ZJ:WDQE:GTIZ:RXHA:P4AQ:QJD7:H6GG:YIQB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 16
Goroutines: 24
System Time: 2017-08-17T14:17:07.800849127+02:00
EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
EDIT: This will be fixed by any release after December 18, 2017 of Moby via this merge. Will update again when fully incorporated into Docker.
If your container is unprivileged, this appears to be an issue with the overlay2 storage driver for Docker. This does not appear to be an issue with overlay (GitHub issue). So either utilize the overlay storage driver instead of overlay2, or make your container privileged.
I have almost the same environment as you, and met the same problem.
Some image works perfectly (alpine), while some images fails at cleaning up (ubuntu).
strace -f dockerd -D then docker pull or docker load gives the reason:
mknodat(AT_FDCWD, "/dev/agpgart", S_IFCHR|0660, makedev(10, 175)) = -1 EPERM (Operation not permitted)
Unprivileged container prohibit mknod by design. If you insist nesting Docker inside lxc, you will have to choose privileged container. (And notice that existing unprivileged container cannot be converted to privileged container directly due to uid/gid mapping)
I am trying to setup this https://github.com/jwasham/computer-science-flash-cards on my local pc using docker but after I have built my image,when I try
docker run -d -p 8000:8000 --name cs-flash-cards cs-flash-cards
it says
flag provided but not defined: -d
Any ideas how to fix this and run this container?
EDIT(on docker info;docker version I get the following info):
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 10
Server Version: 1.12.5
Storage Driver: devicemapper
Pool Name: docker-202:1-312980-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 877.5 MB
Data Space Total: 107.4 GB
Data Space Available: 2.019 GB
Metadata Space Used: 1.913 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.019 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.110 (2015-10-30)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-45-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 990.7 MiB
Name: ip-172-31-33-253
ID: QPUK:E7BB:Y2PW:MPJR:L2X4:4AMT:VHAT:SOXK:3A2N:UKI2:ZXRK:QF4S
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
Client:
Version: 1.12.5
API version: 1.24
Go version: go1.6.4
Git commit: 7392c3b
Built: Fri Dec 16 02:42:17 2016
OS/Arch: linux/amd64
Server:
Version: 1.12.5
API version: 1.24
Go version: go1.6.4
Git commit: 7392c3b
Built: Fri Dec 16 02:42:17 2016
OS/Arch: linux/amd64
Just for the next person who has the same problem.
this was for SystemD on Ubuntu LTS and the docker.service file
was the fix.
If you are on Centos7, you may have an extra file called override.conf
this is used to override the docker.service in systemd.
This file also has the -d flag
and is located in
/etc/systemd/system/docker.service.d/override.conf
replace -d with deamon here
do systemctl daemon-reload and docker will start up.
I'm new to docker. I've been trying it for less than two weeks. We have a service that we'd like to migrate into a container. The service makes use of about 50G worth of data, so we expect the image to be very large. We've written a Dockerfile for it. When we run the build it fails with the ff:
ApplyLayer exit status 1 stdout: stderr: write /mnt/spine_features/spine_features_subset.lmdb/data.mdb: input/output error
When we check docker ps -a for containers, we can see the build container listed with status:
Exited (1) About a minute ago
When we try to commit the container, we get the same error:
Error response from daemon: ApplyLayer exit status 1 stdout: stderr: write /mnt/spine_features/spine_features_subset.lmdb/data.mdb: input/output error
We can also docker inspect the container. When we exclude downloading the largest files, we are able to complete building the service image. Is there some sort of configuration we can change succeed while still including the larger files?
docker version
Client:
Version: 1.12.1
API version: 1.24
Go version: go1.7
Git commit: 23cf638
Built: Fri Aug 19 02:03:02 2016
OS/Arch: linux/amd64
Server:
Version: 1.12.1
API version: 1.24
Go version: go1.7
Git commit: 23cf638
Built: Fri Aug 19 02:03:02 2016
OS/Arch: linux/amd64
docker info
Containers: 2
Running: 0
Paused: 0
Stopped: 2
Images: 22
Server Version: 1.12.1
Storage Driver: devicemapper
Pool Name: docker-8:2-7603782-pool
Pool Blocksize: 65.54 kB
Base Device Size: 214.7 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 67.78 GB
Data Space Total: 107.4 GB
Data Space Available: 39.59 GB
Metadata Space Used: 37.04 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.11 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.134 (2016-09-07)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.7.4-1-ARCH
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 15.58 GiB
Name: mega-haro
ID: MDQ5:JIT3:BVQX:XYO6:YTXI:HTRE:N2UQ:ML4V:ENIE:DDCO:ZGYF:3P5F
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
misty:5000
127.0.0.0/8
As Haoming Zhang recommended, mounting the host directory into the container is an acceptable solution. We are also exploring the possibility of using FUSE to load the data into the container at runtime instead of baking it into the image during build or having the host pass it into the container when the container is run.
I was trying to install gitlab using docker containers and was able to bring up gitlab successfully using docker compose file from sameersbn.
However after few uninstalls and (docker rm ) reinstalls (docker-compose up) as part of CI testing, I started getting this weird error while running docker-compose up or docker run
[root#server.com ~]# docker run java
Unable to find image 'java:latest' locally
Pulling repository docker.io/library/java
docker: Error while pulling image: Get https://index.docker.io/v1/repositories/library/java/images: malformed MIME header line: Too Many Requests (HAP429)..
See 'docker run --help'.
I can't seem to be able to pull any of the docker containers using docker run or docker-compose.
Couldn't find much help online reg this issue.
As per the docker hub forum the issue https://forums.docker.com/t/429-too-many-requests-how-to-fix-this-isssue/3971/7 should disappear after an hour but I waited half a day without much luck!
Here are the details of my installation:
[root#server build]# docker version
Client:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built:
OS/Arch: linux/amd64
Server:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built:
OS/Arch: linux/amd64
[root#server build]# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 15
Server Version: 1.12.1
Storage Driver: devicemapper
Pool Name: docker-thinpool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 3.077 GB
Data Space Total: 61.2 GB
Data Space Available: 58.12 GB
Metadata Space Used: 1.204 MB
Metadata Space Total: 641.7 MB
Metadata Space Available: 640.5 MB
Thin Pool Minimum Free Space: 6.119 GB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Library Version: 1.02.107-RHEL7 (2015-10-14)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge null host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.64 GiB
Name: server.com
ID: SDFS:SDEF:GKY5:UKGK:QHWR:H4EC:wEFw:YVAS:JE2V:A5YB:FDSW
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 17
Goroutines: 23
System Time: 2016-10-09T18:34:43.969512367-05:00
EventsListeners: 0
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8
Any help would be much appreciated. I'm stuck with this error and can't proceed any further with my gitlab.
Thanks.
This may or may not be relevant to your situation, but I can report that I had the same error (didn't go away within an hour) and it was related to the fact that I was on a VPN to my office. I don't know if the VPN was the issue, or the NAT of my workplace, but when I turned off the VPN, the issue went away.
Note, I was running Docker for Windows (W7), so my circumstances are quite different from yours. But perhaps this answer will be useful to you or to anyone else looking for an answer.
Bottom line: If you are using a VPN, switch it off and try again. If you are inside a corporate filewall, try from outside.
If I kill my docker daemon process and then restart it, any containers that were running are now listed with Exited status and cannot be restarted using docker-compose as it will complain about the container name already being in use.
Docker containers can be started again using docker start but this could be hard when you've many containers.
Is there any way to restart the docker daemon, leave containers running (so as to not disrupt traffic) and have the daemon reattach to the containers?
uname -a:
Linux localhost.localdomain 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
docker info:
Containers: 23
Running: 2
Paused: 0
Stopped: 21
Images: 16
Server Version: 1.11.1
Storage Driver: devicemapper
Pool Name: docker-253:0-1567975-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 3.738 GB
Data Space Total: 107.4 GB
Data Space Available: 28 GB
Metadata Space Used: 7.688 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.14 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2015-10-14)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 7.64 GiB
Name:
ID:
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Username:
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Looks like this is something that will be addressed in 1.12:
https://github.com/docker/docker/issues/2658