I am unable to remove the dead container, it appears again after i restart the Docker service.
docker ps -a
CONTAINER ID STATUS
11667ef16239 Dead
Then
docker rm -f 11667ef16239
Then, when I ran the docker ps -a, no docker containers showing.
docker ps -a
CONTAINER ID STATUS
However, when I restart the docker service:
service docker restart
And run the docker ps -a again:
docker ps -a
CONTAINER ID STATUS
11667ef16239 Dead
You can also remove dead containers with this command
docker rm $(docker ps --all -q -f status=dead)
But, I'm really not sure why & how the dead containers are created. This error seems related https://github.com/typesafehub/mesos-spark-integration-tests/issues/34 whenever i get dead containers
[Update]
With Docker 1.13 update, we can easily remove both unwanted containers, dangling images
$ docker system df #will show used space, similar to the unix tool df
$ docker system prune # will remove all unused data.
Most likely, an error occurred when the daemon attempted to cleanup the container, and he is now stuck in this "zombie" state.
I'm afraid your only option here is to manually clean it up:
$ sudo rm -rf /var/lib/docker/<storage_driver>/11667ef16239.../
Where <storage_driver> is the name of your driver (aufs, overlay, btrfs, or devicemapper).
Actually things changed slightly these days in order to get rid of those dead containers you may try to unmount those blocked filesystems to release them
So if you get message like this
Error response from daemon: Cannot destroy container elated_wozniak: Driver devicemapper failed to remove root filesystem 656cfd09aee399c8ae8c8d3e735fe48d70be6672773616e15579c8de18e2a3b3: Device is Busy
just run this
umount /var/lib/docker/devicemapper/mnt/656cfd09aee399c8ae8c8d3e735fe48d70be6672773616e15579c8de18e2a3b3
and you can normally remove container after that
I had the following error when removing a dead container (docker 17.06.1-ce on CentOS 7):
Error response from daemon: driver "overlay" failed to remove root filesystem for <some-id>:
remove /var/lib/docker/overlay/<some-id>/merged: device or resource busy
Here is how I fixed it:
1. Check which other processes are also using docker resources
$ grep docker /proc/*/mountinfo
which outputs something like this, where the number after /proc/ is the pid:
/proc/10001/mountinfo:179...
/proc/10002/mountinfo:149...
/proc/12345/mountinfo:159 149 0:36 / /var/lib/docker/overlay/...
2. Check the process name of the above pid
$ ps -p 10001 -o comm=
dockerd
$ ps -p 10002 -o comm=
docker-containe
$ ps -p 12345 -o comm=
nginx <<<-- This is suspicious!!!
So, nginx with pid 12345 seems to also be using /var/lib/docker/overlay/..., which is why we cannot remove the related container and get the device or resource busy error. (See here for a discussion on how nginx shares the same mount namespace with docker containers thus prevents its deletion.)
3. Stop nginx and then I can remove the container successfully.
$ sudo service nginx stop
$ docker rm <container-id>
I got the same issue and both answers did not help.
What helped for me is just creating the directories that are missing and them remove them:
mkdir /var/lib/docker/devicemapper/mnt/656cfd09aee399c8ae8c8d3e735fe48d70be6672773616e15579c8de18e2a3b3
mkdir /var/lib/docker/devicemapper/mnt/656cfd09aee399c8ae8c8d3e735fe48d70be6672773616e15579c8de18e2a3b3-init
docker rm 656cfd09aee399c8ae8c8d3e735fe48d70be6672773616e15579c8de18e2a3b3
Removing container by force worked for me.
docker rm -f <id_of_the_dead_container>
Notes:
Be aware that this command might throw this error
Error response from daemon: Driver devicemapper failed to remove root filesystem <id_of_the_dead_container>: Device is Busy
The mount of your's dead container device mapper should be removed despite this message. That is, you will no longer access this path:
/var/lib/docker/devicemapper/mnt/<id_of_the_dead_container>
Tried all of the above (short of reboot/ restart docker).
So here is the error om docker rm:
$ docker rm 08d51aad0e74
Error response from daemon: driver "devicemapper" failed to remove root filesystem for 08d51aad0e74060f54bba36268386fe991eff74570e7ee29b7c4d74047d809aa: remove /var/lib/docker/devicemapper/mnt/670cdbd30a3627ae4801044d32a423284b540c5057002dd010186c69b6cc7eea: device or resource busy
Then I did a the following:
$ grep docker /proc/*/mountinfo | grep 958722d105f8586978361409c9d70aff17c0af3a1970cb3c2fb7908fe5a310ac
/proc/20416/mountinfo:629 574 253:15 / /var/lib/docker/devicemapper/mnt/958722d105f8586978361409c9d70aff17c0af3a1970cb3c2fb7908fe5a310ac rw,relatime shared:288 - xfs /dev/mapper/docker-253:5-786536-958722d105f8586978361409c9d70aff17c0af3a1970cb3c2fb7908fe5a310ac rw,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota
This got be the PID of the offending process keeping it busy - 20416 (the item after /proc/
So I did a ps -p and to my surprise find:
[devops#dp01app5030 SeGrid]$ ps -p 20416
PID TTY TIME CMD
20416 ? 00:00:19 ntpd
A true WTF moment. So I pair problem solved with Google and found this:
Then found this https://github.com/docker/for-linux/issues/124
Turns out I had to restart ntp daemon and that fixed the issue!!!
For Deleting all dead container
docker rm -f $(docker ps --all -q -f status=dead)
For deleting all exited container docker rm -f $(docker ps --all -q -f status=exited)
As I have -f is necessary
Try running the following commands. It always works for me.
# docker volume rm $(docker volume ls -qf dangling=true)
# docker rm $(docker ps -q -f 'status=exited')
After execution of the above commands, restart docker by,
# service docker restart
grep 656cfd09aee399c8ae8c8d3e735fe48d70be6672773616e15579c8de18e2a3b3 /proc/*/mountinfo
then find the pid of 656cfd09aee399c8ae8c8d3e735fe48d70be6672773616e15579c8de18e2a3b3and and kill it
I have tried the suggestions above but didn't work.
Then
I try : docker system prune -a, it didn't work the first time
I reboot the system
I try again docker system prune -a. This time it works. It will send a warning message and in the end ask "Are you sure you want to continue? y/n? . Ans:y . It will time a time and in the end the dead containers are gone.
Verify with docker ps -a
IMPORTANT - this is the nuclear option as it destroys all containers + images
for Windows:
del D:\ProgramData\docker\containers\{CONTAINER ID}
del D:\ProgramData\docker\windowsfilter\{CONTAINER ID}
Then restart the Docker Desktop
Try kill it and then remove >:) i.e.
docker kill $(docker ps -q)
Try, It worked for me:
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4f13b53be9dd 5b0bbf1173ea "/opt/app/netjet..." 5 months ago Dead appname_chess
$ docker rm $(docker ps --all -q -f status=dead)
Error response from daemon: driver "devicemapper" failed to remove root filesystem for 4f13b53be9ddef3e9ba281546aef1c544805282971f324291a1dc91b50eeb440: failed to remove device 487b4b73c58d19ef79201cf6d5fcd6b7316e612e99c14505a6bf24399cad9795-init: devicemapper: Error running DeleteDevice dm_task_run failed
su
cd /var/lib/docker/containers
[root#localhost containers]# ls -l
total 0
drwx------. 1 root root 312 Nov 17 08:58 4f13b53be9ddef3e9ba281546aef1c544805282971f324291a1dc91b50eeb440
[root#localhost containers]# rm -rf 4f13b53be9ddef3e9ba281546aef1c544805282971f324291a1dc91b50eeb440
systemctl restart docker
Try this it worked for me:
docker rm -f <container_name>
eg. docker rm -f 11667ef16239
There are a lot of answers in here but none of them involved the (quick) solution that worked for me.
I'm using Docker version 1.12.3, build 6b644ec.
I simply ran docker rmi <image-name> for the image from whence the dead container came. A docker ps -a then showed the dead container missing completely.
Then, of course, I just re-pulled the image and ran the container again.
I have no idea how it found itself in this state but so it is...
In my case, I had to remove it with
rm -r /var/lib/docker/containers/<container-id>/
and it worked. Maybe that's how you solve it in docker version ~19. My docker version was 19.03.12,
For future users I wanna share my experience: (2022)
To see all containers I ran
docker container ls -a
then to kill containers I ran:
docker rm -f YOUR_CONTAINER_ID
for example:
docker rm -f d389s8dffe
Running on Centos7 & Docker 1.8.2, I was unable to use Zgr3doo's solution to umount by devicemapper ( I think the response I got was that the volume wasn't mounted/found. )
I think I also had a similar thing happen with sk8terboi87 ツ 's answer: I believe the message was that the volumes couldn't be unmounted, and it listed the specific volumes that it tried to umount in order to delete the dead containers.
What did work for me was stopping docker first, and then deleting the directories manually. I was able to determine which ones they were by the error output of previous command to delete all the dead containers.
Apologies for the vague descriptions above. I found this SO question days after I handled the dead containers. .. However, I noticed a similar pattern today:
$ sudo docker stop fervent_fermi; sudo docker rm fervent_fermi fervent_fermi
Error response from daemon: Cannot destroy container fervent_fermi: Driver devicemapper failed to remove root filesystem a11bae452da3dd776354aae311da5be5ff70ac9ebf33d33b66a24c62c3ec7f35: Device is Busy
Error: failed to remove containers: [fervent_fermi]
$ sudo systemctl docker stop
$ sudo rm -rf /var/lib/docker/devicemapper/mnt/a11bae452da3dd776354aae311da5be5ff70ac9ebf33d33b66a24c62c3ec7f35
$
I did notice, when using this approach that docker re-created the images with different names:
a11bae452da3 trend_av_docker "bash" 2 weeks ago Dead compassionate_ardinghelli
This may have been due to the container being issued with restart=always, however, the container ID matches the ID of the container that previously used the volume that I force-deleted. There were no difficulties deleting this new container:
$ sudo docker rm -v compassionate_ardinghelli
compassionate_ardinghelli
Try this it worked for me on centos
1) docker container ls -a
gives you a list of containers check status which you want to get rid of
2) docker container rm -f 97af2da41b2b
not a big fan force flag but does the work
to check it worked just fire the command again or list it.
3) continue till we clear all dead containers
The best way to get rid of dead container processes is to restart your docker service.
I was unable to remove a container as it was stuck in restarting status, I just restarted the docker service and it worked for me.
I restarted the Docker Agent on Windows bottom right corner, docker iCon, Restart Docker. Then I ran docker ps -a and saw the container still sitting there.
I then ran docker rm -f container id and that worked for me.
If you are a Mac/Windows user and have Docker Desktop, you can just simply restart your Docker Desktop and in the UI you can simply remove the container without any complications.
In my case there was no actually mounted points inside docker container. Error raised after some ifup/down game inside privileged ct. The usual way of removing container didn’t work too uprising the similar error:
user#server:~$ docker rm -f tklc.dead
Error response from daemon: containers 896c259600c951323d5db4438d8faaa0de9de4b0cf0b998ab0896e82ec1b3769: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/3670d1e338eaf11b8fc7ada95a91995417fa3bb4ac1c36928c3c4195525988d4/diff/var/spool/postfix/dev/urandom: operation not permitted
Nor worked any attempt to unmount any links or mount points inside. Nor to find and stop any processes associated with dead container. Nor worked any attempts to remove files directly.
user#server:~$ sudo rm -rf /var/lib/docker/overlay2/3670d1e338eaf11b8fc7ada95a91995417fa3bb4ac1c36928c3c4195525988d4/
rm: cannot remove '/var/lib/docker/overlay2/3670d1e338eaf11b8fc7ada95a91995417fa3bb4ac1c36928c3c4195525988d4/diff/var/spool/postfix/dev/urandom': Operation not permitted
rm: cannot remove '/var/lib/docker/overlay2/3670d1e338eaf11b8fc7ada95a91995417fa3bb4ac1c36928c3c4195525988d4/diff/var/spool/postfix/dev/random': Operation not permitted
And finally I listed the file preventing docker container to be removed.
user#server:~$ sudo ls -l /var/lib/docker/overlay2/3670d1e338eaf11b8fc7ada95a91995417fa3bb4ac1c36928c3c4195525988d4/diff/var/spool/postfix/dev
total 0
---------- 1 root root 0 окт 31 18:17 random
---------- 1 root root 0 окт 31 18:17 urandom
Bingo! No permissions to access to file. Even for superuser! It is definitely extended attributes.
user#server:~$ sudo lsattr /var/lib/docker/overlay2/3670d1e338eaf11b8fc7ada95a91995417fa3bb4ac1c36928c3c4195525988d4/diff/var/spool/postfix/dev/urandom
----i----------------- /var/lib/docker/overlay2/3670d1e338eaf11b8fc7ada95a91995417fa3bb4ac1c36928c3c4195525988d4/diff/var/spool/postfix/dev/urandom
Here it is. Nice. After changing immutable bits we'll be able to retry removing container itself or any of its files. Finally
user#server:~$ sudo chattr -i /var/lib/docker/overlay2/3670d1e338eaf11b8fc7ada95a91995417fa3bb4ac1c36928c3c4195525988d4/diff/var/spool/postfix/dev/random
user#server:~$ docker ps -a|grep ‘Dead|Removal’|sed -e 's/\s.*$//'|xargs docker rm
896c259600c9
vou a la.
Related
I'm trying to kill my docker containers with the command:
$ docker container kill $(docker ps -q)
however, the containers aren't responding.
I'm able to log into them with:
$ docker exec -it container_id bash
but any commands within the terminal hangs.
Whats interesting is somehow the process also doesn't seem to exist. I get the list of running containers with the process ids as so:
$ for i in $(docker container ls --format "{{.ID}}"); do docker inspect -f '{{.State.Pid}} {{.Name}}' $i; done
12821 /brave_carson
12661 /trusting_hoover0
12617 /peaceful_franklin
12534 /frosty_volhard
12702 /zealous_sammet
12678 /flamboyant_jang
12690 /dreamy_driscoll
When I try to kill it with kill -9 pid I get the error:
$ kill -9 12821
-bash: kill: (12821) - No such process
This is very unusual. How do I resolve this? I'd prefer not to restart docker unless it is the last and only option.
If you kill the container, you cannot exec into it. The container must be running for exec to work (you should get an error message from this).
When the container is not running, there should be no process. However the container definition in docker, including logs, and changes to the container filesystem, will remain until you remove it with docker container rm (same as docker rm), e.g.:
docker container rm brave_carson
As a side note you can use docker run ... --rm ... to automatically remove containers after stop
I'm reading through the Docker documentation and I don't understand the difference between:
docker container prune
and
docker rm $(docker container ls -aq)
Note that in the link, the second command I've listed is docker rm $(docker ps -a -q), but there is no difference between that and what I've written. container ls is just the newer version of the ps command.
It seems that both of these commands remove all stopped containers. Is there more to it than that, or are these just synonyms?
I don't think there is substantial difference. This -a though means list all containers and as a result docker rm ... will also try to remove running containers. This gives the error that you see below:
Error response from daemon: You cannot remove a running container [...] Stop the container before attempting removal or force remove
example:
$ docker container run --rm -itd alpine:latest
0ec4d7459d35749ecc24cc5c6fd748f4254b0782f73f1ede76cf49b1fc53b2d4
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0ec4d7459d35 alpine:latest "/bin/sh" 4 seconds ago Up 1 second jovial_ritchie
$ docker rm $(docker container ls -aq)
Error response from daemon: You cannot remove a running container 0ec4d7459d35749ecc24cc5c6fd748f4254b0782f73f1ede76cf49b1fc53b2d4. Stop the container before attempting removal or force remove
$ docker container prune
WARNING! This will remove all stopped containers.
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B
But... difference when --force, -f is used:
In this case, the commands do 2 different things:
docker rm -f ... Forces the removal of a running container (uses SIGKILL) which means that it will remove running containers.
$ docker rm -f $(docker container ls -aq)
0ec4d7459d35
docker container prune -f will remove all stopped containers without asking for confirmation (no [y/N] prompt will be printed).
$ docker container prune -f
Total reclaimed space: 0B
The effects of the two commands are indeed similar, but there are some nuances to consider:
docker container prune can be used with the --filter option.
docker container prune has a synchronous protection that blocks concurrent prune executions on the daemon.
docker container prune attempts to remove only the containers that are not running, instead of trying to delete all containers and relying on the daemon to throw an exception for those that are not stopped, therefore is quicker and does not generate unnecessary error logs in case someone is tracking the daemon logs.
docker container prune builds a report at the end of its execution, providing the reclaimed space. The report is added in daemon.EventsService and implicitly displayed on the screen.
docker container prune is shorter
In the end of this answer I have a question: Why would someone type 15 additional characters to get the same result or worse?
docker system prune -f : to remove all the stopped containers (docker do not touch the running containers)
docker system prune -a : to remove all the stopped containers (docker do not touch the running containers) + unused images
docker rm <container_id> : remove a specific container, it should be stopped before (docker stop <container_id>)
While trying to build docker image to my application, somehow I accidentally create 2 containers with status Created.
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
597a3d226a08 2eba970a4fc8 "entrypoint.sh" 43 hours ago Created pedantic_almeida
bae7a9324bc8 b0898d035451 "entrypoint.sh" 43 hours ago Created zen_franklin
The problem is I cannot do anything with these containers. docker rm/restart/inspect <container-id> all hangs up indefinitely with no message printed. However, starting new container from these 2 images work fine.
So my question is how to remove these 2 containers? Please tell me if you need any additional information.
Docker version: 18.03.0-ce.
You can use the below command to remove the two containers hung in "created" state:
docker container prune --force
This will remove all stopped containers, so be careful!
docker volume rm $(docker volume ls -qf dangling=true)
docker rm $(docker ps -q -f 'status=created')
After execution of the above commands, restart docker by,
service docker restart
I'm reading a book on docker. It is a couple of years old.
I'll cite:
If you want to get rid of all your stopped containers, you can use
the output of docker ps -aq -f status=exited , which gets the
IDs of all stopped containers. For example:
$ docker rm -v $(docker ps -aq -f status=exited)
When I run this, I get:
michael#michael-desktop:~$ sudo docker rm -v $(docker ps -aq -f status=exited)
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.30/containers/json?all=1&filters=%7B%22status%22%3A%7B%22exited%22%3Atrue%7D%7D: dial unix /var/run/docker.sock: connect: permission denied
"docker rm" requires at least 1 argument(s).
See 'docker rm --help'.
Usage: docker rm [OPTIONS] CONTAINER [CONTAINER...]
Remove one or more containers
Could you help me understand what should I do to gain what is intended.
In order to remove all our stopped containers, you can first run
$ docker ps -a
This gives you the list of running and stopped containers, from which you can select what are the containers that you wanted to get rid. But if you want to get rid of all stopped containers, then you need to use
$ docker container prune
This removes all stopped containers by giving you the following messages.
Warning! This will remove all stopped containers.
Are you sure you want to continue? [y/N] y
Deleted Containers:
your container id list will be printed.
It could simply means that you have no container with a status 'exited'.
The commands becomes then:
sudo docker rm -v
The lack of any parameter would trigger the error message you see.
But today, this would be done with docker container prune anyway.
$ sudo docker rm -v $(docker ps -aq -f status=exited)
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get
http://%2Fvar%2Frun%2Fdocker.sock/v1.30/containers/json?all=1&filters=%7B%22status%22%3A%7B%22exited%22%3Atrue%7D%7D:
dial unix /var/run/docker.sock: connect: permission denied
"docker rm" requires at least 1 argument(s).
See 'docker rm --help'.
The permission denied message comes from the embedded docker ps command. It is run by the shell outside of your parent sudo command, and the output is passed to sudo to run the docker rm as root. There are several fixes.
The easy option, run the docker ps with sudo:
$ sudo docker rm -v $(sudo docker ps -aq -f status=exited)
Option two is to run an entire shell as root:
$ sudo -s
# docker rm -v $(docker ps -aq -f status=exited)
# exit
Or you can give your user access to the docker socket so sudo is no longer needed:
$ sudo usermod -aG docker $USER
$ newgrp docker
The above is a one time change, and gives that user root access implicitly with docker. Then you can run:
$ docker rm -v $(docker ps -aq -f status=exited)
What seems to be happening is docker was started with different user. Hence, docker ps -aq -f status=exited could not be run due permission issue and as a result got blank result.
Running docker rm complains of missing argument due to blank result.
michael#michael-desktop:~$ sudo docker rm -v $(docker ps -aq -f status=exited)
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock:
Actually its an rights issue.... the error message:
Got permission denied while trying to connect to the Docker daemon socket at unix://
Tells you that you cant connect to you docker daemon which is running under root. You should decide if you want to use docker with # sudo or as root user.
Manage Docker as a non-root user
Further as said the docker rm complains about no images found for deletion therefore it wouldn't possible to delete images.
The command docker rm $(docker ps -aq -f status=exited) is just fine with newest docker version 18.09.0 but you could use docker container prune as well that is the more interactive way.
I have run several containers from an image in different ports, but I cant stop or remove these containers. I have run this command:
# sudo docker rm -f f85956145f61
And no message are displayed, however container is still displayed.
I have tried with these commands:
# docker stop $(docker ps -a -q)
# docker rm $(docker ps -a -q)
But I get the following error:
# Get http:///var/run/docker.sock/v1.15/containers/json?all=1: dial unix /var/run/docker.sock: permission denied
So, how can I remove all containers from my docker?
I don't know if you are trying to stop the container as root, but if not try to stop/kill the container as root or using sudo.
$ sudo docker kill $(docker ps -q)
$ sudo docker rm $(docker ps -a -q)
If this does not work try restarting docker service
$ sudo service docker restart
and then try again to stop and delete the container. If it doesn't work you can try to kill the process(es) running inside the container from your host machine.
And to reference the container in docker kill, docker stop, docker rm,... and so on, you can either specify the container id or the container name, both are valid.
Another reason for this happening is that the daemon is busy committing the container.