Zombie docker container that can't be killed - docker

I use docker-compose to create a bunch of containers and link them together. For some of the container definitions, I might have restart: always as the restart policy.
Now I have a postgres container that respawns back to life if stopped.
$docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a8bb2b781630 postgres:latest "docker-entrypoint.s…" About an hour ago Up About an hour 5432/tcp dcat_postgres.1.z3pyl24kiq2n4clt0ua77nfx5
docker stop a8bb2b781630
a8bb2b781630
$ docker rm -f a8bb2b781630
a8bb2b781630
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
93fa7b72c2ca postgres:latest "docker-entrypoint.s…" 12 seconds ago Up 4 seconds 5432/tcp dcat_postgres.1.oucuo5zg3y9ws3p7jvlfztflb
Using docker-compose down in the dir that started the service doesn't work either.
$ docker-compose down
Removing dcat_postgres_1 ... done
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7ee7fb0e98cd postgres:latest "docker-entrypoint.s…" 13 seconds ago Up 5 seconds 5432/tcp dcat_postgres.1.jhv1q6vns1avakqjxdusmbb78
How can I kill a container and keep it from coming back to life?
EDIT: The container respawns even after restarting the Docker service.
Docker - 18.06.1-ce-mac73 (26764)
macOS High-Sierra, (10.13.6)

I figured it out. Turns out it was related to docker swarm. I had experimented with it at some point without fully understanding what it is and what it does and apparently it just stayed there.
All I had to do was:
docker swarm leave --force
and it worked like a head-shot to an actual zombie.

Can you try an option like moby/moby issue 10032:
docker stop $(docker ps -a -q) &
docker update --restart=no $(docker ps -a -q) &
systemctl restart docker
(this assume here you have only one running container: the one you cannot prevent to start)
A docker rm -f should be enough though, unless you are using docker with a provision tool like Puppet.
As it turned out, another process (other than docker itself) was responsible for the container to restart (here docker swarm)
Update 2020/2021: For multiple containers, possibly without having to restart the docker daemon
docker ps -a --format="{{.ID}}" | \
xargs docker update --restart=no | \
xargs docker stop
Check if you need, as in the issue, remove the images as well ( | xargs docker rmi $(docker images -qa) --force)

Related

list all containers with restart policies

Is it possible to list all docker contianers with a restart policy set? I could inspect the running containers (How to quickly show policies of all docker containers), but what if one has been previously stopped, or has an on-failure policy and has completed successfully?
I could inspect the running containers (How to quickly show policies of all docker containers), but what if one has been previously stopped, or has an on-failure policy and has completed successfully?
Exactly, so do not stop at running containers - inspect all containers.
Just list all containers with that policy.
docker ps -a -q | xargs docker inspect | jq -r '.[] | select(.HostConfig.RestartPolicy.Name != "no") | .Name'
How does docker system prune work with this?
It does not work with this (?).
Will it skip the removal of any containers that are stopped, but have a restart policy set?
No, a stopped container is also removed. Observe the following terminal transcript:
$ docker run --restart always -d --name test alpine
7f22c1f1439b9a211ac85bf10b4da563aaa8f76a2229c290f1fceb0404ded2a8
kamil#leonidas /home/kamil
$ docker stop test
test
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7f22c1f1439b alpine "/bin/sh" 3 seconds ago Exited (0) Less than a second ago test
$ docker ps -a | grep test
7f22c1f1439b alpine "/bin/sh" 10 seconds ago Exited (0) 7 seconds ago test
$ yes y | docker system prune
WARNING! This will remove:
- all stopped containers
- all networks not used by at least one container
- all dangling images
- all dangling build cache
Are you sure you want to continue? [y/N] Deleted Containers:
7f22c1f1439b9a211ac85bf10b4da563aaa8f76a2229c290f1fceb0404ded2a8
Total reclaimed space: 41B
$ docker ps -a | grep test
/* empty */

docker + why we cant kill the container

we have Linux machine with the following container
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6de660db9fdb kafka-exporter:v1.9.0 "/bin/kafka_export" 23 hours ago Up 17 hours kafka-export
we want to kill the container so we did that:
docker kill 6de660db9fdb
but its hang for along time ( more then hour and not killed )
any advice how to stop/kill the container ?
You could try restarting the Docker service first:
sudo systemctl restart docker
And then removing the container with the force -f flag:
sudo docker rm -f 6de660db9fdb

docker container not removing from docker

I am trying to remove container but when i run docker-compose rm ,runs fine but when i run docker ps then again it shows container:
root#datafinance:/tmp# docker-compose rm
Going to remove tmp_zookeeper_1_31dd890a1cbf
Are you sure? [yN] y
Removing tmp_zookeeper_1_31dd890a1cbf ... done
root#datafinance:/tmp# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
03b08e4ef0b3 confluentinc/cp-zookeeper:latest "/etc/confluent/dock…" 14 hours ago Up 14 hours docker_c_zookeeper_1_7c953dce7d69
Use docker-compose ps, it will show the container which only launched by docker-compose up. If it shows there is no container, then this means this container was not launched by this docker-compose.yaml.
And Error starting userland proxy: listen tcp 0.0.0.0:32181: bind: address already in use' means the port 32181 is occupied either by other docker container or other process. You could use docker rm -f $(docker ps -qa) to delete all containers or more you can use netstat -oanltp | grep 32181 to find which process really occupy 32181.
Finally, if for any reason you did not able to delete container as you said, you can just use service docker restart or systemctl restart docker to make all container down. Then repeat above docker rm xxx.
With above steps, you can use docker compose up -d to use your service now.
try this :
docker rm -f 03b08e4ef0b3
DANGER
you may also try this, but be aware that will delete everything (Containers, Images, Networks, ....)
docker system prune -a -f
when all not helped your last resort is to restart Docker daemon
service docker restart
and then repeat the steps...
I think what you are looking for is :
docker-compose down
which removes the containers after stopping them according to this.
According to this, docker-compose rm removes the "stopped" containers. If your container(s) are running, I think it won't remove to prevent accidents.

How to kill container generated by docker-compose?

(sorry using the term "kill" with quotes is not about docker-compose kill, is about "UNIX ps kill" after what the process really go out of the "UNIX ps list")
Usual docker run can be "killed" by usual docker stop, because after stop I not see the container at docker ps -a... If it is correct, there are a semantic bug with docker-compose because I can't "kill" the containers, they stay at docker ps.
After my simple docker-compose up (without &) I do ^C and the containers stay there at docker ps -a... Impossible to kill by docker compose stop.
NOTE: when I use ordinary docker run and after it docker stop there are nothing at docker ps -a, so I can say "I killed it".
Usual docker run can be "killed" by usual docker stop, because after stop I not see the container at docker ps.
No. docker stop just stops a running container, it doesn' t remove the container. This happens only in case you've used docker run --rm .... This --rm option means that when the container is stopped, it will be removed/deleted.
Docker
docker run ... creates and runs a container
docker stop ... stops a running container
docker start ... starts a stopped container
docker rm ... removes a stopped container
Docker Compose
docker-compose up creates and runs a collection of containers
docker-compose stop stops the containers
docker-compose start starts the containers
docker-compose down stops and removes the containers
Be careful...
As it discussed in the comments section, by using docker-compose down other things can also take place regarding volumes, networks. Keep in mind that you might lose data (if your container is a database for example) and make sure you have saved them or you are somehow able to create them again.
Check out running containers:
docker ps
Example output:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e86521d81a96 app_php "docker-php-entrypoi…" 2 hours ago Up About an hour 0.0.0.0:8080->80/tcp app_php_1
7a30681b6255 mysql:5.6 "docker-entrypoint.s…" 3 hours ago Up About an hour 0.0.0.0:3306->3306/tcp app_db_1
21aa3eef5f42 phpmyadmin/phpmyadmin "/run.sh supervisord…" 4 hours ago Up About an hour 9000/tcp, 0.0.0.0:8081->80/tcp app_phpmyadmin_1
9afc52b3f82f mailhog/mailhog "MailHog" 4 hours ago Up About an hour 1025/tcp, 0.0.0.0:8082->8025/tcp app_mailhog_1
then stop one by the container id:
docker kill part_of_the_id/name
For instance:
docker kill e86 or docker kill app_php_1
Docker-compose is just a script to help you manage one or multiple containers running in a group and is absolutely not required to manage your containers.
To remove the container completely you have to remove the container docker rm container_id_or_name
To stop all running containers:
docker stop $(docker ps -q)
You can use docker rm <container-name> to do that. This command will stop and remove service container. Anonymous volumes attached to the container will not be removed.

Ensuring Docker container will start automatically when host starts

Is there a way to start a Docker container automatically when the host starts? Before, I use the ‘—restart always’ parameter with docker run but it only works if Docker Engine is not killed.
As your comment, I think you have misunderstood about --restart always.
Once docker run --restart always container is run, the container is restarted every time the host is restarted even though you stop the container explicitly .
For example.
$ docker run --restart always --detach --name auto-start-redis redis
d04dfbd73eb9d2ba5beac41363aa5c45c0e034e08173daa6146c3c704e0cd1da
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d04dfbd73eb9 redis "docker-entrypoint..." 4 seconds ago Up 4 seconds 6379/tcp auto-start-redis
$ reboot
# After reboot-------------------------------
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d04dfbd73eb9 redis "docker-entrypoint..." About a minute ago Up 21 seconds 6379/tcp auto-start-redis
$ docker stop auto-start-redis
auto-start-redis
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d04dfbd73eb9 redis "docker-entrypoint..." 2 minutes ago Exited (0) 30 seconds ago auto-start-redis
$ reboot
# After reboot-------------------------------
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d04dfbd73eb9 redis "docker-entrypoint..." 3 minutes ago Up 12 seconds 6379/tcp auto-start-redis
However, of course, it is based upon a premise that docker-host is auto-started. docker-host in here means docker daemon process itself. Usually docker-host will auto-start by default but if it is not, you need to set it yourself.
I am not sure which OS you are using but when it comes to Ubuntu16, you can make it with systemctl command.
$ sudo systemctl enable docker
# To tell systemd to start services automatically at boot, you must enable.
If you use docker swarm, you can make global service with --mode global flag that ensures run on every node in docker swarm.
docker service create --mode global ...
If you don't use docker swarm, the best solution I think is to use init system of your system like systemd as #I.R.R said. You can make your own service file for systemd and specify the condition when the service starts like below.
[Unit]
Description=Your App
After=docker
Refer to this article by digital ocean.

Resources