I use docker-compose for developing django apps: I have a fairly simple setup - one service for web serving and one for database. Sometimes I need to run manage.py commands through docker.
I use following command:
$ docker-compose run web ./manage.py migrate
which works just fine. But the problem is that docker keeps the containers even after the command was finished:
$ docker-compose ps
Name Command State Ports
------------------------------------------------------------------------------------
seeder_data_1 sh Exit 0
seeder_postgres_1 /docker-entrypoint.sh postgres Up 0.0.0.0:5432->5432/tcp
seeder_web_1 fab run_local Up 0.0.0.0:8000->8000/tcp
seeder_web_run_14 ./manage.py migrate Up
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
61d3aee5bb39 seeder_web "./manage.py migrate" 4 minutes ago Restarting (2) 55 seconds ago seeder_web_run_14
85e91cb9383c seeder_web "fab run_local" 5 minutes ago Up 5 minutes 0.0.0.0:8000->8000/tcp seeder_web_1
565b01dedb7b postgres:latest "/docker-entrypoint.s" 5 minutes ago Up 5 minutes 0.0.0.0:5432->5432/tcp seeder_postgres_1
This is rather annoying: I don’t want to have ton of zombie containers that were used only for one command. Is there some way how to automatically end the machines after the command returns? I know I can kill the manually but that is kind of annoying.
Or is there some logic behind this?
Sounds like you want the --rm flag for docker-compose run: https://docs.docker.com/compose/reference/run/. It has been mentioned a couple times to make it a default (https://github.com/docker/compose/issues/2774 and https://github.com/docker/compose/issues/943) but looks like they both lost steam.
Related
This question already has an answer here:
Why do I have to delete docker containers?
(1 answer)
Closed 1 year ago.
I am new to Docker and just getting started. I pulled a basic ubuntu image and started a few containers with it and stopped them. When I run the command to list all the docker containers (even the stopped ones) I get an output like this:
> docker container ls -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
099c42011f24 ubuntu:latest "/bin/bash" 6 seconds ago Exited (0) 6 seconds ago sleepy_mccarthy
dde61c10d522 ubuntu:latest "/bin/bash" 8 seconds ago Exited (0) 7 seconds ago determined_rosalind
cd1a6fa35741 ubuntu:latest "/bin/bash" 9 seconds ago Exited (0) 8 seconds ago unruffled_lichterman
ff926b6eba23 ubuntu:latest "/bin/bash" 10 seconds ago Exited (0) 10 seconds ago cool_rosalind
8bd50c2c4729 ubuntu:latest "/bin/bash" 12 seconds ago Exited (0) 11 seconds ago cranky_darwin
My question is, is there a reason why docker does not delete the stopped containers by default?
The examples you've provided show that you're using an Ubuntu container just to run bash. While this is fairly common pattern while learning Docker, it's not what docker is used for in production scenarios, which is what Docker cares about and is optimizing for.
Docker is used to deploy an application within a container with a given configuration.
Say you spool up a database container to hold information about your application. Then your docker host restarts for some reason, and that database disappears by default. That would be a disaster.
It's therefore much safer for Docker to assume that you want to keep your containers, images, volumes, and so on, unless you explicitly ask for them to be removed and decide this is what you want when you start them, with docker run --rm <image> for example.
In my opinion, it may have some reasons. Consider below condition:
I build my image and start the container (production environment, for some reason I stop the current container, do some changes to image and run another instance, so new container with new name is running.
I see new container does not work properly as expected, so as now I have the old container, I can run the old one and stop the new so the clients will not face any issues.
But what if containers were automatically deleted if they were stopped?
Simple answer, I would have lost my clients (even my job) simply:) And one person would be added to unemployed people :D
As #msanford mentioned, Docker assumes you want to keep your data, volumes, etc. so you'll probably re-use them when needed.
Since Docker is used to deploy and run applications (as simple as WordPress with MySQL but with some differences installing on Shared Hosting), usually it's not used for only running bash.
Surely it's good to learn Docker in the first steps by running things like bash or sh to see the contents of container.
I have just installed Ubuntu 20.0 and installed docker using snap. I'm trying to run some different docker images for hbase and rabbitmq but each time I start an image, it immediately exists with 126 status.
$ sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4d58720fce3a dajobe/hbase "/opt/hbase-server" 5 seconds ago Exited (126) 4 seconds ago hbase-docker
b7a84731a05b harisekhon/hbase "/entrypoint.sh" About a minute ago Exited (126) 59 seconds ago optimistic_goldwasser
294b95ef081a harisekhon/hbase "/entrypoint.sh" About a minute ago Exited (126) About a minute ago goofy_tu
I have tried everything and tried to use docker inspect on separate images, but nothing gives away, why the containers exit out immediately. Any suggestions?
EDIT
When i run the command i run the following
$ sudo bash start-hbase.sh
It gives the output exactly like it should
Starting HBase container
Container has ID 3c3e36e1e0fbc59aa0783a4c7f3cb8690781b2d04e8f842749d629a9c25e0604
Updating /etc/hosts to make hbase-docker point to (hbase-docker)
Now connect to hbase at localhost on the standard ports
ZK 2181, Thrift 9090, Master 16000, Region 16020
Or connect to host hbase-docker (in the container) on the same ports
For docker status:
$ id=3c3e36e1e0fbc59aa0783a4c7f3cb8690781b2d04e8f842749d629a9c25e0604
$ docker inspect $id
I think the issue might be due to some permissions, because i tried to chck the logs as suggested in the comments, and get this error:
/bin/bash: /opt/hbase-server: Permission denied
Check if the filesystem is mounted with noexec option using mount command or in /etc/fstab. If yes, remove it and remount the filesystem (or reboot).
Quick solution is restart service docker and network-manager
I am trying to install additional python packages in AWX container awx_tasks so that the changes could enable the ansible modules like snow, ec2_elb_facts run (which have pre-requisites as Python modules). I have made the changes in the container using:
# docker exec -it 80ab6bf562a9 bash
where 80ab6bf562a9 is the container id for awx_task container.
and then installed the required packages inside the custom virtual environment (as mentioned in the AWX documentation). Post this, i have made the changes permanent by creating a new image with the container changes using:
# docker commit 80ab6bf562a9 ansible/awx_task:latest
Post this, ran the following command to map the new container with the newly created image with container changes.
# docker run --name awx_task -d 5290f9b3268c
Following are the containers post the above changes. Here, the newly created container which was mapped with the new image with changes in existing container is 968fb2a7da2f.
# docker container ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
968fb2a7da2f 5290f9b3268c "/tini -- /bin/sh -c…" 2 days ago Exited (143) 2 days ago awx_task
80ab6bf562a9 535bb2b8e1f3 "/tini -- /bin/sh -c…" 3 weeks ago Up 2 days 8052/tcp awx_task_OLD
aea2551951d5 b7c261b76010 "/tini -- /bin/sh -c…" 3 weeks ago Up 2 days 0.0.0.0:80->8052/tcp awx_web
e789a4a82a9e memcached:alpine "docker-entrypoint.s…" 3 weeks ago Up 2 days 11211/tcp memcached
a8c74584255c ansible/awx_rabbitmq:3.7.4 "docker-entrypoint.s…" 3 weeks ago Up 2 days 4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp rabbitmq
25f6f6ca7766 postgres:9.6 "docker-entrypoint.s…" 3 weeks ago Up 2 days 5432/tcp postgres
Following are my images post above changes. Here, the newly created image (with changes) is 5290f9b3268c.
# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ansible/awx_task latest 5290f9b3268c 2 days ago 1.48GB
postgres 9.6 106bdfb062df 8 weeks ago 235MB
ansible/awx_task <none> 535bb2b8e1f3 8 weeks ago 1.07GB
ansible/awx_web <none> b7c261b76010 8 weeks ago 1.04GB
hello-world latest 2cb0d9787c4d 2 months ago 1.85kB
memcached alpine b40e8fa7e3e5 2 months ago 8.69MB
ansible/awx_rabbitmq 3.7.4 e08fe791079e 6 months ago 85.6MB
The new container is properly mapped with the new image (which has got the changes i wanted). The issue now is that when i stop the old container and start the new container AWX doesn't work. I can just view the UI, if i run any tasks like executing templates, it just freezes. It appears like the new container/images are not talking with the other containers like awx_rabbitmq, postgres etc. I have been reading multiple posts regarding this however, i couldn't find any single post which highlights anything regarding this.
I basically want the changes in the awx_task container to work so that i could achieve the goal of making the custom modules work. Could anyone suggest what can be done so that the new awx_task container could take the role of the older awx_task and AWX could work normally?
Since i found the way to do this, i will share the steps to make the required changes.
The python package versions can be controlled from the requirements directory, AWX Task and AWX Web Images related changes can be applied in the Dockerfile.j2 in the roles directory. Once the required changes are applied, we can run the setup using ansible-playbook install.yml -i inventory.
You should use the install.yml to restart the awx_task container, since it ensures the right environment variables are set, the right volumes are mapped, etc. Same command as you've used to install AWX:
ansible-playbook install.yml -i inventory.
See here for a full list of arguments that are used.
based on Moving docker-compose containersets
I have loaded the images :
$ docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
br/irc latest 3203cf074c6b 23 hours ago 377MB
openjdk 8u131-jdk-alpine a2a00e606b82 5 days ago 101MB
nginx 1.13.3-alpine ba60b24dbad5 4 months ago 15.5MB
but now i want to run them, as they would run with docker-compose, but i cannot find any example.
here is the docker-compose.yml
version: '3'
services:
irc:
build: irc
hostname: irc
image: br/irc:latest
command: |
-Djava.net.preferIPv4Stack=true
-Djava.net.preferIPv4Addresses
run-app
volumes:
- ./br/assets/br.properties:/opt/br/src/java/br.properties
nginx:
hostname: nginx
image: nginx:1.13.3-alpine
ports:
- "80:80"
links:
- irc:irc
volumes:
- ./nginx/assets/default.conf:/etc/nginx/conf.d/default.conf
so how can i run the container, and attach to it, to see if its running, and in what order do i run these three images. Just started with docker, so not sure of the typical workflow ( build, run, attach etc )
so even though i do have docker-compose yml file, but since i have the build images from another host, can i possibly run docker commands to run and execute the images ? making sure that the local images are being referenced, and not the ones from docker registry.
Thanks #tgogos, this does give me a general overview, but specifically i was looking for:
$ docker run -dit openjdk:8u131-jdk-alpine
then:
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cc6ceb8a82f8 openjdk:8u131-jdk-alpine "/bin/sh" 52 seconds ago Up 51 seconds vibrant_hodgkin
shows its running
2nd:
$ docker run -dit nginx:1.13.3-alpine
3437cf295f1c7f1c27bc27e46fd46f5649eda460fc839d2d6a2a1367f190cedc
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3437cf295f1c nginx:1.13.3-alpine "nginx -g 'daemon ..." 20 seconds ago Up 19 seconds 80/tcp vigilant_kare
cc6ceb8a82f8 openjdk:8u131-jdk-alpine "/bin/sh" 2 minutes ago Up 2 minutes vibrant_hodgkin
then: finally:
[ec2-user#ip-10-193-206-13 DOCKERLOCAL]$ docker run -dit br/irc
9f72d331beb8dc8ccccee3ff56156202eb548d0fb70c5b5b28629ccee6332bb0
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9f72d331beb8 br/irc "/opt/irc/grailsw" 8 seconds ago Up 7 seconds 8080/tcp cocky_fermi
3437cf295f1c nginx:1.13.3-alpine "nginx -g 'daemon ..." 56 seconds ago Up 55 seconds 80/tcp vigilant_kare
cc6ceb8a82f8 openjdk:8u131-jdk-alpine "/bin/sh" 2 minutes ago Up 2 minutes vibrant_hodgkin
All three UP !!!!
Your question is about docker-compose but you also ask things about run, build, attach which makes me think I should try to help you with some basic information (which wasn't so easy for me to cope with a couple of months ago :-)
images
Images are somehow the base from which containers are created. Docker pulls images from http://hub.docker.com and stores them in your host to be used every time you create a new container. Changes in the container do not affect the base image.
To pull images from docker hub, use docker pull .... To build your own images start reading about Dockerfiles. A simple Dockerfile (in an abstract way) would look like this:
FROM ubuntu # base image
ADD my_super_web_app_files # adds files for your app
CMD run_my_app.sh # starts serving requests
To create the above image to your host, you use docker build ... and this is a very good way to build your images, because you know the steps taken to be created.
If this procedure takes long, you might consider later to store the image in a docker registry like http://hub.docker.com, so that you can pull it from any other machine easily. I had to do this, when dealing with ffmpeg on a Raspberry Pi (the compilation took hours, I needed to pull the already created image, not build it from scratch again in every Raspberry).
containers
Containers are based on images, you can have many different containers from the same image on the same host. docker run [image] creates a new container based on that image and starts it. Many people here start thinking containers are like mini-VMs. They are not!
Consider a container as a process. Every container has a CMD and when started, executes it. If this command finishes, or fails, the container stops, exits. A good example for this is nginx: go check the official Dockerfile, the command is:
CMD ["nginx"]
If you want to see the logs from the CMD, you can docker attach ... to your container. You can also docker stop ... a running container or docker start ... an already stopped one. You can "get inside" to type commands by:
docker exec -it [container_name] /bin/bash
This opens a new tty for you to type commands, while the CMD continues to run.
To read more about the above topics (I've only scratched the surface) I suggest you also read:
Is it possible to start a shell session in a running container (without ssh)
Docker - Enter Running Container with new TTY
How do you attach and detach from Docker's process?
Why docker container exits immediately
~jpetazzo: If you run SSHD in your Docker containers, you're doing it wrong!
docker-compose
After you feel comfortable with these, docker-compose will be your handy tool which will help you manipulate many containers with single line commands. For example:
docker compose up
Builds, (re)creates, starts, and attaches to containers for a service.
Unless they are already running, this command also starts any linked services.
The docker-compose up command aggregates the output of each container (essentially running docker-compose logs -f). When the command exits, all containers are stopped. Running docker-compose up -d starts the containers in the background and leaves them running
To run your docker-compose file you would have to execute:
docker-compose up -d
Then to see if your containers are running you would have to run:
docker ps
This command will display all the running containers
Then you could run the exec command which will allow you to enter inside a running container:
docker-compose exec irc
More about docker-compose up here: https://docs.docker.com/compose/reference/up/
I have a docker-compose setup, which is deployed in three steps:
Build all the containers and dc up -d (dc is an alias for docker-compose)
Create database with: dc run web /usr/local/bin/python create_db.py
Populate database with: dc run -d web /usr/local/bin/python -u manage.py populateDB
Steps 2 and 3 create new containers (see the first two):
~/Documents/Project » docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2ead532ea58b myproject_web "/usr/local/bin/pytho" 8 minutes ago Up 8 minutes 8000/tcp myproject_web_run_2
64e1f81ecd1a myproject_web "/usr/local/bin/pytho" 9 minutes ago Restarting (0) About a minute ago 8000/tcp myproject_web_run_1
9f5c670d4d7f myproject_nginx "/usr/sbin/nginx" 40 minutes ago Up 40 minutes 0.0.0.0:80->80/tcp myproject_nginx_1
46d3e8c09c03 myproject_web "/usr/local/bin/gunic" 40 minutes ago Up 40 minutes 8000/tcp myproject_web_1
ea876e68c8c6 postgres:latest "/docker-entrypoint.s" 40 minutes ago Up 40 minutes 0.0.0.0:5432->5432/tcp myproject_postgres_1
Which is all well and good, except they don't exit when their job is finished.
For example, the create db script, as you can see, is always restarting after it has created the database. Once achromap_web_run_2 has finished populating the database, it will add a second copy of each record, then a third, etc.. forever.
On github, it seems like this was asked for from docker, and the docker run --rm flag handles it. But --rm and -d are incompatible, which I don't understand.
Do you know how to kill containers which have finished executing their functions? Specifically, how to get dc run web /usr/local/bin/python create_db.py to exit once create_db.py calls exit()? Or is there a better way?
I think you may be conflating two things here.
The --rm flag
This exists to clean up after a container is finished, so it doesn't hang around in the dead containers pool. As you already found, it is not compatible with -d. But in this case, you don't need it anyway.
The --restart flag
(Also available in docker-compose as the restart property.)
This flag sets the restart policy. By default it is set to no, but you can set it to a few other values, including always. I would suspect you have it set to always currently, which would force the container to restart every time it stops on its own.
If you manually stop the container (docker stop ...) then the auto-restart would not engage. But if the process exits on its own, or crashes, then the container will be restarted. This is available for the obvious reason, so your service will start up again if it crashes.
How to proceed
I would say what you need is to use exec instead of run for these tasks.
First, run your container normally (i.e. docker-compose up -d).
Instead of using run to execute create_db.py, use exec.
docker-compose exec web /usr/local/bin/python create_db.py
This will use your already-running container, execute the script one time, and when the script exits, you're done. Since you did not create a new container (like run was doing), there is no cleanup to do afterward.
Note that you do not need the -it flag that is often used with docker exec. docker-compose emulates a tty on exec by default.