I have been running docker processes (apps) via
docker run …
But under runit supervision (runit is like daemontools) - so runit ensures that the process stays up, passes signals etc.
Is this reasonable? Docker seems to want to run its own demonization - but it isn't as thorough as runit. Furthermore, when runit restarts the app - a new container is created each time (fine) but it leaves a trace of the old one around - this seems to imply I am doing it in the wrong way.
Should docker not be run this way?
Should I instead set up a container from the image, just once, and then have runit run/supervise that container for all time?
Docker does do some management of daemonized containers: if the system shuts down, then when the Docker daemon starts it will also restart any containers that were running at the time the system shut down. But if the container exits on its own or the kernel (or a user) kills the container while it is running, the Docker daemon won't restart it. In cases where you do want a restart, a process manager makes sense.
I don't know runit so I can't give specific configuration guidance. But you should probably make the process manager communicate with the docker daemon and check to see if a given container id is running (docker ps | grep container_id or equivalent, or use the Docker Remote API directly). If the container has stopped, use Docker to restart it (docker run container_id) instead of running a new container. Or, if you do want a new container each time, then begin with docker run -rm to automatically clean it up when it exits or stops.
If you don't want your process manager to poll docker, you could instead run something that watches docker events.
You can get the container_id when you start the container as the return value of starting a daemon, or you can ask Docker to write this out to a file (docker run -cidfile myfilename, like a PID file)
I hope that helps or helps another runit guru offer more detailed advice.
Yes, I think running docker under runit makes sense. Typically when you start a process there is a way to tell it not to daemonize if it does by default since the normal way to hand-off from the runit run script to a process is via exec on the last line of your run script. For docker this means making sure not to set the -d flag.
For example, with docker you probably want your run script to look something like this:
#!/bin/bash -e
exec 2>&1
exec chpst -u dockeruser docker run -a stdin -a stdout -i ...
Using exec and chpst should resolve most issues with processes not terminating correctly when you bring down a runit service.
Related
For example I run
docker run --rm --name mycontainer -p 8080:8080 myrepo/myimage
then I see an output of my application, everything is OK. Then I press the Ctrl+C but the container is still running and I'm forced to explicitly stop and remove it:
docker rm -f <container_id>
Or even worse:
docker stop <container_id>
docker rm <container_id>
Is there any way to do it automatically? If not it's OK.
PS: What is the purpose of all that stopped containers still kept on the harddrive?!
What is the purpose of all that stopped containers still kept on the harddrive?!
Running containers include the process you are running along with the namespaced environment to run that process inside of (networking, pid, filesystem, etc).
A stopped container has the container specific read/write filesystem layer, any configuration you included to run the container (e.g. environment variables), and logs if you are using the json logging driver (default).
Removing a container deletes that RW filesystem layer, json logs, and the configuration, which also removes the container from the list of stopped containers. This is a permanent operation, so do not remove containers you may want to later inspect or restart.
I press the Ctrl+C but the container is still running and I'm forced to explicitly stop and remove it
First, make sure you are running a current version of docker. I believe somewhere around 1.13 they migrated the processing of the --rm option from the client to the server. Next, make sure your application handles the Ctrl+C command. In a shell script, this would be a handler for a SIGTERM. You also need run the container interactively so that the keyboard input is sent to the container, that is the -it flag. With all three of those done, you should see containers automatically cleaned up with:
docker run --rm -it --name mycontainer -p 8080:8080 myrepo/myimage
followed by a Ctrl+C. The -it will pass the SIGTERM to the container which should then stop the process, which stops the running container. And the --rm will result in the container being automatically removed.
If for some reason you cannot get your container to handle the SIGTERM, then you can send a SIGKILL with a docker kill command, which cannot be trapped and ignored by the application.
Note that if you run a docker stop on your container and see a 10 second delay before it is stopped, then your application is ignoring the SIGTERM. One common cause for this is a /bin/sh running as pid 1. A shell will ignore this signal when it's running as pid 1 by default, on the assumption that you are in signal user mode.
Per default, docker runs the image command as pid 1. pid 1 is handled special by the kernel as it normally is used for the system init process. For this reason, CTRL+C / SIGTERM does not work on pid 1.
Actual docker versions provide option --init to run a minimal init system (tini) as pid 1. Your image command runs as a child of tini. As no longer being pid 1, your image command will accept CTRL+C again.
Add --init to your sample command, and you can stop with CTRL+C
docker run --rm --init --name mycontainer -p 8080:8080 myrepo/myimage
Not needed in your case, just additional info: You can change the signal from docker stop with --stop-signal SIGNAL with SIGNAL being one of the many signals shown by kill -L, for example SIGHUP or SIGINT.
Like most docker users, I periodically need to connect to a running container and execute various arbitrary commands via bash.
I'm using 17.06-CE with an ubuntu 16.04 image, and as far as I understand, the only way to do this without installing ssh into the container is via docker exec -it <container_name> bash
However, as is well-documented, for each bash shell process you generate, you leave a zombie process behind when your connection is interrupted. If you connect to your container often, you end up with 1000s of idle shells -a most undesirable outcome!
How can I ensure these zombie shell processes are killed upon disconnection -as they would be over ssh?
One way is to make sure the linux init process runs in your container.
In recent versions of docker there is an --init option to docker run that should do this. This uses tini to run init which can also be used in previous versions.
Another option is something like the phusion-baseimage project that provides a base docker image with this capability and many others (might be overkill).
What is detached mode in the docker world? I read this article
Link, but it does not explain exactly what detached mode mean.
You can start a docker container in detached mode with a -d option. So the container starts up and run in background. That means, you start up the container and could use the console after startup for other commands.
The opposite of detached mode is foreground mode. That is the default mode, when -d option is not used. In this mode, the console you are using to execute docker run will be attached to standard input, output and error. That means your console is attached to the container's process.
In detached mode, you can follow the standard output of your docker container with docker logs -f <container_ID>.
Just try both options. I always use the detached mode to run my containers. I hope I could explain it a little bit clearer.
The detach option on the docker command line indicates that the docker client (docker) will make a request to the server (dockerd), and then the client will exit while that request continues on the server. Part of the confusion may be that docker looks like a single process, where in reality it is a client/server application where the client is just a thin frontend on a REST API to send every command to the server.
With a docker container run --detach, this means the container will be created, the server will respond with a container id if successful, and the container will continue to run on the server while you are free to run other commands. This is often used for a server (e.g. nginx) you want to start in the background while you continue to run other commands. Note that you can still configure a container with the --interactive and -tty options (often abbreviated -it) and later run a docker container attach to connect to an already running container. (Note, until you attach to the container running with -itd, any attempt by the container to read from stdin would hang, instead of seeing an end of input that often triggers an immediate exit if you just passed -d.)
If you run without the detach option, the client will immediately run an attach API call after the container is created so you can see the output and optionally provide input to the running process on the container. This is useful if your container is running something interactive (e.g. /bin/bash).
Several other commands allow the detach option, including docker-compose up -d which will start an entire project and leave it running on the server in the background. There's also many of the docker service commands which will either detach after submitting the change to the server to create or update a service's target state, or if you do not detach, the client will wait until the service's current state matches the target state and you can see the progress of the deployment. Note with docker service commands, you may have to pass --detach=false to remain attached, the behavior has changed over the past year depending on your version.
What is detached mode in the docker world?
Detached means that the container will run in the background, without being attached to any input or output stream.
docker provide --detach (or -d in short) option and started the program in the background.
This means that the program started but isn’t attached to your terminal.
Example docker run --detach --name web nginx:latest # Note the detach flag.
Why do we need --detach mode?
Server software is generally run in detached containers because it is rare that the software depends on an attached terminal.
Running detached containers is a perfect fit for programs that sit quietly in the background.
Note
Generally, This type of program is called a daemon, or a service. A daemon generally interacts with other programs (or humans over a network) or some other communication channel. When you launch a daemon or other program in a container that you want to run in the background, remember to use either the --detach flag or its short form, -d.
To understand what does mean that -d in docker world, let me explain more clearly with the simulation within it. Docker detached mode means that you can tell the docker that do process(container) until if I am going to write the command docker stop container-id or container-name otherwise I mean without detached mode docker run the process(container) either you press ctrl+c or close a terminal, in other words you have not any choice when docker run the process(container) if you run the process(container) without -d. But you have any choice that docker return id of the container when you type the command with -d because of running process(container) in the background.
docker run -d -t ubuntu:14.04
docker run - Create an instance from docker image as docker container.
(if image not available locally it pulls from docker hub)
ubuntu - Image name
14.04 - Tag
-d, --detach - Detach mode
-t, --tty - Allocate a pseudo-TTY
I have a docker container with JETTY CMD instruction.
After "docker restart", which goes immediately, I cannot access JETTY about 9-10 seconds. After that time docker container or jetty service is UP again and I can access it.
Question is: is there a standard way to check that the docker container is really up?
Surely I can make a loop with test requests to my service and wait to 200 response code. But maybe there is a more beautiful solution?
Thanks
Sergey, youre need to use initialization system as supervisor of your in-Docker processes. You may use distro-built-in init systems like systemd/upstart or init.d depends on your OS for checking a container state.
In theory you should to create independent service in you init system on each docker run command without -d option, because with -d option docker detached a container and returned 0 exit status to init system. As result init system lost a control of target process.
For example, realization of this mechanism in Systemd:
Create something.service file in /etc/systemd/system
And type to it something like this:
[Unit]
Description=Simple Blog Rails Docker Container Service
After=docker.service
Requires=docker.service
[Service]
Restart=on-failure
ExecStartPre=-/usr/bin/docker kill simple-blog-rails-container
ExecStartPre=-/usr/bin/docker rm simple-blog-rails-container
ExecStart=/usr/bin/docker run simple-blog-rails
ExecStop=/usr/bin/docker stop simple-blog-rails-container
[Install]
WantedBy=multi-user.target
Reload Systemd configuration systemctl daemon-reload
Just try to run your container by typing systemctl start something.service or restart instead of start.
You can check service state systemctl status something.service
For more information about using systemd and docker you may read this CoreOS manual: https://coreos.com/docs/launching-containers/launching/getting-started-with-systemd/
Keep in mind that docker restart defaults to a wait time of 10 seconds:
restart Usage: docker restart [OPTIONS] CONTAINER [CONTAINER...]
Restart a running container
-t, --time=10 Number of seconds to try to stop for before
killing the container. Once killed it will then be restarted. Default
is 10 seconds.
When I try this with some long running script in a docker container it takes 10 seconds before it's done. If I change the command line to use a different timeout (like -t=4} it comes back in 4 seconds.
If you want to determine if a container is running (even if your service contained within it is not quite ready yet) you can:
Run docker ps - which will give you a list of all running docker
containers
Use the Docker Remote API command GET /containers/json - this will
give you a json response with a list of running containers.
https://docs.docker.com/reference/api/docker_remote_api_v1.16/
https://docs.docker.com/reference/commandline/cli/#ps
I've seen a bunch of tutorials that seem do the same thing I'm trying to do, but for some reason my Docker containers exit. Basically, I'm setting up a web-server and a few daemons inside a Docker container. I do the final parts of this through a bash script called run-all.sh that I run through CMD in my Dockerfile. run-all.sh looks like this:
service supervisor start
service nginx start
And I start it inside of my Dockerfile as follows:
CMD ["sh", "/root/credentialize_and_run.sh"]
I can see that the services all start up correctly when I run things manually (i.e. getting on to the image with -i -t /bin/bash), and everything looks like it runs correctly when I run the image, but it exits once it finishes starting up my processes. I'd like the processes to run indefinitely, and as far as I understand, the container has to keep running for this to happen. Nevertheless, when I run docker ps -a, I see:
➜ docker_test docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c7706edc4189 some_name/some_repo:blah "sh /root/run-all.sh 8 minutes ago Exited (0) 8 minutes ago grave_jones
What gives? Why is it exiting? I know I could just put a while loop at the end of my bash script to keep it up, but what's the right way to keep it from exiting?
If you are using a Dockerfile, try:
ENTRYPOINT ["tail", "-f", "/dev/null"]
(Obviously this is for dev purposes only, you shouldn't need to keep a container alive unless it's running a process eg. nginx...)
I just had the same problem and I found out that if you are running your container with the -t and -d flag, it keeps running.
docker run -td <image>
Here is what the flags do (according to docker run --help):
-d, --detach=false Run container in background and print container ID
-t, --tty=false Allocate a pseudo-TTY
The most important one is the -t flag. -d just lets you run the container in the background.
This is not really how you should design your Docker containers.
When designing a Docker container, you're supposed to build it such that there is only one process running (i.e. you should have one container for Nginx, and one for supervisord or the app it's running); additionally, that process should run in the foreground.
The container will "exit" when the process itself exits (in your case, that process is your bash script).
However, if you really need (or want) to run multiple service in your Docker container, consider starting from "Docker Base Image", which uses runit as a pseudo-init process (runit will stay online while Nginx and Supervisor run), which will stay in the foreground while your other processes do their thing.
They have substantial docs, so you should be able to achieve what you're trying to do reasonably easily.
you can run plain cat without any arguments as mentioned by bro #Sa'ad to simply keep the container working [actually doing nothing but waiting for user input] (Jenkins' Docker plugin does the same thing)
The reason it exits is because the shell script is run first as PID 1 and when that's complete, PID 1 is gone, and docker only runs while PID 1 is.
You can use supervisor to do everything, if run with the "-n" flag it's told not to daemonize, so it will stay as the first process:
CMD ["/usr/bin/supervisord", "-n"]
And your supervisord.conf:
[supervisord]
nodaemon=true
[program:startup]
priority=1
command=/root/credentialize_and_run.sh
stdout_logfile=/var/log/supervisor/%(program_name)s.log
stderr_logfile=/var/log/supervisor/%(program_name)s.log
autorestart=false
startsecs=0
[program:nginx]
priority=10
command=nginx -g "daemon off;"
stdout_logfile=/var/log/supervisor/nginx.log
stderr_logfile=/var/log/supervisor/nginx.log
autorestart=true
Then you can have as many other processes as you want and supervisor will handle the restarting of them if needed.
That way you could use supervisord in cases where you might need nginx and php5-fpm and it doesn't make much sense to have them apart.
Motivation:
There is nothing wrong in running multiple processes inside of a docker container. If one likes to use docker as a light weight VM - so be it. Others like to split their applications into micro services. Me thinks: A LAMP stack in one container? Just great.
The answer:
Stick with a good base image like the phusion base image. There may be others. Please comment.
And this is yet just another plead for supervisor. Because the phusion base image is providing supervisor besides of some other things like cron and locale setup. Stuff you like to have setup when running such a light weight VM. For what it's worth it also provides ssh connections into the container.
The phusion image itself will just start and keep running if you issue this basic docker run statement:
moin#stretchDEV:~$ docker run -d phusion/baseimage
521e8a12f6ff844fb142d0e2587ed33cdc82b70aa64cce07ed6c0226d857b367
moin#stretchDEV:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS
521e8a12f6ff phusion/baseimage "/sbin/my_init" 12 seconds ago Up 11 seconds
Or dead simple:
If a base image is not for you... For the quick CMD to keep it running I would suppose something like this for bash:
CMD exec /bin/bash -c "trap : TERM INT; sleep infinity & wait"
Or this for busybox:
CMD exec /bin/sh -c "trap : TERM INT; (while true; do sleep 1000; done) & wait"
This is nice, because it will exit immediately on a docker stop.
Just plain sleep or cat will take a few seconds before the container is forcefully killed by docker.
Updates
As response to Charles Desbiens concerning running multiple processes in one container:
This is an opinion. And the docs are pointing in this direction. A quote: "It’s ok to have multiple processes, but to get the most benefit out of Docker, avoid one container being responsible for multiple aspects of your overall application." For sure it obviously much more powerful to devide your complex service into multiple containers. But there are situations where it can be beneficial to go the one container route. Especially for appliances. The GitLab Docker image is my favourite example of a multi process container. It makes deployment of this complex system easy. There is no way for mis-configuration. GitLab retains all control over their appliance. Win-Win.
Make sure that you add daemon off; to you nginx.conf or run it with CMD ["nginx", "-g", "daemon off;"] as per the official nginx image
Then use the following to run both supervisor as service and nginx as foreground process that will prevent the container from exiting
service supervisor start && nginx
In some cases you will need to have more than one process in your container, so forcing the container to have exactly one process won't work and can create more problems in deployment.
So you need to understand the trade-offs and make your decision accordingly.
Since docker engine v1.25 there is an option called init.
Docker-compose included this command as of version 3.7.
So my current CMD when running a container that should run into infinity:
CMD ["sleep", "infinity"]
and then run it using:
docker build
docker run --rm --init app
crf.:
rm docs and init docs
Capture the PID of the ngnix process in a variable (for example $NGNIX_PID) and at the end of the entrypoint file do
wait $NGNIX_PID
In that way, your container should run until ngnix is alive, when ngnix stops, the container stops as well
Along with having something along the lines of : ENTRYPOINT ["tail", "-f", "/dev/null"] in your docker file, you should also run the docker container with -td option. This is particularly useful when the container runs on a remote m/c. Think of it more like you have ssh'ed into a remote m/c having the image and started the container. In this case, when you exit the ssh session, the container will get killed unless it's started with -td option. Sample command for running your image would be: docker run -td <any other additional options> <image name>
This holds good for docker version 20.10.2
There are some cases during development when there is no service yet but you want to simulate it and keep the container alive.
It is very easy to write a bash placeholder that simulates a running service:
while true; do
sleep 100
done
You replace this by something more serious as the development progress.
How about using the supervise form of service if available?
service YOUR_SERVICE supervise
Once supervise is successfully running, it will not exit unless it is
killed or specifically asked to exit.
Saves having to create a supervisord.conf