I have created a systemd service that starts a set of Docker containers using Docker-Compose, as outlined in this answer:
# /etc/systemd/system/docker-compose-app.service
[Unit]
Description=Docker Compose Application Service
Requires=docker.service
After=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/srv/docker
ExecStart=/usr/local/bin/docker-compose up -d
ExecStop=/usr/local/bin/docker-compose down
TimeoutStartSec=0
[Install]
WantedBy=multi-user.target
This allows me to start the Docker-Compose services using
sudo systemctl start docker-compose-app
which uses docker-compose up -d under the hood, and shutting them down using
sudo systemctl stop docker-compose-app
which uses docker-compose down. Please note that the down command is run without the -v flag, which means that volumes will remain in place, preserving the data of my containers across restarts/recreation. This is pretty much what I want in the majority of cases.
There are situations where I want to erase all data in the services, basically running the down -v command instead of just down.
Is there a way to extend the above systemd service definition to allow for an additional command (or using one of the existing systemctl commands) that would allow me to run the occasional down -v if needed. I want to do this ad-hoc if needed, not scheduled or anything like that.
How can I run
docker-compose down -v
occasionally if needed through the same systemd setup, while keeping the standard functionality of maintaining the containers' data across restarts?
You may try to use ExecReload definition:
ExecReload=/usr/local/bin/docker-compose down -v && /usr/local/bin/docker-compose up -d
And then you can use:
sudo systemctl reload docker-compose-app
So "reload" command will be used for "reset" in this case.
I pulled centos6 image and made a container from it. I got its bash by:
$ docker run -i -t centos:centos6 /bin/bash
On the centos6 container, I could use "service" command without any problem. But when I pulled&used centos7 image:
$ docker run -i -t centos:centos7 /bin/bash
Both of "service" and "systemctl" didn't work. The error message is:
Failed to get D-Bus connection: Operation not permitted
My question is:
1. How are people developing without "service" and "systemctl" commands?
2. If I want to use, for example, httpd.service on the centos7 container, what should I do? Or maybe running services on a container is not recommended?
There is no process supervisor running inside either container. The service command in your CentOS 6 container works by virtue of the fact that it just runs a script from /etc/init.d, which by design ultimately launch a command in the background and return control to you.
CentOS 7 uses systemd, and systemd is not running inside your container, so there is nothing for systemctl to talk to.
In either situation, using the service or systemctl command is generally the wrong thing to do: you want to run a single application, and you want to run it in the foreground, so that your container continues to run (from Docker's perspective, a command that goes into the background has exited, and if that was pid 1 in the container, the container will exit).
How are people developing without "service" and "systemctl" commands?
They are starting their programs directly, by consulting the necessary documentation to figure out the appropriate command line.
If I want to use, for example, httpd.service on the centos7 container, what should I do? Or maybe running services on a container is recommended?
You would start the httpd binary using something like:
CMD ["httpd", "-DFOREGROUND"]
If you like to stick with service/sytemctl commands to start/stop services then you can do that in a centos7 container by using the docker-systemctl-replacement script.
I had some deployment scripts that were using th service start/stop commands on a real machine - and they work fine with a container. Without any further modification. When putting the systemctl.py script into the CMD then it will simply start all enabled services somewhat like the init-process on a real machine.
systemd is included but not enabled by default in CentOS 7 docker image. It is mentioned on the repository page along with steps to enable it.
https://hub.docker.com/_/centos/
I have a docker container with JETTY CMD instruction.
After "docker restart", which goes immediately, I cannot access JETTY about 9-10 seconds. After that time docker container or jetty service is UP again and I can access it.
Question is: is there a standard way to check that the docker container is really up?
Surely I can make a loop with test requests to my service and wait to 200 response code. But maybe there is a more beautiful solution?
Thanks
Sergey, youre need to use initialization system as supervisor of your in-Docker processes. You may use distro-built-in init systems like systemd/upstart or init.d depends on your OS for checking a container state.
In theory you should to create independent service in you init system on each docker run command without -d option, because with -d option docker detached a container and returned 0 exit status to init system. As result init system lost a control of target process.
For example, realization of this mechanism in Systemd:
Create something.service file in /etc/systemd/system
And type to it something like this:
[Unit]
Description=Simple Blog Rails Docker Container Service
After=docker.service
Requires=docker.service
[Service]
Restart=on-failure
ExecStartPre=-/usr/bin/docker kill simple-blog-rails-container
ExecStartPre=-/usr/bin/docker rm simple-blog-rails-container
ExecStart=/usr/bin/docker run simple-blog-rails
ExecStop=/usr/bin/docker stop simple-blog-rails-container
[Install]
WantedBy=multi-user.target
Reload Systemd configuration systemctl daemon-reload
Just try to run your container by typing systemctl start something.service or restart instead of start.
You can check service state systemctl status something.service
For more information about using systemd and docker you may read this CoreOS manual: https://coreos.com/docs/launching-containers/launching/getting-started-with-systemd/
Keep in mind that docker restart defaults to a wait time of 10 seconds:
restart Usage: docker restart [OPTIONS] CONTAINER [CONTAINER...]
Restart a running container
-t, --time=10 Number of seconds to try to stop for before
killing the container. Once killed it will then be restarted. Default
is 10 seconds.
When I try this with some long running script in a docker container it takes 10 seconds before it's done. If I change the command line to use a different timeout (like -t=4} it comes back in 4 seconds.
If you want to determine if a container is running (even if your service contained within it is not quite ready yet) you can:
Run docker ps - which will give you a list of all running docker
containers
Use the Docker Remote API command GET /containers/json - this will
give you a json response with a list of running containers.
https://docs.docker.com/reference/api/docker_remote_api_v1.16/
https://docs.docker.com/reference/commandline/cli/#ps
I've seen a bunch of tutorials that seem do the same thing I'm trying to do, but for some reason my Docker containers exit. Basically, I'm setting up a web-server and a few daemons inside a Docker container. I do the final parts of this through a bash script called run-all.sh that I run through CMD in my Dockerfile. run-all.sh looks like this:
service supervisor start
service nginx start
And I start it inside of my Dockerfile as follows:
CMD ["sh", "/root/credentialize_and_run.sh"]
I can see that the services all start up correctly when I run things manually (i.e. getting on to the image with -i -t /bin/bash), and everything looks like it runs correctly when I run the image, but it exits once it finishes starting up my processes. I'd like the processes to run indefinitely, and as far as I understand, the container has to keep running for this to happen. Nevertheless, when I run docker ps -a, I see:
➜ docker_test docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c7706edc4189 some_name/some_repo:blah "sh /root/run-all.sh 8 minutes ago Exited (0) 8 minutes ago grave_jones
What gives? Why is it exiting? I know I could just put a while loop at the end of my bash script to keep it up, but what's the right way to keep it from exiting?
If you are using a Dockerfile, try:
ENTRYPOINT ["tail", "-f", "/dev/null"]
(Obviously this is for dev purposes only, you shouldn't need to keep a container alive unless it's running a process eg. nginx...)
I just had the same problem and I found out that if you are running your container with the -t and -d flag, it keeps running.
docker run -td <image>
Here is what the flags do (according to docker run --help):
-d, --detach=false Run container in background and print container ID
-t, --tty=false Allocate a pseudo-TTY
The most important one is the -t flag. -d just lets you run the container in the background.
This is not really how you should design your Docker containers.
When designing a Docker container, you're supposed to build it such that there is only one process running (i.e. you should have one container for Nginx, and one for supervisord or the app it's running); additionally, that process should run in the foreground.
The container will "exit" when the process itself exits (in your case, that process is your bash script).
However, if you really need (or want) to run multiple service in your Docker container, consider starting from "Docker Base Image", which uses runit as a pseudo-init process (runit will stay online while Nginx and Supervisor run), which will stay in the foreground while your other processes do their thing.
They have substantial docs, so you should be able to achieve what you're trying to do reasonably easily.
you can run plain cat without any arguments as mentioned by bro #Sa'ad to simply keep the container working [actually doing nothing but waiting for user input] (Jenkins' Docker plugin does the same thing)
The reason it exits is because the shell script is run first as PID 1 and when that's complete, PID 1 is gone, and docker only runs while PID 1 is.
You can use supervisor to do everything, if run with the "-n" flag it's told not to daemonize, so it will stay as the first process:
CMD ["/usr/bin/supervisord", "-n"]
And your supervisord.conf:
[supervisord]
nodaemon=true
[program:startup]
priority=1
command=/root/credentialize_and_run.sh
stdout_logfile=/var/log/supervisor/%(program_name)s.log
stderr_logfile=/var/log/supervisor/%(program_name)s.log
autorestart=false
startsecs=0
[program:nginx]
priority=10
command=nginx -g "daemon off;"
stdout_logfile=/var/log/supervisor/nginx.log
stderr_logfile=/var/log/supervisor/nginx.log
autorestart=true
Then you can have as many other processes as you want and supervisor will handle the restarting of them if needed.
That way you could use supervisord in cases where you might need nginx and php5-fpm and it doesn't make much sense to have them apart.
Motivation:
There is nothing wrong in running multiple processes inside of a docker container. If one likes to use docker as a light weight VM - so be it. Others like to split their applications into micro services. Me thinks: A LAMP stack in one container? Just great.
The answer:
Stick with a good base image like the phusion base image. There may be others. Please comment.
And this is yet just another plead for supervisor. Because the phusion base image is providing supervisor besides of some other things like cron and locale setup. Stuff you like to have setup when running such a light weight VM. For what it's worth it also provides ssh connections into the container.
The phusion image itself will just start and keep running if you issue this basic docker run statement:
moin#stretchDEV:~$ docker run -d phusion/baseimage
521e8a12f6ff844fb142d0e2587ed33cdc82b70aa64cce07ed6c0226d857b367
moin#stretchDEV:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS
521e8a12f6ff phusion/baseimage "/sbin/my_init" 12 seconds ago Up 11 seconds
Or dead simple:
If a base image is not for you... For the quick CMD to keep it running I would suppose something like this for bash:
CMD exec /bin/bash -c "trap : TERM INT; sleep infinity & wait"
Or this for busybox:
CMD exec /bin/sh -c "trap : TERM INT; (while true; do sleep 1000; done) & wait"
This is nice, because it will exit immediately on a docker stop.
Just plain sleep or cat will take a few seconds before the container is forcefully killed by docker.
Updates
As response to Charles Desbiens concerning running multiple processes in one container:
This is an opinion. And the docs are pointing in this direction. A quote: "It’s ok to have multiple processes, but to get the most benefit out of Docker, avoid one container being responsible for multiple aspects of your overall application." For sure it obviously much more powerful to devide your complex service into multiple containers. But there are situations where it can be beneficial to go the one container route. Especially for appliances. The GitLab Docker image is my favourite example of a multi process container. It makes deployment of this complex system easy. There is no way for mis-configuration. GitLab retains all control over their appliance. Win-Win.
Make sure that you add daemon off; to you nginx.conf or run it with CMD ["nginx", "-g", "daemon off;"] as per the official nginx image
Then use the following to run both supervisor as service and nginx as foreground process that will prevent the container from exiting
service supervisor start && nginx
In some cases you will need to have more than one process in your container, so forcing the container to have exactly one process won't work and can create more problems in deployment.
So you need to understand the trade-offs and make your decision accordingly.
Since docker engine v1.25 there is an option called init.
Docker-compose included this command as of version 3.7.
So my current CMD when running a container that should run into infinity:
CMD ["sleep", "infinity"]
and then run it using:
docker build
docker run --rm --init app
crf.:
rm docs and init docs
Capture the PID of the ngnix process in a variable (for example $NGNIX_PID) and at the end of the entrypoint file do
wait $NGNIX_PID
In that way, your container should run until ngnix is alive, when ngnix stops, the container stops as well
Along with having something along the lines of : ENTRYPOINT ["tail", "-f", "/dev/null"] in your docker file, you should also run the docker container with -td option. This is particularly useful when the container runs on a remote m/c. Think of it more like you have ssh'ed into a remote m/c having the image and started the container. In this case, when you exit the ssh session, the container will get killed unless it's started with -td option. Sample command for running your image would be: docker run -td <any other additional options> <image name>
This holds good for docker version 20.10.2
There are some cases during development when there is no service yet but you want to simulate it and keep the container alive.
It is very easy to write a bash placeholder that simulates a running service:
while true; do
sleep 100
done
You replace this by something more serious as the development progress.
How about using the supervise form of service if available?
service YOUR_SERVICE supervise
Once supervise is successfully running, it will not exit unless it is
killed or specifically asked to exit.
Saves having to create a supervisord.conf
I have been running docker processes (apps) via
docker run …
But under runit supervision (runit is like daemontools) - so runit ensures that the process stays up, passes signals etc.
Is this reasonable? Docker seems to want to run its own demonization - but it isn't as thorough as runit. Furthermore, when runit restarts the app - a new container is created each time (fine) but it leaves a trace of the old one around - this seems to imply I am doing it in the wrong way.
Should docker not be run this way?
Should I instead set up a container from the image, just once, and then have runit run/supervise that container for all time?
Docker does do some management of daemonized containers: if the system shuts down, then when the Docker daemon starts it will also restart any containers that were running at the time the system shut down. But if the container exits on its own or the kernel (or a user) kills the container while it is running, the Docker daemon won't restart it. In cases where you do want a restart, a process manager makes sense.
I don't know runit so I can't give specific configuration guidance. But you should probably make the process manager communicate with the docker daemon and check to see if a given container id is running (docker ps | grep container_id or equivalent, or use the Docker Remote API directly). If the container has stopped, use Docker to restart it (docker run container_id) instead of running a new container. Or, if you do want a new container each time, then begin with docker run -rm to automatically clean it up when it exits or stops.
If you don't want your process manager to poll docker, you could instead run something that watches docker events.
You can get the container_id when you start the container as the return value of starting a daemon, or you can ask Docker to write this out to a file (docker run -cidfile myfilename, like a PID file)
I hope that helps or helps another runit guru offer more detailed advice.
Yes, I think running docker under runit makes sense. Typically when you start a process there is a way to tell it not to daemonize if it does by default since the normal way to hand-off from the runit run script to a process is via exec on the last line of your run script. For docker this means making sure not to set the -d flag.
For example, with docker you probably want your run script to look something like this:
#!/bin/bash -e
exec 2>&1
exec chpst -u dockeruser docker run -a stdin -a stdout -i ...
Using exec and chpst should resolve most issues with processes not terminating correctly when you bring down a runit service.