I have a Docker container that runs just a Go binary I created that is a http server with Gin framework. I don't use any other Web Server, just Go's internal http server. Inside my Dockerfile at the end of file I have this:
EXPOSE 80
CMD ["/home/project/microservices/backend/engine/cmd/main"]
I use docker-compose to run the container and restart: always for each container. And it works!
But my question is that, if the http server that I created fails due to programming error or something, It will restart? how can I check this? does Docker has tools for this?
I tried go with Supervisord but it has some problems and I was not successful on running it.
I want a workaround to keep the http server inside container always running.
What can I do?
You can try killing the process from the host. Find the process id using something like
ps -aux | grep main
Then kill it using
sudo kill <process id>
Docker will then restart it. By using
docker ps
you should see that the 'status' has changed to something like Up 10 seconds.
I'm trying to get a docker container to never shutdown.
If I run a docker container with the -d flag the container will be run in the background.
For example, can this be done:
Start docker container with -it flags
start entrypoint application
entrypoint application creates 10 other services/processes to run in that same container
entrypoint application terminates
Will the docker container stay up now that the application mentioned in the entrypoint has exited?
Why don't you simply try? AFAIU -d or -it won't affect the container termination. And I guess you understand that starting those 10 processes means you violate the docker's idea of one process per container. Why don't you start 10 containers instead? You can also do that from your start container, and they will keep running even if the starting container will terminate.
You could also give docker docs a try: https://docs.docker.com/articles/host_integration/
-it flag means you want your standard input and output routed to/from container. This is basically and conceptually incompatible with running forever.
In general container will shutdown when entry point exits. if you want to keep container running you should run one (probably the last application) not as a background daemon but in foreground. for example if nginx is the last service you want to run. What you would do as your last line of entry point script (if thats a shell script) should be some thing like this:
nginx -g "daemon off;"
I'm using the default HAProxy Docker image from https://github.com/dockerfile/haproxy
Unfortunately I can't get it to reload my config properly.
If I run
$ sudo docker exec haprox haproxy -f /etc/haproxy/haproxy.cfg -p '$(</var/run/haproxy.pid)' -st '$(</var/run/haproxy.pid)'
it just dumps out the help file. If I run
$ sudo docker exec haprox 'haproxy -f /etc/haproxy/haproxy.cfg -p $(</var/run/haproxy.pid) -st $(</var/run/haproxy.pid)'
I get
2014/12/30 00:03:23 docker-exec: failed to exec: exec: "haproxy -f /etc/haproxy/haproxy.cfg -p $(</var/run/haproxy.pid) -st $(</var/run/haproxy.pid)": stat haproxy -f /etc/haproxy/haproxy.cfg -p $(</var/run/haproxy.pid) -st $(</var/run/haproxy.pid): no such file or directory
Boo. None of those things are what I want. I can run docker exec haprox service haproxy reload - but this ends out spawning several haproxy processes, so when I connect via the unix socket I get one set of information from show stat but I get an entirely different set of information from the http stats page.
I'm trying to set it up so that I can do graceful redeploys of our legacy software, but it does very very bad things with Tomcat sessions, so my only option is to keep existing sessions alive and pinging the same server.
backend legacy
cookie SERVERID insert indirect preserve
server A 123.0.0.123:8080 cookie A check
server B 123.0.0.123:8080 cookie B check
does the trick. I can call up the socket and run set weight legacy/A 0 and it will drain connections from server A.
But (remember that legacy part?) I have to bop my server A/B containers on the head and bring up new ones. I've got my system setup where I'm generating the new config just fine, but when I reload... strange things happen.
As mentioned earlier, it ends out spawning several haproxy processes. I get different information from the stats page and the unix socket. It also appears that the pid file of the process that I'm communicating with in the browser vs. socat are different.
Worst of all, though, is that it will stop http connections with a 503 - and using ab to test will report some dropped connections. This part is not OK.
Old sessions MUST continue to function until the old server goes down/cookies are cleared. It seems like the rest of the Internet is able to do what I'm trying to do... What am I doing wrong here?
You can now reload the config:
docker kill -s HUP haproxy_container_name
More info: https://hub.docker.com/_/haproxy
I know this is an old question and does not help after 6 years! :) But maybe useful for someone!
if you run ps inside the container as follows you will see the container you have linked runs haproxy as pid 1 which cannot be killed without killing the container and also it is run in foreground so without a pid file. If you want to reload run haproxy in the background in your container and make some other process such as supervisor the primary process.
docker exec -it haproxy ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
haproxy 1 0.0 0.2 28988 4576 ? Ss 02:41 0:00 haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
I've seen a bunch of tutorials that seem do the same thing I'm trying to do, but for some reason my Docker containers exit. Basically, I'm setting up a web-server and a few daemons inside a Docker container. I do the final parts of this through a bash script called run-all.sh that I run through CMD in my Dockerfile. run-all.sh looks like this:
service supervisor start
service nginx start
And I start it inside of my Dockerfile as follows:
CMD ["sh", "/root/credentialize_and_run.sh"]
I can see that the services all start up correctly when I run things manually (i.e. getting on to the image with -i -t /bin/bash), and everything looks like it runs correctly when I run the image, but it exits once it finishes starting up my processes. I'd like the processes to run indefinitely, and as far as I understand, the container has to keep running for this to happen. Nevertheless, when I run docker ps -a, I see:
➜ docker_test docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c7706edc4189 some_name/some_repo:blah "sh /root/run-all.sh 8 minutes ago Exited (0) 8 minutes ago grave_jones
What gives? Why is it exiting? I know I could just put a while loop at the end of my bash script to keep it up, but what's the right way to keep it from exiting?
If you are using a Dockerfile, try:
ENTRYPOINT ["tail", "-f", "/dev/null"]
(Obviously this is for dev purposes only, you shouldn't need to keep a container alive unless it's running a process eg. nginx...)
I just had the same problem and I found out that if you are running your container with the -t and -d flag, it keeps running.
docker run -td <image>
Here is what the flags do (according to docker run --help):
-d, --detach=false Run container in background and print container ID
-t, --tty=false Allocate a pseudo-TTY
The most important one is the -t flag. -d just lets you run the container in the background.
This is not really how you should design your Docker containers.
When designing a Docker container, you're supposed to build it such that there is only one process running (i.e. you should have one container for Nginx, and one for supervisord or the app it's running); additionally, that process should run in the foreground.
The container will "exit" when the process itself exits (in your case, that process is your bash script).
However, if you really need (or want) to run multiple service in your Docker container, consider starting from "Docker Base Image", which uses runit as a pseudo-init process (runit will stay online while Nginx and Supervisor run), which will stay in the foreground while your other processes do their thing.
They have substantial docs, so you should be able to achieve what you're trying to do reasonably easily.
you can run plain cat without any arguments as mentioned by bro #Sa'ad to simply keep the container working [actually doing nothing but waiting for user input] (Jenkins' Docker plugin does the same thing)
The reason it exits is because the shell script is run first as PID 1 and when that's complete, PID 1 is gone, and docker only runs while PID 1 is.
You can use supervisor to do everything, if run with the "-n" flag it's told not to daemonize, so it will stay as the first process:
CMD ["/usr/bin/supervisord", "-n"]
And your supervisord.conf:
[supervisord]
nodaemon=true
[program:startup]
priority=1
command=/root/credentialize_and_run.sh
stdout_logfile=/var/log/supervisor/%(program_name)s.log
stderr_logfile=/var/log/supervisor/%(program_name)s.log
autorestart=false
startsecs=0
[program:nginx]
priority=10
command=nginx -g "daemon off;"
stdout_logfile=/var/log/supervisor/nginx.log
stderr_logfile=/var/log/supervisor/nginx.log
autorestart=true
Then you can have as many other processes as you want and supervisor will handle the restarting of them if needed.
That way you could use supervisord in cases where you might need nginx and php5-fpm and it doesn't make much sense to have them apart.
Motivation:
There is nothing wrong in running multiple processes inside of a docker container. If one likes to use docker as a light weight VM - so be it. Others like to split their applications into micro services. Me thinks: A LAMP stack in one container? Just great.
The answer:
Stick with a good base image like the phusion base image. There may be others. Please comment.
And this is yet just another plead for supervisor. Because the phusion base image is providing supervisor besides of some other things like cron and locale setup. Stuff you like to have setup when running such a light weight VM. For what it's worth it also provides ssh connections into the container.
The phusion image itself will just start and keep running if you issue this basic docker run statement:
moin#stretchDEV:~$ docker run -d phusion/baseimage
521e8a12f6ff844fb142d0e2587ed33cdc82b70aa64cce07ed6c0226d857b367
moin#stretchDEV:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS
521e8a12f6ff phusion/baseimage "/sbin/my_init" 12 seconds ago Up 11 seconds
Or dead simple:
If a base image is not for you... For the quick CMD to keep it running I would suppose something like this for bash:
CMD exec /bin/bash -c "trap : TERM INT; sleep infinity & wait"
Or this for busybox:
CMD exec /bin/sh -c "trap : TERM INT; (while true; do sleep 1000; done) & wait"
This is nice, because it will exit immediately on a docker stop.
Just plain sleep or cat will take a few seconds before the container is forcefully killed by docker.
Updates
As response to Charles Desbiens concerning running multiple processes in one container:
This is an opinion. And the docs are pointing in this direction. A quote: "It’s ok to have multiple processes, but to get the most benefit out of Docker, avoid one container being responsible for multiple aspects of your overall application." For sure it obviously much more powerful to devide your complex service into multiple containers. But there are situations where it can be beneficial to go the one container route. Especially for appliances. The GitLab Docker image is my favourite example of a multi process container. It makes deployment of this complex system easy. There is no way for mis-configuration. GitLab retains all control over their appliance. Win-Win.
Make sure that you add daemon off; to you nginx.conf or run it with CMD ["nginx", "-g", "daemon off;"] as per the official nginx image
Then use the following to run both supervisor as service and nginx as foreground process that will prevent the container from exiting
service supervisor start && nginx
In some cases you will need to have more than one process in your container, so forcing the container to have exactly one process won't work and can create more problems in deployment.
So you need to understand the trade-offs and make your decision accordingly.
Since docker engine v1.25 there is an option called init.
Docker-compose included this command as of version 3.7.
So my current CMD when running a container that should run into infinity:
CMD ["sleep", "infinity"]
and then run it using:
docker build
docker run --rm --init app
crf.:
rm docs and init docs
Capture the PID of the ngnix process in a variable (for example $NGNIX_PID) and at the end of the entrypoint file do
wait $NGNIX_PID
In that way, your container should run until ngnix is alive, when ngnix stops, the container stops as well
Along with having something along the lines of : ENTRYPOINT ["tail", "-f", "/dev/null"] in your docker file, you should also run the docker container with -td option. This is particularly useful when the container runs on a remote m/c. Think of it more like you have ssh'ed into a remote m/c having the image and started the container. In this case, when you exit the ssh session, the container will get killed unless it's started with -td option. Sample command for running your image would be: docker run -td <any other additional options> <image name>
This holds good for docker version 20.10.2
There are some cases during development when there is no service yet but you want to simulate it and keep the container alive.
It is very easy to write a bash placeholder that simulates a running service:
while true; do
sleep 100
done
You replace this by something more serious as the development progress.
How about using the supervise form of service if available?
service YOUR_SERVICE supervise
Once supervise is successfully running, it will not exit unless it is
killed or specifically asked to exit.
Saves having to create a supervisord.conf
I have been running docker processes (apps) via
docker run …
But under runit supervision (runit is like daemontools) - so runit ensures that the process stays up, passes signals etc.
Is this reasonable? Docker seems to want to run its own demonization - but it isn't as thorough as runit. Furthermore, when runit restarts the app - a new container is created each time (fine) but it leaves a trace of the old one around - this seems to imply I am doing it in the wrong way.
Should docker not be run this way?
Should I instead set up a container from the image, just once, and then have runit run/supervise that container for all time?
Docker does do some management of daemonized containers: if the system shuts down, then when the Docker daemon starts it will also restart any containers that were running at the time the system shut down. But if the container exits on its own or the kernel (or a user) kills the container while it is running, the Docker daemon won't restart it. In cases where you do want a restart, a process manager makes sense.
I don't know runit so I can't give specific configuration guidance. But you should probably make the process manager communicate with the docker daemon and check to see if a given container id is running (docker ps | grep container_id or equivalent, or use the Docker Remote API directly). If the container has stopped, use Docker to restart it (docker run container_id) instead of running a new container. Or, if you do want a new container each time, then begin with docker run -rm to automatically clean it up when it exits or stops.
If you don't want your process manager to poll docker, you could instead run something that watches docker events.
You can get the container_id when you start the container as the return value of starting a daemon, or you can ask Docker to write this out to a file (docker run -cidfile myfilename, like a PID file)
I hope that helps or helps another runit guru offer more detailed advice.
Yes, I think running docker under runit makes sense. Typically when you start a process there is a way to tell it not to daemonize if it does by default since the normal way to hand-off from the runit run script to a process is via exec on the last line of your run script. For docker this means making sure not to set the -d flag.
For example, with docker you probably want your run script to look something like this:
#!/bin/bash -e
exec 2>&1
exec chpst -u dockeruser docker run -a stdin -a stdout -i ...
Using exec and chpst should resolve most issues with processes not terminating correctly when you bring down a runit service.