I'm using the default HAProxy Docker image from https://github.com/dockerfile/haproxy
Unfortunately I can't get it to reload my config properly.
If I run
$ sudo docker exec haprox haproxy -f /etc/haproxy/haproxy.cfg -p '$(</var/run/haproxy.pid)' -st '$(</var/run/haproxy.pid)'
it just dumps out the help file. If I run
$ sudo docker exec haprox 'haproxy -f /etc/haproxy/haproxy.cfg -p $(</var/run/haproxy.pid) -st $(</var/run/haproxy.pid)'
I get
2014/12/30 00:03:23 docker-exec: failed to exec: exec: "haproxy -f /etc/haproxy/haproxy.cfg -p $(</var/run/haproxy.pid) -st $(</var/run/haproxy.pid)": stat haproxy -f /etc/haproxy/haproxy.cfg -p $(</var/run/haproxy.pid) -st $(</var/run/haproxy.pid): no such file or directory
Boo. None of those things are what I want. I can run docker exec haprox service haproxy reload - but this ends out spawning several haproxy processes, so when I connect via the unix socket I get one set of information from show stat but I get an entirely different set of information from the http stats page.
I'm trying to set it up so that I can do graceful redeploys of our legacy software, but it does very very bad things with Tomcat sessions, so my only option is to keep existing sessions alive and pinging the same server.
backend legacy
cookie SERVERID insert indirect preserve
server A 123.0.0.123:8080 cookie A check
server B 123.0.0.123:8080 cookie B check
does the trick. I can call up the socket and run set weight legacy/A 0 and it will drain connections from server A.
But (remember that legacy part?) I have to bop my server A/B containers on the head and bring up new ones. I've got my system setup where I'm generating the new config just fine, but when I reload... strange things happen.
As mentioned earlier, it ends out spawning several haproxy processes. I get different information from the stats page and the unix socket. It also appears that the pid file of the process that I'm communicating with in the browser vs. socat are different.
Worst of all, though, is that it will stop http connections with a 503 - and using ab to test will report some dropped connections. This part is not OK.
Old sessions MUST continue to function until the old server goes down/cookies are cleared. It seems like the rest of the Internet is able to do what I'm trying to do... What am I doing wrong here?
You can now reload the config:
docker kill -s HUP haproxy_container_name
More info: https://hub.docker.com/_/haproxy
I know this is an old question and does not help after 6 years! :) But maybe useful for someone!
if you run ps inside the container as follows you will see the container you have linked runs haproxy as pid 1 which cannot be killed without killing the container and also it is run in foreground so without a pid file. If you want to reload run haproxy in the background in your container and make some other process such as supervisor the primary process.
docker exec -it haproxy ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
haproxy 1 0.0 0.2 28988 4576 ? Ss 02:41 0:00 haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
Related
Im restarting my nginx in a docker container in a sh script and i would like to make sure i retrieve the right pid when doing so.
So nginx is already running and im doing nginx -s reload to restart the service.
If I do nginx -s reload & nginx_pid=$!, will nginx_pid indeed correspond to nginx pid ?
Thank you
Best
A Docker container only normally runs a single process, and inside the container its process ID is 1. In an nginx container I'd expect the Nginx pid to always be 1. The server won't be process 1 only if there's a separate init process (tini is a common one) or if your process is launched by a shell script without using exec; the Docker Hub nginx image does neither of these things.
In the fragment you show, nginx -s reload sends a command to the main Nginx server and exits (and it doesn't need to be run in the background). $! is the pid of that reload command and not the Nginx server.
In typical use you never need to know process IDs inside containers, manually send signals to processes, or otherwise be "inside" a container. If your application code calls fork(2) or getppid(2) it can know another process's pid but even these are somewhat unusual. If you need to restart a container process it's usually simplest to delete and recreate the entire container.
I am using Docker version 17.09.0-ce, and I see that containers are marked as unhealthy. Is there an option to get the container restart instead of keeping the container as unhealthy?
Restarting of unhealty container feature was in the original PR (https://github.com/moby/moby/pull/22719), but was removed after a discussion and considered to be done later as enhancement of RestartPolicy.
At this moment you can use this workaround to automatically restarting unhealty containers: https://hub.docker.com/r/willfarrell/autoheal/
Here is a sample compose file:
version: '2'
services:
autoheal:
restart: always
image: willfarrell/autoheal
environment:
- AUTOHEAL_CONTAINER_LABEL=all
volumes:
- /var/run/docker.sock:/var/run/docker.sock
Simply execute docker-compose up -d on this
You can restart automatically an unhealthy container by setting a smart HEALTHCHECK and a proper restart policy.
The Docker restart policy should be one of always or unless-stopped.
The HEALTHCHECK instead should implement a logic that kills the container when it's unhealthy.
In the following example I used curl with its internal retry mechanism and piped it (in case of failure/service unhealthy) to the kill command.
HEALTHCHECK --interval=5m --timeout=2m --start-period=45s \
CMD curl -f --retry 6 --max-time 5 --retry-delay 10 --retry-max-time 60 "http://localhost:8080/health" || bash -c 'kill -s 15 -1 && (sleep 10; kill -s 9 -1)'
The important step to understand here is that the retry logic is self-contained in the curl command, the Docker retry here actually is mandatory but useless. Then if the curl HTTP request fails 3 times, then kill is executed. First it sends a SIGTERM to all the processes in the container, to allow them to gracefully stop, then after 10 seconds it sends a SIGKILL to completely kill all the processes in the container. It must be noted that when the PID1 of a container dies, then the container itself dies and the restart policy is invoked.
kill docs: https://linux.die.net/man/1/kill
curl docs: https://curl.haxx.se/docs/manpage.html
docker restart docs: https://docs.docker.com/compose/compose-file/compose-file-v2/#restart
Gotchas: kill behaves differently in bash than in sh. In bash you can use -1 to signal all the processes with PID greater than 1 to die.
For standalone containers, Docker does not have native integration to restart the container on health check failure though we can achieve the same using Docker events and a script. Health check is better integrated with Swarm. With health check integrated to Swarm, when a container in a service is unhealthy, Swarm automatically shuts down the unhealthy container and starts a new container to maintain the container count as specified in the replica count of a service.
You can try put in your Dockerfile something like this:
HEALTHCHECK --interval=5s --timeout=2s CMD curl --fail http://localhost || kill 1
Don't forget --restart always option.
kill 1 will kill process with pid 1 in container and force container exit. Usually the process started by CMD or ENTRYPOINT has pid 1.
Unfortunally, this method likely don't change container's state to unhealthy, so be careful with it.
Unhealthy docker containers may be restarted with simple crontab rule:
* * * * * docker ps -f health=unhealthy --format "docker restart {{.ID}}" | sh
Docker has a couple of ways to get details on container health. You can configure health checks and how often they run. Also, health checks can be run on applications running inside a container, like http (this would use curl --fail option.) You can view the health_status event to get details.
For detailed information on an unhealthy container the inspect command comes in handy, docker inspect --format='{{json .State.Health}}' container-name (see https://blog.newrelic.com/2016/08/24/docker-health-check-instruction/ for more details.)
You should resolve the error condition causing the "unhealthy" tag (anytime the health check command runs and gets an exit code of 1) first. This may or may not require that Docker restart the container, depending on the error. If you are starting/restarting your containers automatically, then either trapping the start errors or logging them and the health check status can help address errors quickly. Check the link if you are interested in auto start.
According to https://codeblog.dotsandbrackets.com/docker-health-check/
Create container and add " restart: always".
In the use of healthcheck, pay attention to the following points:
For standalone containers, Docker does not have native integration to restart the container on health check failure though we can achieve the same using Docker events and a script. Health check is better integrated with Swarm. With health check integrated to Swarm, when a container in a service is unhealthy, Swarm automatically shuts down the unhealthy container and starts a new container to maintain the container count as specified in the replica count of a service.
I am using Docker version 17.09.0-ce, and I see that containers are marked as unhealthy. Is there an option to get the container restart instead of keeping the container as unhealthy?
Restarting of unhealty container feature was in the original PR (https://github.com/moby/moby/pull/22719), but was removed after a discussion and considered to be done later as enhancement of RestartPolicy.
At this moment you can use this workaround to automatically restarting unhealty containers: https://hub.docker.com/r/willfarrell/autoheal/
Here is a sample compose file:
version: '2'
services:
autoheal:
restart: always
image: willfarrell/autoheal
environment:
- AUTOHEAL_CONTAINER_LABEL=all
volumes:
- /var/run/docker.sock:/var/run/docker.sock
Simply execute docker-compose up -d on this
You can restart automatically an unhealthy container by setting a smart HEALTHCHECK and a proper restart policy.
The Docker restart policy should be one of always or unless-stopped.
The HEALTHCHECK instead should implement a logic that kills the container when it's unhealthy.
In the following example I used curl with its internal retry mechanism and piped it (in case of failure/service unhealthy) to the kill command.
HEALTHCHECK --interval=5m --timeout=2m --start-period=45s \
CMD curl -f --retry 6 --max-time 5 --retry-delay 10 --retry-max-time 60 "http://localhost:8080/health" || bash -c 'kill -s 15 -1 && (sleep 10; kill -s 9 -1)'
The important step to understand here is that the retry logic is self-contained in the curl command, the Docker retry here actually is mandatory but useless. Then if the curl HTTP request fails 3 times, then kill is executed. First it sends a SIGTERM to all the processes in the container, to allow them to gracefully stop, then after 10 seconds it sends a SIGKILL to completely kill all the processes in the container. It must be noted that when the PID1 of a container dies, then the container itself dies and the restart policy is invoked.
kill docs: https://linux.die.net/man/1/kill
curl docs: https://curl.haxx.se/docs/manpage.html
docker restart docs: https://docs.docker.com/compose/compose-file/compose-file-v2/#restart
Gotchas: kill behaves differently in bash than in sh. In bash you can use -1 to signal all the processes with PID greater than 1 to die.
For standalone containers, Docker does not have native integration to restart the container on health check failure though we can achieve the same using Docker events and a script. Health check is better integrated with Swarm. With health check integrated to Swarm, when a container in a service is unhealthy, Swarm automatically shuts down the unhealthy container and starts a new container to maintain the container count as specified in the replica count of a service.
You can try put in your Dockerfile something like this:
HEALTHCHECK --interval=5s --timeout=2s CMD curl --fail http://localhost || kill 1
Don't forget --restart always option.
kill 1 will kill process with pid 1 in container and force container exit. Usually the process started by CMD or ENTRYPOINT has pid 1.
Unfortunally, this method likely don't change container's state to unhealthy, so be careful with it.
Unhealthy docker containers may be restarted with simple crontab rule:
* * * * * docker ps -f health=unhealthy --format "docker restart {{.ID}}" | sh
Docker has a couple of ways to get details on container health. You can configure health checks and how often they run. Also, health checks can be run on applications running inside a container, like http (this would use curl --fail option.) You can view the health_status event to get details.
For detailed information on an unhealthy container the inspect command comes in handy, docker inspect --format='{{json .State.Health}}' container-name (see https://blog.newrelic.com/2016/08/24/docker-health-check-instruction/ for more details.)
You should resolve the error condition causing the "unhealthy" tag (anytime the health check command runs and gets an exit code of 1) first. This may or may not require that Docker restart the container, depending on the error. If you are starting/restarting your containers automatically, then either trapping the start errors or logging them and the health check status can help address errors quickly. Check the link if you are interested in auto start.
According to https://codeblog.dotsandbrackets.com/docker-health-check/
Create container and add " restart: always".
In the use of healthcheck, pay attention to the following points:
For standalone containers, Docker does not have native integration to restart the container on health check failure though we can achieve the same using Docker events and a script. Health check is better integrated with Swarm. With health check integrated to Swarm, when a container in a service is unhealthy, Swarm automatically shuts down the unhealthy container and starts a new container to maintain the container count as specified in the replica count of a service.
I have a server-side application running in a Docker container. One of the processes in it has hung and needs to be killed (the application will then spawn another process to replace it).
Is there any way to kill that process without stopping the Docker container?
It is not possible with Docker for now, but seems to be scheduled for 0.8, see issue #1228 here.
It is however possible to use lxc-attach to run a shell in an existing container (seen in the above issue comments) and you then can kill your hung process from there :
$ lxc-attach -n FULLCONTAINERID /bin/bash
You can get the FULLCONTAINERID with docker ps --no-trunc=true :
root#turmes /home/zoobab [35]# docker ps --no-trunc=true
CONTAINER ID IMAGE > COMMAND C STATUS PORTS NAMES
2741d88a51148e66d7b2b44d8c1cc6ed7d1515f370be5d00bd003d40cf8d575b zoobab/centos57:latest kamailio -P /var/run/kamailio.pid -m 64 -M 4 -u kamailio -g kamailio -D 1 Up 19 minutes angry_fermat
root#turmes /home/zoobab [36]#
If running Docker 1.3 or newer is not an option, you can still obtain access to a root shell inside the Docker Container using nsenter.
This blog post has all the instructions you need.
Once you have root shell access, you can of course perform any operation you like.
maybe times have changed but as of docker.latest:
docker kill 593690fe0087 killed the CONTAINER ID when I ran docker ps. I had a container there for 2 weeks and only saw it now when the environment wasn't up.
You can do this now in Docker 1.3 using the exec command:
docker exec container_name kill process_name
I have been running docker processes (apps) via
docker run …
But under runit supervision (runit is like daemontools) - so runit ensures that the process stays up, passes signals etc.
Is this reasonable? Docker seems to want to run its own demonization - but it isn't as thorough as runit. Furthermore, when runit restarts the app - a new container is created each time (fine) but it leaves a trace of the old one around - this seems to imply I am doing it in the wrong way.
Should docker not be run this way?
Should I instead set up a container from the image, just once, and then have runit run/supervise that container for all time?
Docker does do some management of daemonized containers: if the system shuts down, then when the Docker daemon starts it will also restart any containers that were running at the time the system shut down. But if the container exits on its own or the kernel (or a user) kills the container while it is running, the Docker daemon won't restart it. In cases where you do want a restart, a process manager makes sense.
I don't know runit so I can't give specific configuration guidance. But you should probably make the process manager communicate with the docker daemon and check to see if a given container id is running (docker ps | grep container_id or equivalent, or use the Docker Remote API directly). If the container has stopped, use Docker to restart it (docker run container_id) instead of running a new container. Or, if you do want a new container each time, then begin with docker run -rm to automatically clean it up when it exits or stops.
If you don't want your process manager to poll docker, you could instead run something that watches docker events.
You can get the container_id when you start the container as the return value of starting a daemon, or you can ask Docker to write this out to a file (docker run -cidfile myfilename, like a PID file)
I hope that helps or helps another runit guru offer more detailed advice.
Yes, I think running docker under runit makes sense. Typically when you start a process there is a way to tell it not to daemonize if it does by default since the normal way to hand-off from the runit run script to a process is via exec on the last line of your run script. For docker this means making sure not to set the -d flag.
For example, with docker you probably want your run script to look something like this:
#!/bin/bash -e
exec 2>&1
exec chpst -u dockeruser docker run -a stdin -a stdout -i ...
Using exec and chpst should resolve most issues with processes not terminating correctly when you bring down a runit service.