I am using RabbitMQ in a project and am running my integration tests against it. Since the tests need to be independent from each other, I would like to reset the RabbitMQ instance before each test and currently solve this by restarting the (automatically created) RabbitMQ docker container. However, this is extremely slow (for integration tests).
I know from this answer that it's possible to reset the rabbitmq-instance with rabbitmqctl stop && rabbitmqctl reset && rabbitmqctl start - but in case of the docker-image, the stop-signal kills the main container process (which is rabbitmq-server), which in turn leads to dockerd killing the complete container.
The only solution I found so far is running the management-api-plugin, iterating over all queues, exchanges, policies, etc. and deleting them through that - which in turn takes a while as well and requires the management-plugin to run.
Is it possible to reset a running rabbitmq-node programmatically via AMQP, some other API-endpoint or running a command, without stopping it first?
The answer you're referring to is correct in that you should be using stop_app, not stop like in your message.
There's an important difference between the two:
stop:
...stops RabbitMQ and its runtime (Erlang VM)
stop_app:
...stops the RabbitMQ application, leaving the runtime (Erlang VM) running
Because in rabbitmq container process containing Erlang VM is PID = 1, stopping it will obviously cause container to stop. Luckly, rabbitmq authors added stop_app command specifically to improve user experience related to testing.
The code from the answer you're referring to should work just fine. Here's the same code as a one-liner:
docker exec my_queue sh -c "rabbitmqctl stop_app; rabbitmqctl reset; rabbitmqctl start_app"
The output will look like this:
$ docker exec my_queue sh -c "rabbitmqctl stop_app; rabbitmqctl reset; rabbitmqctl start_app"
Stopping rabbit application on node rabbit#40420e95dcee
Resetting node rabbit#40420e95dcee
Starting node rabbit#40420e95dcee
$
Related
I am using Docker version 17.09.0-ce, and I see that containers are marked as unhealthy. Is there an option to get the container restart instead of keeping the container as unhealthy?
Restarting of unhealty container feature was in the original PR (https://github.com/moby/moby/pull/22719), but was removed after a discussion and considered to be done later as enhancement of RestartPolicy.
At this moment you can use this workaround to automatically restarting unhealty containers: https://hub.docker.com/r/willfarrell/autoheal/
Here is a sample compose file:
version: '2'
services:
autoheal:
restart: always
image: willfarrell/autoheal
environment:
- AUTOHEAL_CONTAINER_LABEL=all
volumes:
- /var/run/docker.sock:/var/run/docker.sock
Simply execute docker-compose up -d on this
You can restart automatically an unhealthy container by setting a smart HEALTHCHECK and a proper restart policy.
The Docker restart policy should be one of always or unless-stopped.
The HEALTHCHECK instead should implement a logic that kills the container when it's unhealthy.
In the following example I used curl with its internal retry mechanism and piped it (in case of failure/service unhealthy) to the kill command.
HEALTHCHECK --interval=5m --timeout=2m --start-period=45s \
CMD curl -f --retry 6 --max-time 5 --retry-delay 10 --retry-max-time 60 "http://localhost:8080/health" || bash -c 'kill -s 15 -1 && (sleep 10; kill -s 9 -1)'
The important step to understand here is that the retry logic is self-contained in the curl command, the Docker retry here actually is mandatory but useless. Then if the curl HTTP request fails 3 times, then kill is executed. First it sends a SIGTERM to all the processes in the container, to allow them to gracefully stop, then after 10 seconds it sends a SIGKILL to completely kill all the processes in the container. It must be noted that when the PID1 of a container dies, then the container itself dies and the restart policy is invoked.
kill docs: https://linux.die.net/man/1/kill
curl docs: https://curl.haxx.se/docs/manpage.html
docker restart docs: https://docs.docker.com/compose/compose-file/compose-file-v2/#restart
Gotchas: kill behaves differently in bash than in sh. In bash you can use -1 to signal all the processes with PID greater than 1 to die.
For standalone containers, Docker does not have native integration to restart the container on health check failure though we can achieve the same using Docker events and a script. Health check is better integrated with Swarm. With health check integrated to Swarm, when a container in a service is unhealthy, Swarm automatically shuts down the unhealthy container and starts a new container to maintain the container count as specified in the replica count of a service.
You can try put in your Dockerfile something like this:
HEALTHCHECK --interval=5s --timeout=2s CMD curl --fail http://localhost || kill 1
Don't forget --restart always option.
kill 1 will kill process with pid 1 in container and force container exit. Usually the process started by CMD or ENTRYPOINT has pid 1.
Unfortunally, this method likely don't change container's state to unhealthy, so be careful with it.
Unhealthy docker containers may be restarted with simple crontab rule:
* * * * * docker ps -f health=unhealthy --format "docker restart {{.ID}}" | sh
Docker has a couple of ways to get details on container health. You can configure health checks and how often they run. Also, health checks can be run on applications running inside a container, like http (this would use curl --fail option.) You can view the health_status event to get details.
For detailed information on an unhealthy container the inspect command comes in handy, docker inspect --format='{{json .State.Health}}' container-name (see https://blog.newrelic.com/2016/08/24/docker-health-check-instruction/ for more details.)
You should resolve the error condition causing the "unhealthy" tag (anytime the health check command runs and gets an exit code of 1) first. This may or may not require that Docker restart the container, depending on the error. If you are starting/restarting your containers automatically, then either trapping the start errors or logging them and the health check status can help address errors quickly. Check the link if you are interested in auto start.
According to https://codeblog.dotsandbrackets.com/docker-health-check/
Create container and add " restart: always".
In the use of healthcheck, pay attention to the following points:
For standalone containers, Docker does not have native integration to restart the container on health check failure though we can achieve the same using Docker events and a script. Health check is better integrated with Swarm. With health check integrated to Swarm, when a container in a service is unhealthy, Swarm automatically shuts down the unhealthy container and starts a new container to maintain the container count as specified in the replica count of a service.
How would you run a daemon or background process in Docker? I've seen some suggestions, like this answer that launches supervisor from CMD.
However, I'm trying to test a server configuration tool that connects via SSH. So I need to launch the SSH daemon in the background, and then run my tool.sh to test connecting via SSH to its own container. I need to monitor my tool's output in order to verify it's working. What's the best way to accomplish this?
Is there any way to make a RUN command run in the background, like RUN /usr/sbin/sshd -D & or would I have to have some wrapper script launched from CMD that does something like this?
#!/bin/bash
/usr/sbin/sshd -D
tool.sh
You can run a daemon inside a docker container the same as you would on a bare metal linux machine. The only hard part is getting it to start without the nice runlevel scripts to help.
How about this:
#!/bin/sh
function run_script() {
ssh_pids=0
while [ ${ssh_pids} -lt 1 ]; do
sleep 5
ssh_pids=$(pgrep sshd | wc -l)
done
test.sh
}
run_script &
sshd -D > /dev/null 2>&1
I've used this trick before to do what you describe, and it's worked OK. It will just background the call to run_script and proceed to start SSHD in non-daemon mode, piping it to /dev/null. Meanwhile, run_script polls for sshd; when it finds it, it quits polling and runs test.sh, which should still have the terminal as it's stdout. you'll probably need to use some external kill signal to stop the whole thing, once test.sh is done.
If you don't like this tomfoolery, the other option would be to do as you described: write a wrapper script to use as the CMD/ENTRYPOINT, and have it start SSHD without the debug flag, and then start test.sh.
The advantage of doing it with the script I posted is that the container will stick around after test.sh is finished, so you can log in and poke around, while also making your script wait until the daemon is running.
Like most docker users, I periodically need to connect to a running container and execute various arbitrary commands via bash.
I'm using 17.06-CE with an ubuntu 16.04 image, and as far as I understand, the only way to do this without installing ssh into the container is via docker exec -it <container_name> bash
However, as is well-documented, for each bash shell process you generate, you leave a zombie process behind when your connection is interrupted. If you connect to your container often, you end up with 1000s of idle shells -a most undesirable outcome!
How can I ensure these zombie shell processes are killed upon disconnection -as they would be over ssh?
One way is to make sure the linux init process runs in your container.
In recent versions of docker there is an --init option to docker run that should do this. This uses tini to run init which can also be used in previous versions.
Another option is something like the phusion-baseimage project that provides a base docker image with this capability and many others (might be overkill).
I would like to have a RUnit service to supervise a set of containers launched by docker-compose tool, here's my the runit script:
In /etc/sv/app/run
#!/bin/bash
exec 2>&1
APP_HOME=/home/myapp
source $APP_HOME/env.sh
exec docker-compose -f $APP_HOME/docker-compose.yml up
Here's what I have then:
sv start app - launches the docker-compose thing just fine
sv stop app - stops docker-compose process itself but for unknown reason it leaves the containers running
Is there any chance to have the stop command to stop containers as well? I thought that is what docker-compose should do when it gets stopped by RUnit.
I'm not familiar with docker (yet) but I have familiarity with runit.
When you issue sv stop app you are actually telling runsvdir to signal the runsv for your docker launch to tear down the process. If you need something to signal the container to shut down, it won't happen because runsv will haul off and kill any child processes that are attached. You may wish to read up on ./finish scripts, which are tasked with cleaning up things.
I have been running docker processes (apps) via
docker run …
But under runit supervision (runit is like daemontools) - so runit ensures that the process stays up, passes signals etc.
Is this reasonable? Docker seems to want to run its own demonization - but it isn't as thorough as runit. Furthermore, when runit restarts the app - a new container is created each time (fine) but it leaves a trace of the old one around - this seems to imply I am doing it in the wrong way.
Should docker not be run this way?
Should I instead set up a container from the image, just once, and then have runit run/supervise that container for all time?
Docker does do some management of daemonized containers: if the system shuts down, then when the Docker daemon starts it will also restart any containers that were running at the time the system shut down. But if the container exits on its own or the kernel (or a user) kills the container while it is running, the Docker daemon won't restart it. In cases where you do want a restart, a process manager makes sense.
I don't know runit so I can't give specific configuration guidance. But you should probably make the process manager communicate with the docker daemon and check to see if a given container id is running (docker ps | grep container_id or equivalent, or use the Docker Remote API directly). If the container has stopped, use Docker to restart it (docker run container_id) instead of running a new container. Or, if you do want a new container each time, then begin with docker run -rm to automatically clean it up when it exits or stops.
If you don't want your process manager to poll docker, you could instead run something that watches docker events.
You can get the container_id when you start the container as the return value of starting a daemon, or you can ask Docker to write this out to a file (docker run -cidfile myfilename, like a PID file)
I hope that helps or helps another runit guru offer more detailed advice.
Yes, I think running docker under runit makes sense. Typically when you start a process there is a way to tell it not to daemonize if it does by default since the normal way to hand-off from the runit run script to a process is via exec on the last line of your run script. For docker this means making sure not to set the -d flag.
For example, with docker you probably want your run script to look something like this:
#!/bin/bash -e
exec 2>&1
exec chpst -u dockeruser docker run -a stdin -a stdout -i ...
Using exec and chpst should resolve most issues with processes not terminating correctly when you bring down a runit service.