I'm trying to run a service that does cron tasks using celery and it is very critical for our business. That certain container that runs the celery sometimes exits unexpectedly and I'm not being notified of the exit as well.
I have two questions:
How can I be notified via e-mail if one of my container exits?
How to do a disaster recovery for example if that container exits or stops another container will start and continue the process of that exited container
you can start the docker container with the parameter:
--restart="always"
for example:
docker run --restart="always" <IMAGE>
for your first Q i would suggest to look into that link:
How to programmatically monitor if a docker container exited?
Related
I have a container with one Node.js script which is launched with CMD npm start. The script runs, does some work, and exits. The node process exits because no work is pending. The npm start exits successfully. The container then stops.
I run this container on a Synology NAS from a cronjob via docker start xxxx. When it finishes, I get an alert Docker container xxxx stopped unexpectedly from their alert system. docker container ls -a shows its status as Exited (0) 5 hours ago. If I monitor docker events I see the event die with exitCode=0
It seems like I need to signal to the system that the exit is expected by producing a stop event instead of a die event. Is that something I can do in my image or on the docker start command line?
The Synology Docker package will generate the notification Docker container xxxx stopped unexpectedly when the following two conditions are met:
The container exits with a die docker event (you can see this happen by monitoring docker events when the container exits). This is any case where the main process in the container exits on its own. The exitCode does not matter.
The container is considered "enabled" by the Synology Docker GUI. This information is stored in /var/packages/Docker/etc/container_name.config:
{
"enabled" : true,
"exporting" : false,
"id" : "dbee87466fb70ea26cd9845fd79af16d793dc64d9453e4eba43430594ab4fa9b",
"image" : "busybox",
"is_ddsm" : false,
"is_package" : false,
"name" : "musing_cori",
"shortcut" : {
"enable_shortcut" : false,
"enable_status_page" : false,
"enable_web_page" : false,
"web_page_url" : ""
}
}
How to enable/disable containers with Synology's Docker GUI
Containers are automatically enabled if you start them from the GUI. All of these things will cause the container to become "enabled" and start notifying on exit:
Sliding the "toggle switch" in the container view to "on"
Using Action start on the container.
Opening the container detail panel and clicking "start"
This is probably how your container ended up "enabled" and why it is now notifying whenever it exits. Containers created with docker run -d ... do not start out enabled, and will not initially warn on exit. This is probably why things like docker run -it --rm busybox and other ephemeral containers do not cause notifications.
Containers can be disabled if you stop them while they are running. There appears to be no way to disable a container which is currently stopped. So to disable a container you must start it and then stop it before it exits on its own:
Slide the toggle switch on then off as soon as it will let you.
Use Action start and then stop as soon as it will let you (this is hard because of the extra click if your container is very shortlived).
Open the controller detail panel, click start, and then as soon as "stop" is not grayed out, click "stop".
Check your work by looking at /var/packages/Docker/etc/container_name.config.
Another option for stopping/starting the container without the notifications is to do it via the Synology Web API.
To stop a container:
synowebapi --exec api=SYNO.Docker.Container version=1 method=stop name="CONTAINER_NAME"
Then to restart it:
synowebapi --exec api=SYNO.Docker.Container version=1 method=start name="CONTAINER_NAME"
Notes:
The commands need to be run as root
You will get a warning [Line 255] Not a json value: CONTAINER_NAME but the commands work and give a response message indicating "success" : true
I don't really have any more information on it as I stumbled across it in a reddit post and there's not a lot to back it up, but it's working for me on DSM 7.1.1-42962 and I'm using it in a scheduled task.
Source and referenced links:
A Reddit post with the commands
Linked GitHub page showing the commands in use
Linked Synology Developer's Guide for DSM Login Web API
I'm not familiar with Synology so I'm not sure which component is raising the "alert" you mention, but I guess this is just a warning and not an error, because:
an exit status of 0 is very fine from a POSIX perspective;
a "die" docker event also seems quite common, e.g. running docker events then docker run --rm -it debian bash -c "echo Hello" yields the same event (while a "kill" event would be more dubious).
So maybe you get one such warning just because Synology assumes a container should be running for a long time?
Anyway, here are a couple of remarks related to your question:
Is the image/container you run really ephemeral? (regarding the data the container handles) because if this the case, instead of doing docker start container_name, you might prefer using docker run --rm -i image_name … or docker run --rm -d -i image_name …. (In this case thanks to --rm, the container removal will be automatically triggered when the container stops.)
Even if the setup you mention sounds quite reasonable for a cron job (namely, the fact that your container stops early and automatically), you might be interested in this other SO answer that gives further details on how to catch the signals raised by docker stop etc.
I'm new in docker.
What is the difference between these?
docker run 'an image'
docker-compose run 'something'
docker-compose start 'docker-compose.yml'
docker-compose up 'docker-compose.yml'
Thanks in advance.
https://docs.docker.com/compose/faq/#whats-the-difference-between-up-run-and-start
What’s the difference between up, run, and start?
Typically, you want docker-compose up. Use up to start or restart all the services defined in a docker-compose.yml. In the default “attached” mode, you see all the logs from all the containers. In “detached” mode (-d), Compose exits after starting the containers, but the containers continue to run in the background.
The docker-compose run command is for running “one-off” or “adhoc” tasks. It requires the service name you want to run and only starts containers for services that the running service depends on. Use run to run tests or perform an administrative task such as removing or adding data to a data volume container. The run command acts like docker run -ti in that it opens an interactive terminal to the container and returns an exit status matching the exit status of the process in the container.
The docker-compose start command is useful only to restart containers that were previously created, but were stopped. It never creates new containers.
Also: https://docs.docker.com/compose/reference/
I am running a docker container which is trying to access a port in another docker container. Both of these are running are configured together to run on the same network. But as soon as I start this container it gets killed and doesn't throw any error. There are no error logs. I also tried using docker inspect but couldn't find much.
PS: I am a newbie docker user.
Following from OP comment w/ ENTRYPOINT
ENTRYPOINT /configure.sh && bash
Answer
Given your ENTRYPOINT the container will always exit since the process is bash. You need to have a continuously running process in the foreground for the container to stay running i.e. an application daemon.
I am new to docker, hence may be missing a simple piece. Here is my scenario. I started a container with command 'docker run -it ubuntu:14.04'. Then with Ctrl+P+Q, I exited such that the container keeps running. I verified with docker ps, and saw the container running. Then I again entered the container with 'docker exec -it bash. This took me inside the container again. Now on typing 'exit' command, I come out of the container, but the container is still in running mode. Normally with exit command, the container stops. Any idea why this is happening?
The container's running status is tied to the initial process that it was created for/with.
If you do docker run then this will create a new container with some inital process. When that process terminates, the whole container is stopped. If that initial process was bash, and you exit it, then this terminates the container itself.
docker exec starts a new process inside of the running container. When that process terminates, the container still keeps running.
Typing exit into an interactive bash shell will just exit that shell. It will not affect other processes running inside the same container (just like closing one terminal window in your host OS does not affect any other processes).
With the exit command in your case, the container stops only the /bin/bash/ executable. Probably some other application like NGINX or Apache is running inside the container and does not let it shut down.
I have been running docker processes (apps) via
docker run …
But under runit supervision (runit is like daemontools) - so runit ensures that the process stays up, passes signals etc.
Is this reasonable? Docker seems to want to run its own demonization - but it isn't as thorough as runit. Furthermore, when runit restarts the app - a new container is created each time (fine) but it leaves a trace of the old one around - this seems to imply I am doing it in the wrong way.
Should docker not be run this way?
Should I instead set up a container from the image, just once, and then have runit run/supervise that container for all time?
Docker does do some management of daemonized containers: if the system shuts down, then when the Docker daemon starts it will also restart any containers that were running at the time the system shut down. But if the container exits on its own or the kernel (or a user) kills the container while it is running, the Docker daemon won't restart it. In cases where you do want a restart, a process manager makes sense.
I don't know runit so I can't give specific configuration guidance. But you should probably make the process manager communicate with the docker daemon and check to see if a given container id is running (docker ps | grep container_id or equivalent, or use the Docker Remote API directly). If the container has stopped, use Docker to restart it (docker run container_id) instead of running a new container. Or, if you do want a new container each time, then begin with docker run -rm to automatically clean it up when it exits or stops.
If you don't want your process manager to poll docker, you could instead run something that watches docker events.
You can get the container_id when you start the container as the return value of starting a daemon, or you can ask Docker to write this out to a file (docker run -cidfile myfilename, like a PID file)
I hope that helps or helps another runit guru offer more detailed advice.
Yes, I think running docker under runit makes sense. Typically when you start a process there is a way to tell it not to daemonize if it does by default since the normal way to hand-off from the runit run script to a process is via exec on the last line of your run script. For docker this means making sure not to set the -d flag.
For example, with docker you probably want your run script to look something like this:
#!/bin/bash -e
exec 2>&1
exec chpst -u dockeruser docker run -a stdin -a stdout -i ...
Using exec and chpst should resolve most issues with processes not terminating correctly when you bring down a runit service.