I have a container with one Node.js script which is launched with CMD npm start. The script runs, does some work, and exits. The node process exits because no work is pending. The npm start exits successfully. The container then stops.
I run this container on a Synology NAS from a cronjob via docker start xxxx. When it finishes, I get an alert Docker container xxxx stopped unexpectedly from their alert system. docker container ls -a shows its status as Exited (0) 5 hours ago. If I monitor docker events I see the event die with exitCode=0
It seems like I need to signal to the system that the exit is expected by producing a stop event instead of a die event. Is that something I can do in my image or on the docker start command line?
The Synology Docker package will generate the notification Docker container xxxx stopped unexpectedly when the following two conditions are met:
The container exits with a die docker event (you can see this happen by monitoring docker events when the container exits). This is any case where the main process in the container exits on its own. The exitCode does not matter.
The container is considered "enabled" by the Synology Docker GUI. This information is stored in /var/packages/Docker/etc/container_name.config:
{
"enabled" : true,
"exporting" : false,
"id" : "dbee87466fb70ea26cd9845fd79af16d793dc64d9453e4eba43430594ab4fa9b",
"image" : "busybox",
"is_ddsm" : false,
"is_package" : false,
"name" : "musing_cori",
"shortcut" : {
"enable_shortcut" : false,
"enable_status_page" : false,
"enable_web_page" : false,
"web_page_url" : ""
}
}
How to enable/disable containers with Synology's Docker GUI
Containers are automatically enabled if you start them from the GUI. All of these things will cause the container to become "enabled" and start notifying on exit:
Sliding the "toggle switch" in the container view to "on"
Using Action start on the container.
Opening the container detail panel and clicking "start"
This is probably how your container ended up "enabled" and why it is now notifying whenever it exits. Containers created with docker run -d ... do not start out enabled, and will not initially warn on exit. This is probably why things like docker run -it --rm busybox and other ephemeral containers do not cause notifications.
Containers can be disabled if you stop them while they are running. There appears to be no way to disable a container which is currently stopped. So to disable a container you must start it and then stop it before it exits on its own:
Slide the toggle switch on then off as soon as it will let you.
Use Action start and then stop as soon as it will let you (this is hard because of the extra click if your container is very shortlived).
Open the controller detail panel, click start, and then as soon as "stop" is not grayed out, click "stop".
Check your work by looking at /var/packages/Docker/etc/container_name.config.
Another option for stopping/starting the container without the notifications is to do it via the Synology Web API.
To stop a container:
synowebapi --exec api=SYNO.Docker.Container version=1 method=stop name="CONTAINER_NAME"
Then to restart it:
synowebapi --exec api=SYNO.Docker.Container version=1 method=start name="CONTAINER_NAME"
Notes:
The commands need to be run as root
You will get a warning [Line 255] Not a json value: CONTAINER_NAME but the commands work and give a response message indicating "success" : true
I don't really have any more information on it as I stumbled across it in a reddit post and there's not a lot to back it up, but it's working for me on DSM 7.1.1-42962 and I'm using it in a scheduled task.
Source and referenced links:
A Reddit post with the commands
Linked GitHub page showing the commands in use
Linked Synology Developer's Guide for DSM Login Web API
I'm not familiar with Synology so I'm not sure which component is raising the "alert" you mention, but I guess this is just a warning and not an error, because:
an exit status of 0 is very fine from a POSIX perspective;
a "die" docker event also seems quite common, e.g. running docker events then docker run --rm -it debian bash -c "echo Hello" yields the same event (while a "kill" event would be more dubious).
So maybe you get one such warning just because Synology assumes a container should be running for a long time?
Anyway, here are a couple of remarks related to your question:
Is the image/container you run really ephemeral? (regarding the data the container handles) because if this the case, instead of doing docker start container_name, you might prefer using docker run --rm -i image_name … or docker run --rm -d -i image_name …. (In this case thanks to --rm, the container removal will be automatically triggered when the container stops.)
Even if the setup you mention sounds quite reasonable for a cron job (namely, the fact that your container stops early and automatically), you might be interested in this other SO answer that gives further details on how to catch the signals raised by docker stop etc.
Related
faced with issue : when i am trying to run container :
docker run -t -p 3000:3000 --name container-dev-local image-dev-local
i am getting pretty much colored output with my logs :
But :
how can i exit without stopping container? Ctrl+P + Ctrl+Q don't work, it exiting and stop container
Update:
found the way to start without attaching by adding -d param :
docker run -d -p //other params
so now, after run - it starts in detached state, but colored logs dissapearing when i tried to attach
docker attach --sig-proxy=false container-dev-local
But after attaching i still can't exit from it without stopping process, CTRL+C, CTRL+P+Q does not work - it suspending process.
And how can i return back colorized logs? Have no ideas
My pre-requisites are : Windows 10, .net core inside of .nix container, visual studio 2019 and powershell terminal.
Found the solution!
First, you need to apply to docker run "-dti" flag, it enabling tty + detached mode (see docs)
It allows you to run container and trying to attach to it by
docker attach --sig-proxy=false container-dev-local
So now, you should see colored logs as -ti is also applied during run command.
How to exit without termination?
First, you can try Ctrl+Z+C (Z and C should be pressed in time of pressing Ctrl) command. It will stop your outer task and your terminal will ask you "Stop job?(y/n)" - then type "n" and ... should work :) You will be detached, container will continue work.
Second, we can override "detached keys" by adding --detach-keys :
docker attach --sig-proxy=false --detach-keys="ctrl-c" container-dev-local
And then, just try to simply use Ctrl+C, the same result will be achieved.
About CTRL+Z you can get some info here : What's different between Ctrl+Z and Ctrl+C in Unix command line
I use Dokku to run my app, and for some reason, the container is dying every few hours and recreates itself.
In order to investigate the issue, I am willing to read the error logs to this container and understand why it's crashing. Since Docker clears logs of dead containers, this is impossible.
I turned on docker events and it shows many events (like container update, container kill, container die, etc.) But no sign of what triggered this kill.
How can I investigate the issue?
Versions:
Docker version 19.03.13, build 4484c46d9d
dokku version 0.25.1
Logs are deleted when the container is deleted. If you want the logs to persist, then you need to avoid deleting the container. Make sure you aren't running the container with an option like --rm that automatically deletes it on exit. And check for the obvious issues like running out of disk space.
There are several things you can do to investigate the issue:
You can run the container in the foreground and allow it to log to your console.
If you were previously starting the container in the background with docker run -d (or docker-compose up -d), just remove the -d from the command line and allow the container to log to your terminal. When it crashes, you'll be able to see the most recent logs and scroll back to the limits of your terminal's history buffer.
You can even capture this output to a file using e.g. the script tool:
script -c 'docker run ...`
This will dump all the output to a file named typescript, although you can of course provide a different output name on the command line.
You can change the log driver.
You can configure your container to use a different logging driver. If you select something like syslog or journald, your container logs will be sent to the corrresponding service, and will continue to be available even after the container has been deleted.
I like use the journald logging driver because this allows searching for output by container id. For example, if I start a container like this:
docker run --log-driver journald --name web -p 8080:8080 -d docker.io/alpinelinux/darkhttpd
I can see logs from that container by running:
$ journalctl CONTAINER_NAME=web
Feb 25 20:50:04 docker 0bff1aec9b65[660]: darkhttpd/1.13, copyright (c) 2003-2021 Emil Mikulic.
These logs will persist even after the container exits.
(You can also search by container id instead of name by using CONTAINER_ID_FULL (the full id) or CONTAINER_ID (the short id), or even by image name with IMAGE_NAME.)
I am running a jar package within a container by docker. I spot when there is database connect timeout issue or kafka connect issue, the container will fail. However, I will be fine if I print java error log to console or log file. Anyone can clarify the logic to define a container as failed/error, Thanks!
Well there is no such thing as failed/errorneus container. Docker image can have default ENTRYPOIN or CMD which executes as docker container is started but when command ends docker lifecycle ends as well.
I assume you run some server app in docker container which serves forever which makes one think that docker images all run without stopping. Your docker which should always run stops after your app crashes, you can see the details in docker logs if your didn't run it with --rm option. Try docker ps -a to see your container with exited status and see execution logs or extract files from it's filesystem to debug what went wrong.
To the extent Docker has this concept at all, it follows normal Unix semantics. The container runs a single process, and when that process exits, the container exits too. If the process exits with a status code of 0 it is "successful" and if it exits with any other status code it "fails".
In the context of a Java container, Is there a complete List of JVM exit codes asserts that a JVM will always exit with status code 0 ("success") even if the program terminates with an uncaught exception; it will only return "failure" if the JVM itself fails in some way.
The most significant place this can matter is around a restart policy. If you start your container with docker run --restart on-failure, an uncaught exception won't be considered "failure" and your container won't restart.
I'm trying to run a service that does cron tasks using celery and it is very critical for our business. That certain container that runs the celery sometimes exits unexpectedly and I'm not being notified of the exit as well.
I have two questions:
How can I be notified via e-mail if one of my container exits?
How to do a disaster recovery for example if that container exits or stops another container will start and continue the process of that exited container
you can start the docker container with the parameter:
--restart="always"
for example:
docker run --restart="always" <IMAGE>
for your first Q i would suggest to look into that link:
How to programmatically monitor if a docker container exited?
What is detached mode in the docker world? I read this article
Link, but it does not explain exactly what detached mode mean.
You can start a docker container in detached mode with a -d option. So the container starts up and run in background. That means, you start up the container and could use the console after startup for other commands.
The opposite of detached mode is foreground mode. That is the default mode, when -d option is not used. In this mode, the console you are using to execute docker run will be attached to standard input, output and error. That means your console is attached to the container's process.
In detached mode, you can follow the standard output of your docker container with docker logs -f <container_ID>.
Just try both options. I always use the detached mode to run my containers. I hope I could explain it a little bit clearer.
The detach option on the docker command line indicates that the docker client (docker) will make a request to the server (dockerd), and then the client will exit while that request continues on the server. Part of the confusion may be that docker looks like a single process, where in reality it is a client/server application where the client is just a thin frontend on a REST API to send every command to the server.
With a docker container run --detach, this means the container will be created, the server will respond with a container id if successful, and the container will continue to run on the server while you are free to run other commands. This is often used for a server (e.g. nginx) you want to start in the background while you continue to run other commands. Note that you can still configure a container with the --interactive and -tty options (often abbreviated -it) and later run a docker container attach to connect to an already running container. (Note, until you attach to the container running with -itd, any attempt by the container to read from stdin would hang, instead of seeing an end of input that often triggers an immediate exit if you just passed -d.)
If you run without the detach option, the client will immediately run an attach API call after the container is created so you can see the output and optionally provide input to the running process on the container. This is useful if your container is running something interactive (e.g. /bin/bash).
Several other commands allow the detach option, including docker-compose up -d which will start an entire project and leave it running on the server in the background. There's also many of the docker service commands which will either detach after submitting the change to the server to create or update a service's target state, or if you do not detach, the client will wait until the service's current state matches the target state and you can see the progress of the deployment. Note with docker service commands, you may have to pass --detach=false to remain attached, the behavior has changed over the past year depending on your version.
What is detached mode in the docker world?
Detached means that the container will run in the background, without being attached to any input or output stream.
docker provide --detach (or -d in short) option and started the program in the background.
This means that the program started but isn’t attached to your terminal.
Example docker run --detach --name web nginx:latest # Note the detach flag.
Why do we need --detach mode?
Server software is generally run in detached containers because it is rare that the software depends on an attached terminal.
Running detached containers is a perfect fit for programs that sit quietly in the background.
Note
Generally, This type of program is called a daemon, or a service. A daemon generally interacts with other programs (or humans over a network) or some other communication channel. When you launch a daemon or other program in a container that you want to run in the background, remember to use either the --detach flag or its short form, -d.
To understand what does mean that -d in docker world, let me explain more clearly with the simulation within it. Docker detached mode means that you can tell the docker that do process(container) until if I am going to write the command docker stop container-id or container-name otherwise I mean without detached mode docker run the process(container) either you press ctrl+c or close a terminal, in other words you have not any choice when docker run the process(container) if you run the process(container) without -d. But you have any choice that docker return id of the container when you type the command with -d because of running process(container) in the background.
docker run -d -t ubuntu:14.04
docker run - Create an instance from docker image as docker container.
(if image not available locally it pulls from docker hub)
ubuntu - Image name
14.04 - Tag
-d, --detach - Detach mode
-t, --tty - Allocate a pseudo-TTY