I have a docker container with JETTY CMD instruction.
After "docker restart", which goes immediately, I cannot access JETTY about 9-10 seconds. After that time docker container or jetty service is UP again and I can access it.
Question is: is there a standard way to check that the docker container is really up?
Surely I can make a loop with test requests to my service and wait to 200 response code. But maybe there is a more beautiful solution?
Thanks
Sergey, youre need to use initialization system as supervisor of your in-Docker processes. You may use distro-built-in init systems like systemd/upstart or init.d depends on your OS for checking a container state.
In theory you should to create independent service in you init system on each docker run command without -d option, because with -d option docker detached a container and returned 0 exit status to init system. As result init system lost a control of target process.
For example, realization of this mechanism in Systemd:
Create something.service file in /etc/systemd/system
And type to it something like this:
[Unit]
Description=Simple Blog Rails Docker Container Service
After=docker.service
Requires=docker.service
[Service]
Restart=on-failure
ExecStartPre=-/usr/bin/docker kill simple-blog-rails-container
ExecStartPre=-/usr/bin/docker rm simple-blog-rails-container
ExecStart=/usr/bin/docker run simple-blog-rails
ExecStop=/usr/bin/docker stop simple-blog-rails-container
[Install]
WantedBy=multi-user.target
Reload Systemd configuration systemctl daemon-reload
Just try to run your container by typing systemctl start something.service or restart instead of start.
You can check service state systemctl status something.service
For more information about using systemd and docker you may read this CoreOS manual: https://coreos.com/docs/launching-containers/launching/getting-started-with-systemd/
Keep in mind that docker restart defaults to a wait time of 10 seconds:
restart Usage: docker restart [OPTIONS] CONTAINER [CONTAINER...]
Restart a running container
-t, --time=10 Number of seconds to try to stop for before
killing the container. Once killed it will then be restarted. Default
is 10 seconds.
When I try this with some long running script in a docker container it takes 10 seconds before it's done. If I change the command line to use a different timeout (like -t=4} it comes back in 4 seconds.
If you want to determine if a container is running (even if your service contained within it is not quite ready yet) you can:
Run docker ps - which will give you a list of all running docker
containers
Use the Docker Remote API command GET /containers/json - this will
give you a json response with a list of running containers.
https://docs.docker.com/reference/api/docker_remote_api_v1.16/
https://docs.docker.com/reference/commandline/cli/#ps
Related
I have been running a nvidia docker image since 13 days and it used to restart without any problems using docker start -i <containerid> command. But, today while I was downloading pytorch inside the container, download got stuck at 5% and gave no response for a while.
I couldn't exit the container either by ctrl+d or ctrl+c. So, I exited the terminal and in new terminal I ran this docker start -i <containerid> again. But ever since this particular container is not responding to any command. Be it start/restart/exec/commit ...nothing! any command with this container ID or name is just non-responsive and had to exit out of it only after ctrl+c
I cannot restart the docker service since it will kill all running docker containers.
Cannot even stop the container using this docker container stop <containerid>
Please help.
You can make use of docker RestartPolicy:
docker update --restart=always <container>
while mindful of caveats on the docker version you running.
or explore an answer by #Yale Huang from a similar question: How to add a restart policy to a container that was already created
I had to restart docker process to revive my container. There was nothing else I could do to solve it. used sudo service docker restart and then revived my container using docker run. I will try to build the dockerfile out of it in order to avoid future mishaps.
I would like to run docker exec on some, possibly stopped docker container.
By possibly I mean that it could be running just fine but in some cases, namely server reboot and so on, container that I want run docker exec would be stopped.
Is there any good way to make sure that docker exec would execute without error in both cases (container running, container stopped). And in case of stopped wouldn't return:
Error response from daemon: Container is not running
From docker exec --help
you can find, among other things
Run a command in a running container
I do not know what you want to do with a stopped container?
Maybe try to restart it?
You know you can start a container with a restart policy to always
see the doc
https://docs.docker.com/engine/reference/run/
Extract
Always restart the container regardless of the exit status. When you specify always, the Docker daemon will try to restart the container indefinitely. The container will also always start on daemon startup, regardless of the current state of the container.
You can't run docker exec against a stopped container. From docker help exec:
$ docker help exec
Usage: docker exec [OPTIONS] CONTAINER COMMAND [ARG...]
Run a command in a running container
So if your target container has been stopped by some reason, you need to start it by docker start <your_container> before you can do docker exec ....
BTW, docker run command has an option called --restart to let you to specify a restart policy for the container, you can find more details on docker run --restart docs. There're 4 policies available:
no: Do not automatically restart the container when it exits. This is the default.
on-failure[:max-retries]: Restart only if the container exits with a non-zero exit status. Optionally, limit the number of restart retries the Docker daemon attempts.
always: Always restart the container regardless of the exit status. When you specify always, the Docker daemon will try to restart the container indefinitely. The container will also always start on daemon startup, regardless of the current state of the container.
unless-stopped: Always restart the container regardless of the exit status, but do not start it on daemon startup if the container has been put to a stopped state before.
By default it's no, you could choose another one based on your requirement. For example if you choose non-stopped, your container will got restarted automatically when docker daemon is ready after your server reboot.
I am testing CoreOS to see if it meets our needs, and so far things are going a little slow, but OK. I like systemd, but it doesn't seem to be working properly - specifically, on shutdown.
My Goal
My goal is to have a script run on service start and stop that adds and removes records from our DNS server respectively for a service. It works when the service is started by the system on boot, or when it is manually started or shut down - but not when the system is rebooted or halted (shutdown -r now, shutdown -h now).
Here is a slightly simplified version of a docker registry service I am using for an example:
[Unit]
Description=Docker Registry
After=docker.service
Before=registry-ui.service
Wants=registry-ui.service
[Service]
Conflicts=halt.target poweroff.target reboot.target shutdown.target sleep.target
TimeoutStartSec=0
Restart=on-failure
ExecStartPre=-/usr/bin/docker kill Registry
ExecStartPre=-/usr/bin/docker rm Registry
ExecStartPre=-/usr/bin/docker run --rm myrepo:5000/tool runtool arg1 arg2
ExecStart=/usr/bin/docker run various args registry:latest
ExecStop=-/usr/bin/docker run --rm myrepo:5000/tool runtool arg1 arg2
ExecStop=-/usr/bin/docker stop Registry
[X-Fleet]
MachineID=coreos1
[Install]
WantedBy=multi-user.target
RequiredBy=registry-ui.service
Also=registry-ui.service
(This unit works together with another unit - registry-ui.service. When one is started the other does as well.)
Note the Conflicts=... line. I added it after spending time trying to figure out why the service wasn't shutting down properly. It did nothing. According to the docs, however, services have a Conflicts=shutdown.target line by default. When services conflict and one starts up, the other shuts down - or so the docs say.
What did I miss? Why won't my ExecStop= lines run?
Update
I've determined that ExecStop= lines do run. Using journalctl -u registry.service -n 200 gave me this message:
Error response from daemon: Cannot start container 7b9083a3f81710febc24256b715fcff1e8146c867500c6e8ce4d170ad1cfd11a: dbus: connection closed by user
Which indicates that the problem is (as I speculated in the comments) that my docker container won't start during shutdown. I've added the following lines to my [Unit] section:
[Unit]
After=docker.service docker.socket
Requires=docker.service docker.socket
...
The new lines have no effect on the journalctl error, so my question now becomes, is there a way to run a utility docker container prior to shutdown?
If I understood your goal, you would like to run some DNS cleanup when a server is being powered off, and you are attempting to do this inside the systemd docker service file.
Why don't you use fleet for that task ?
try to create a fleet unit-file that monitors your DNS server, and when it detect the server is not reachable you can launch your cleanup tasks.
On fleet, when you destroy a service using fleetctl destroy, the exec stop lines don't run (similar to power-off). If you want to have a cleanup session, you usually achieve this using a satellite service with this directives
serviceFile.service
[Unit]
Wants=cleanup#serviceFile.service
cleanup#serviceFile.service
[Unit]
PartOf=%i.service
Instead of starting everything in one unit file, you can start 2 different services and bind them using bindsTo so that they start and stop together : http://www.freedesktop.org/software/systemd/man/systemd.unit.html#BindsTo=
That would make two easy and smaller unit files, each taking care of one container. It would also make debugging easier for you.
I am using docker version 1.1.0, started by systemd using the command line /usr/bin/docker -d, and tried to:
run a container
stop the docker service
restart the docker service (either using systemd or manually, specifying --restart=true on the command line)
see if my container was still running
As I understand the docs, my container should be restarted. But it is not. Its public facing port doesn't respond, and docker ps doesn't show it.
docker ps -a shows my container with an empty status:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cb0d05b4e0d9 mildred/p2pweb:latest node server-cli.js - 7 minutes ago 0.0.0.0:8888->8888/tcp jovial_ritchie
...
And when I try to docker restart cb0d05b4e0d9, I get an error:
Error response from daemon: Cannot restart container cb0d05b4e0d9: Unit docker-cb0d05b4e0d9be2aadd4276497e80f4ae56d96f8e2ab98ccdb26ef510e21d2cc.scope already exists.
2014/07/16 13:18:35 Error: failed to restart one or more containers
I can always recreate a container using the same base image using docker run ..., but how do I make sure that my running containers will be restarted if docker is restarted. Is there a solution that exists even in case the docker is not stopped properly (imagine I remove the power plug from the server).
Thank you
As mentioned in a comment, the container flag you're likely looking for is --restart=always, which will instruct Docker that unless you explicitly docker stop the container, Docker should start it back up any time either Docker dies or the container does.
I have been running docker processes (apps) via
docker run …
But under runit supervision (runit is like daemontools) - so runit ensures that the process stays up, passes signals etc.
Is this reasonable? Docker seems to want to run its own demonization - but it isn't as thorough as runit. Furthermore, when runit restarts the app - a new container is created each time (fine) but it leaves a trace of the old one around - this seems to imply I am doing it in the wrong way.
Should docker not be run this way?
Should I instead set up a container from the image, just once, and then have runit run/supervise that container for all time?
Docker does do some management of daemonized containers: if the system shuts down, then when the Docker daemon starts it will also restart any containers that were running at the time the system shut down. But if the container exits on its own or the kernel (or a user) kills the container while it is running, the Docker daemon won't restart it. In cases where you do want a restart, a process manager makes sense.
I don't know runit so I can't give specific configuration guidance. But you should probably make the process manager communicate with the docker daemon and check to see if a given container id is running (docker ps | grep container_id or equivalent, or use the Docker Remote API directly). If the container has stopped, use Docker to restart it (docker run container_id) instead of running a new container. Or, if you do want a new container each time, then begin with docker run -rm to automatically clean it up when it exits or stops.
If you don't want your process manager to poll docker, you could instead run something that watches docker events.
You can get the container_id when you start the container as the return value of starting a daemon, or you can ask Docker to write this out to a file (docker run -cidfile myfilename, like a PID file)
I hope that helps or helps another runit guru offer more detailed advice.
Yes, I think running docker under runit makes sense. Typically when you start a process there is a way to tell it not to daemonize if it does by default since the normal way to hand-off from the runit run script to a process is via exec on the last line of your run script. For docker this means making sure not to set the -d flag.
For example, with docker you probably want your run script to look something like this:
#!/bin/bash -e
exec 2>&1
exec chpst -u dockeruser docker run -a stdin -a stdout -i ...
Using exec and chpst should resolve most issues with processes not terminating correctly when you bring down a runit service.