I am using docker-compose.yml. Just to give a bit background the git repo has a shell script named "sandbox" when i run "./sandbox up" it just does "docker-compose up -d --build" after setting some varables.
I just added
restart: always
to all the services in the existing .yml file and restarted all the containers. All the containers are up and running as expected. In order to check restart functionality, i just closed on the contained but it never automatically restated.
docker stop sandbox_chronograf_1_de92359427a0
The container exited with code 0 but this should not stop it from restarting because the restart is set to always but not on on-failure
Thanks
Related
I have some containers which all of them have the always restart value in the docker-compose file like this:
version: "3.7"
services:
container:
image: ghost:latest
container_name: some_container
restart: always
depends_on:
- ...
ports:
- ...
...
As soon as the OS (Flatcar Linux / CoreOS) has updated itself none of the containers restart. But if I just do $ sudo docker ps all of the containers starts at once. Whats up with that and how do I fix it so my containers automatically restarts after an update?
EDIT:
Not sure what is unclear about my question, restart: always is turned on. Unless I'm missing some vital thing in the documentation, this command should restart the container even if the docker daemon is restarted (after an os reboot).
Copy of one my comments from below:
Ok, so help me out here. As you can see in my question, I have
restart: always turned on. All these containers are started
successfully and are running well. Then the OS updates itself
automatically and restarts itself. After this restart the docker
daemon is restarted. But for some reasons the containers I had running
WITH RESTART: ALWAYS turned on DOES NOT START. If I enter my server
at this moment, type sudo docker ps to list my running containers,
suddenly all containers are booted up and I see the list. So why
wasn't the containers started, even though the daemon is running?
From the comments it appears the docker service was not configured to automatically start on boot. Docker is a client server app, and the server runs from systemd with a separate service for the docker socket used by the client to talk to the server. Therefore it's possible for any call with the docker command to cause the server to get launched by hitting the docker socket.
The service state in systemd can be checked with:
systemctl status docker
or you may want to check:
systemctl is-enabled docker
It can be manually started with:
systemctl start docker
And it can be enabled to start with:
systemctl enable docker
All of the above commands need to be run as root.
This requires the Docker service to get started on boot instead of using the default socket activation that starts on-demand like you decribed with execution of "docker ps"
Here is the required Container Linux Config to enable the Docker service while disabling socket activation:
systemd:
units:
# Ensure docker starts automatically instead of being socket-activated
- name: docker.socket
enabled: false
- name: docker.service
enabled: true
always Always restart the container if it stops. If it is manually stopped, it is restarted only when the Docker daemon restarts, or the container itself is manually restarted.
unless-stopped Similar to always, except that when the container is stopped (manually or otherwise), it is not restarted even after the Docker daemon restarts.
If you had an already running container that you wanted to change the restart policy for, you could use the docker update command to change that, and the below command will ensure all currently running containers will be restarted unless stopped
$ docker update --restart unless-stopped $(docker ps -q)
NOTE: Keep the following in mind when using restart policies
A restart policy only takes effect after a container starts successfully. In this case, starting successfully means that the container is up for at least 10 seconds and Docker has started monitoring it. This prevents a container that does not start at all from going into a restart loop.
If you manually stop a container, its restart policy is ignored until the Docker daemon restarts, or the container is manually restarted. This is another attempt to prevent a restart loop.
Restart policies only apply to containers. Restart policies for swarm services are configured differently
Documentation
If the docker container has been created before, it's [restart policy][1] may not be updated automatically by changing it in the docker compose YAML file.
If you change Restart Policy in the YAML file:
# cat docker-compose.yml
version: "3"
services:
<your-service>:
restart: always
You can see the container details in which RestartPolicy has old value yet:
# docker inspect <your-container> | fgrep -i restart -A 5
"RestartCount": 0,
--
"RestartPolicy": {
"Name": "",
Name is the Restart Policy name! and has no value that means no restart policy is set and the default value [no][1] is used.
So you may not only need to update the Restart Policy in the file, but also update pre created container manually:
# docker update <your-container> --restart always
So new value is changed:
# docker inspect <your-container> | fgrep -i restart -A 5
"RestartCount": 0,
--
"RestartPolicy": {
"Name": "always",
"MaximumRetryCount": 0
},
Just this :(
[1]: https://docs.docker.com/config/containers/start-containers-automatically/
It's container restart policy. restart: always restart the container if it stops. If it is manually stopped, it is restarted only when Docker daemon restarts or the container itself is manually restarted.Please check this link restart_policy.
I have some containers which all of them have the always restart value in the docker-compose file like this:
version: "3.7"
services:
container:
image: ghost:latest
container_name: some_container
restart: always
depends_on:
- ...
ports:
- ...
...
As soon as the OS (Flatcar Linux / CoreOS) has updated itself none of the containers restart. But if I just do $ sudo docker ps all of the containers starts at once. Whats up with that and how do I fix it so my containers automatically restarts after an update?
EDIT:
Not sure what is unclear about my question, restart: always is turned on. Unless I'm missing some vital thing in the documentation, this command should restart the container even if the docker daemon is restarted (after an os reboot).
Copy of one my comments from below:
Ok, so help me out here. As you can see in my question, I have
restart: always turned on. All these containers are started
successfully and are running well. Then the OS updates itself
automatically and restarts itself. After this restart the docker
daemon is restarted. But for some reasons the containers I had running
WITH RESTART: ALWAYS turned on DOES NOT START. If I enter my server
at this moment, type sudo docker ps to list my running containers,
suddenly all containers are booted up and I see the list. So why
wasn't the containers started, even though the daemon is running?
From the comments it appears the docker service was not configured to automatically start on boot. Docker is a client server app, and the server runs from systemd with a separate service for the docker socket used by the client to talk to the server. Therefore it's possible for any call with the docker command to cause the server to get launched by hitting the docker socket.
The service state in systemd can be checked with:
systemctl status docker
or you may want to check:
systemctl is-enabled docker
It can be manually started with:
systemctl start docker
And it can be enabled to start with:
systemctl enable docker
All of the above commands need to be run as root.
This requires the Docker service to get started on boot instead of using the default socket activation that starts on-demand like you decribed with execution of "docker ps"
Here is the required Container Linux Config to enable the Docker service while disabling socket activation:
systemd:
units:
# Ensure docker starts automatically instead of being socket-activated
- name: docker.socket
enabled: false
- name: docker.service
enabled: true
always Always restart the container if it stops. If it is manually stopped, it is restarted only when the Docker daemon restarts, or the container itself is manually restarted.
unless-stopped Similar to always, except that when the container is stopped (manually or otherwise), it is not restarted even after the Docker daemon restarts.
If you had an already running container that you wanted to change the restart policy for, you could use the docker update command to change that, and the below command will ensure all currently running containers will be restarted unless stopped
$ docker update --restart unless-stopped $(docker ps -q)
NOTE: Keep the following in mind when using restart policies
A restart policy only takes effect after a container starts successfully. In this case, starting successfully means that the container is up for at least 10 seconds and Docker has started monitoring it. This prevents a container that does not start at all from going into a restart loop.
If you manually stop a container, its restart policy is ignored until the Docker daemon restarts, or the container is manually restarted. This is another attempt to prevent a restart loop.
Restart policies only apply to containers. Restart policies for swarm services are configured differently
Documentation
If the docker container has been created before, it's [restart policy][1] may not be updated automatically by changing it in the docker compose YAML file.
If you change Restart Policy in the YAML file:
# cat docker-compose.yml
version: "3"
services:
<your-service>:
restart: always
You can see the container details in which RestartPolicy has old value yet:
# docker inspect <your-container> | fgrep -i restart -A 5
"RestartCount": 0,
--
"RestartPolicy": {
"Name": "",
Name is the Restart Policy name! and has no value that means no restart policy is set and the default value [no][1] is used.
So you may not only need to update the Restart Policy in the file, but also update pre created container manually:
# docker update <your-container> --restart always
So new value is changed:
# docker inspect <your-container> | fgrep -i restart -A 5
"RestartCount": 0,
--
"RestartPolicy": {
"Name": "always",
"MaximumRetryCount": 0
},
Just this :(
[1]: https://docs.docker.com/config/containers/start-containers-automatically/
It's container restart policy. restart: always restart the container if it stops. If it is manually stopped, it is restarted only when Docker daemon restarts or the container itself is manually restarted.Please check this link restart_policy.
In .yml file I have defined: restart: always. Is it possible to create this restart as the equivalent of --force-recreate flag?
I have an issue with XVFB and standard restart doesn't solve an issue but restarts with the flag --force-recreate help and I'm looking for an opportunity to do it automatically.
Always restart the container if it stops. If it is manually stopped, it is restarted only when Docker daemon restarts or the container itself is manually restarted. (See the second bullet listed in restart policy details) Source Link:
No --force-recreate is not the equivalent to restart: always
"--force-recreate Recreate containers even if their configuration and image haven't changed."
I use a Makefile for start/stop is also more practical.
Example:
SHELL := /bin/bash
# Docker: up
up:
docker-compose up -d --force-recreate --build
# Docker: down
down:
docker-compose down
... and so on
And than i can use like "make up, make down, make logs, make attach ..."
By the way, in most projects I also use for Automatic Restart and better logging Supervisor
I have read the docker-compose documentation about restart policy of containers,
However I failed to understand the difference between on-failure and unless-stopped.
When will I use one over the other? In which situations a certain policy will lead to start a container and the other policy not?
on-failure will issue a restart if the exit code indicated a failure, whereas unless-stopped behaves like always and will keep an instance running unless the container is stopped.
You can try with the hello-world to see the difference.
docker run --restart on-failure hello-world will run once and exit successfully, and running a subsequent docker ps will indicate no currently running instances of the container.
However, docker run --restart unless-stopped hello-world will restart the container even if it successfully exits, so subsequently running docker ps will show you a restarting instance until you stop the container.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4d498ebd13a6 hello-world "/hello" 2 seconds ago Restarting (0) Less than a second ago modest_keldysh
Docker restart policies are there to keep containers active in all possible downfalls, we can leverage it in multiple ways as an example if we have a web server running on a container and have to keep it active even on bad request we can use unless-stopped flag, it will keep the server up and running till we stopped it manually.
Restart flag can be any one of these :-
"no" :- it is the default value, and it will never restart the container.
on-failure :- it will restart the container whenever it encounters an error, or say, whenever the process running inside the container exit with non-zero exit code. Exit code :- 0 means no error, we terminated the process intentionally, but any non-zero value is an error.
always :- as the name, it will always restart the container, no matter whatever be the exit code is. Also, it will restart the container even when we manually stopped it but for that we need to restart the docker daemon.
unless-stopped :- it is similar to the always flag the only difference is once the container is stopped manually it will not restart automatically even after restarting the docker daemon, until we start the container manually again.
The difference between unless-stopped and on-failure is the first will always restart until we stopped it manually no matter whatever be the exit code will be and another will only restart the container on real failure, i.e. exit code = non-zero.
Once the container is stopped its restart flags are ignored, this is one way to overcome from restarting loop. That's why in the case of always flag once we stopped it manually the container will not restart till we restart the docker daemon.
You can easily test all of these flags by creating a simple redis-server:
docker run -d --restart=always --name redis-server redis # start redis image
docker container ls # test the status
docker stop redis-server # stop the container manually
docker container ls # test the status again, got the redis-server did not restarted
sudo service docker restart # restart the docker daemon
# test the status again will find the container is again up and running
# try the same steps by changing the restart flag with *unless-stopped*
docker update --restart=unless-stopped redis-server # will update the restart flag of running container.
I run the one of the open source microservices from here. When i run docker ps then all the containers status are UP, means they keep running. My issue is when I separately run a container then it did not keep running and exits. Below is one of the service defined in docker-compose file.
social-graph-service:
image: yg397/social-network-microservices
hostname: social-graph-service
restart: always
entrypoint: SocialGraphService
when i run it using command
sudo docker run -d --restart always --entrypoint SocialGraphService --hostname social-graph-service yg397/social-network-microservices
then its status does not UP, it exits after running. Why all the containers run continuously when i run them using sudo docker-compose up? and exit when i run them individually?
It looks like the graph service depends on MongoDB in order to run. My guess is it crashes when you run it individually because the mongo instance doesn't exist and it fails to connect.
The author of the repo wrote the docker-compose file to hide away some of the complexity from you, but that's a substantial tree of relationships between microservices, and most of them seem to depend on others existing in order to boot up.
-- Update --
The real issue is in the comments below. OP was already running the docker-compose stack while attempting to start another container, but forgot to connect the container to the docker network generated by docker-compose.