I have a swarm stack deployed and I removed couple services from the stack and tried to deploy them again. these services are showing with desired state remove and current state preparing .. also their name got changed from the custom service name to a random docker name. swarm also trying to start these services which are also stuck in preparing. I ran docker system prune on all nodes and them removed the stack. all the services in the stack are not existent anymore except for the random ones. now I cant delete them and they still in preparing state. the services are not running anywhere in the swarm but I want to know if there is a way to remove them.
I had the same problem. Later I found that the current state, 'Preparing' indicates that docker is trying to pull images from docker hub. But there is no clear indicator in docker service logs <serviceName> available in the docker-compose-version above '3.1'.
But it sometimes imposes the latency due to n\w bandwidth or other docker internal reasons.
Hope it helps! I will update the answer if I find more relevant information.
P.S. I identified that docker stack deploy -c <your-compose-file> <appGroupName> is not stuck when switching the command to docker-compose up. For me, it took 20+ minutes to download my image for some reasons.
So, it proves that there is no open issues with docker stack deploy,
Adding reference from Christian to club and complete this answer.
Use docker-machine ssh to connect to a particular machine:
docker-machine ssh <nameOfNode/Machine>
Your prompt will change. You are now inside another machine. Inside this other machine do this:
tail -f /var/log/docker.log
You'll see the "daemon" log for that machine. There you'll see if that particular daemon is doing the "pull" or what's is doing as part of the service preparation. In my case, I found something like this:
time="2016-09-05T19:04:07.881790998Z" level=debug msg="pull progress map[progress:[===========================================> ] 112.4 MB/130.2 MB status:Downloading
Which made me realise that it was just downloading some images from my docker account.
Related
Does anyone know if there is a way to have docker swarm restart one service that is part of a stack without restarting the whole stack?
Doing docker stack deploy again for me is the way to go to update services. As Francois' Answer, and also in my own experience, doing so updates only services that need to be updated.
But sometimes, it seems easier when testing stuff to only restart a single service. In my case, I had to clear the volume and update the service to start it like it was fresh. I'm not sure if there is downside to the method I will describe. I tested it on my development stack and it worked great for me.
Get the service id you want to tear down then use docker service update --force <id> to force the update of the service which effectively re-deploy it
$ docker stack services <stack_name>
ID NAME ...
3xrdy2c7pfm3 stack-name_api ...
$ docker service update --force 3xrdy2c7pfm3
The --force flag will force the service to update causing it to restart.
Scale to 0 and back up:
docker service scale myservice=0
docker service scale myservice=10
Looking at the docker stack documentation:
Extended description
Create and update a stack from a compose or a dab file on the swarm
From this blog article: docker stack works in a similar way as docker compose. It’s idempotent. If the stack is already deployed, docker stack deploy will restart only those services which has the digest or tag that is updated:
From my experience, when I deploy the same stack again with one service changing, only the updated service will be restarted.
BUT... there seems to be some limitations to changes that are taken into account (some report bugs with image tags), so give it a try and see if works as expected.
You can also use service update if you want to be sure that only targeted service if updated with your changes.
You can also refer to this similar SO QA.
As per the example in the documentation for rolling updates:
$ docker service update --image redis:3.0.7 redis
However, that only works if your image is already on the local machines. If not then you need to use --with-registry-auth to send registry authentication details to the swarm agents. See details in the docker service update documentation.
$ docker service update --with-registry-auth --image redis:3.0.7 redis
To restart a single service (with rolling restart to avoid downtime in case the service has multiple replicas) in already configured, existing stack, you can do:
docker service update --force stack_service_name
I don't recommend running the same command again (in another shell) until this one completes (because otherwise the rolling restart isn't guaranteed; it might restart all replicas of that service).
The docker service update command also checks for newer version of the image:tag you are trying to use.
If your registry requires auth, also pass --with-registry-auth argument, like so:
docker service update --force --with-registry-auth stack_service_name
If you don't pass this argument, the service will still be restarted, but the check won't be made and the service will still use the old container image without pulling the new one first. Which might be what you want.
In case you want to also switch to different image tag (or completely different image), you can do it from here too, but remember to also change the tag in your docker-stack.yml, or your next docker stack deploy will revert it back to the verison defined there:
docker service update --with-registry-auth --force --image nginx:edge stack_service_name
remove it:
docker stack rm stack_name
redeploy it:
docker stack deploy -c docker-compose.yml stack_name
I have correctly deployed a Docker container which runs a Python script that grabs some data from the internet and slaps it in BigQuery. The container works well on my machine and on a GCE instance that I've provisioned.
Now, everything works well for the most part but I am failing to understand why the docker container always restarts after exiting (apparently correctly). Logs, in this case, seems to be fairly useless as there is no error whatsoever. My current hunch is that something is failing silently, forcing the instance to restart.
Is there any way to find out the reboot reason for a given Docker container?
Things tried so far
I've tried to print the exit code of the container in the following way. The result is always 0, no matter those restart cycles.
while true
do
docker inspect my_container --format='{{.State.ExitCode}}'
sleep 1
done
The Google Cloud documentation provides you different ways in which you can review your container related logs including container starts and stops.
In any way, I think there is no problem with your container: by default Compute Engine will restart a container on exit, although you can specify a different restart policy if you need to. Please, see the relevant documentation.
I am new to Docker, and I find the definitions of containers' lifecycle differ a lot.
here is what "Manning.Docker.in.Action.2016.3" shows：
here is what google gives me:
https://medium.com/#nagarwal/lifecycle-of-docker-container-d2da9f85959
here is what the official document says：
status： One of created, restarting, running, removing, paused, exited, or dead
https://docs.docker.com/engine/reference/commandline/ps/
So what's going on here? I guess some new states(and renaming) are introduced in newer version of Docker?
Thanks in advance
Your linked diagram separates docker create from docker start, it includes "die" as a state transition, and it shows how to get to the "restarting" state. That's all valid, though it leads to a more complicated state machine.
(docker create wasn't in the very first versions of Docker but it appeared in Docker 1.3.0 in 2014, which should predate your diagram.)
Practically I might suggest an even simpler state machine:
-------> running -+------> stopped ------>
run | stop rm
\------> exited ------>
process exits rm
That is, never try to restart a container or make changes inside a running container; if you need to tweak anything, delete the existing container and create a new one. This gives you a consistent environment (when the main container process starts you always know what's in its filesystem, up to mounted data). It also matches what happens in cluster environments like Kubernetes, where the cluster manager will routinely create and delete containers for you.
When you get in a situation where internet gives you different answers, you should consider trying it yourself. Especially with technologies like docker, where it is pretty simple to make tests. For example:
I want to run a container (I will use nginx):
docker run -d nginx
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
258cd2edbed8 nginx "nginx -g 'daemon of…" 3 seconds ago Up 2 seconds 80/tcp jolly_golick
Note: docker will keep a container running only if there is a process running in it.
If you would start a debian container (for example), you would see how it immediately stop, as there is nothing running in it. So you could do
docker run -d debian sleep 10
and see that the container is up for 10 seconds.
When a container is running, you can do some things on it. You can't do other things, like removing it. To remove a container, you need to stop it first (or kill it), or force container removal.
Note: You would get all this info from docker itself, if you would be playing around with it, as it would return these info. Like if you would try to remove a running container, you would get this error:
Error response from daemon: You cannot remove a running container 258cd2edbed85bed23ab543312968bd893c1fbd9ba81de40366337f434daedff. Stop the container before attempting removal or force remove
I can't do all possible combinations here. You would get a similar error if you would try removing a paused container. Just play with it, and you will get a clear picture of how it works.
I am using docker-compose to deploy an application combining a number of different images.
Using Docker version 18.09.2, build 6247962
Docker-compose 1.117
Primarily, I have
ZooKeeper
Kafka
MYSQLDb
I notice a strange problem where i could not start my application with docker-compose up due to port already being assigned. I then checked docker stats and saw that there were three containers named "test_ZooKeeper.1slehgaior"
"test_Kafka.kgjdorgsr"
"test_MYSQLDB.kgjdorgsr"
I have tried kill the containers, removing them and pruning the system. When ever I kill one of these containers, it instantly restarts and I cannot for the life of me determine where they are being created from!
Please help :)
If you look into your docker-compose.yaml I'm pretty sure you'll find a restart:always somewhere. If you want to correctly shut down a running docker container managed by docker-compose, one way is to use docker-compose down from the directory where your yaml sits.
More information on the subject:
https://docs.docker.com/config/containers/start-containers-automatically/
Otherwise, you might try out to stop a single running container instead of killing it, which according to my memory tells docker not to restart it again, while a killed container looks to the service like it just has crashed. Not too sure about the last part though.
The docker-compose utility is attached to the terminal by default allowing you to see that's happening with all of your containers which is very convenient for development. Does the docker stack deploy command support something like this when the activity of the running containers gets rendered in one terminal in real time?
According to Docker website the only log displayed is:
docker stack deploy --compose-file docker-compose.yml vossibility
Ignoring unsupported options: links
Creating network vossibility_vossibility
Creating network vossibility_default
Creating service vossibility_nsqd
Creating service vossibility_logstash
Creating service vossibility_elasticsearch
Creating service vossibility_kibana
Creating service vossibility_ghollector
Creating service vossibility_lookupd
However, there's a command which displays the logs:
docker service logs --follow
Therefore, on a Linux system you could combine both commands and you will get the desired output
What you're looking for is a merged output of the logs ("attached" for a stack deploy is a different thing with progress bars).
You can't get the logs for the full stack just yet (see issue #31458 to track the progress of this request), but you can get the logs for all of the containers in a service with docker service logs.