I'm deploying a container using docker stack with a restart-policy. One of the container's attached volumes had some corrupted data, causing the container to repeatedly crash and eventually hit the max_restarts limit. I manually fixed the volume, and now want to restart the container. I have been unable to find a graceful way to ask Docker "please reset the max_restarts on this container". Does one exist?
I was able to proceed by doing a service rm and then re-create the service, but this seems a bit heavy-handed just for resetting a flag
Related
Apologies in advance if these questions are very basic. But I failed to find answers that are on-point:
In terms of clearing the container cache (not policies and not SIGTERM):
1- What is the difference between running the following commands:
docker restart
docker-compose restart
docker-compose down followed by docker-compose up
2- When restarting a container, in what order the following command get executed:
docker stop
docker kill
docker rm
docker start
docker run
UPDATE: Explaining the Problem
According to this issue of the docker registry, the Garbage Collection command doesn't clear the cache related to the deleted image blobs. This causes an error when pushing any deleted image again to the registry. One solution is to restart the container. Basically, it does the job of clearing the container cache including the Garbage Collection blobs cache. I tested it and it works.
But I am just confused because I used to run docker-compose down followed by docker-compose up and the cache didn't get cleared. Only when restarting the container, either by docker restart, docker-compose restart or restarting the server itself, that the push process (of previously deleted images) work again.
(Note that pushing new images always works).
That is why I would appreciate it if someone explain to me the difference between the above commands from a container cache point of view.
Thank you
This is probably because I don't understand something about how docker works, but I'm trying to get all the data from a container cleared. In this case the stack is for a jellyfin server.
I've tried removing the container and recreating it, I've tried removing the stack and recreating it, but no matter what I do as soon as I navigate to the jellyfin server in a browser all the accounts and metadata are still there!
All access is being done through Portainer, to delete a container I'm stopping it, then clicking remove:
To Delete a stack, I'm deleting any containers it has, then stopping it and deleting it:
Is there any way to use Docker's --rm option that auto-removes the container once it exits but allow the container's logs to persist?
I have an application that creates containers to process jobs, and then once all jobs are complete, the container exits and is deleted to conserve space. However, in case a bug caused the container's process to exit prematurely, I'd like to persist the log files so I can confirm it exited cleanly or diagnose a faulty exit.
However, the --rm option appears to remove the container's logs along with the container.
Log to somewhere outside of the container.
You could mount a directory of the host in your container, so logs will be written to the host directory and kept after rm.
Or you can mount a volume on your container; which will be persisted after rm
Or you can setup rsyslog - or some similar log collection agent - to export your logs to a remote service. See https://www.simulmedia.com/blog/2016/02/19/centralized-docker-logging-with-rsyslog/ for more on this solution.
The first 2 are hacks but easier to get up and running on your workstation/server. If this is all cloud hosted there might be a decent log offloading option (Cloudwatch on AWS) which saves you the hassle of configuring rsyslog
I have a swarm stack deployed and I removed couple services from the stack and tried to deploy them again. these services are showing with desired state remove and current state preparing .. also their name got changed from the custom service name to a random docker name. swarm also trying to start these services which are also stuck in preparing. I ran docker system prune on all nodes and them removed the stack. all the services in the stack are not existent anymore except for the random ones. now I cant delete them and they still in preparing state. the services are not running anywhere in the swarm but I want to know if there is a way to remove them.
I had the same problem. Later I found that the current state, 'Preparing' indicates that docker is trying to pull images from docker hub. But there is no clear indicator in docker service logs <serviceName> available in the docker-compose-version above '3.1'.
But it sometimes imposes the latency due to n\w bandwidth or other docker internal reasons.
Hope it helps! I will update the answer if I find more relevant information.
P.S. I identified that docker stack deploy -c <your-compose-file> <appGroupName> is not stuck when switching the command to docker-compose up. For me, it took 20+ minutes to download my image for some reasons.
So, it proves that there is no open issues with docker stack deploy,
Adding reference from Christian to club and complete this answer.
Use docker-machine ssh to connect to a particular machine:
docker-machine ssh <nameOfNode/Machine>
Your prompt will change. You are now inside another machine. Inside this other machine do this:
tail -f /var/log/docker.log
You'll see the "daemon" log for that machine. There you'll see if that particular daemon is doing the "pull" or what's is doing as part of the service preparation. In my case, I found something like this:
time="2016-09-05T19:04:07.881790998Z" level=debug msg="pull progress map[progress:[===========================================> ] 112.4 MB/130.2 MB status:Downloading
Which made me realise that it was just downloading some images from my docker account.
We have installed Ephesoft on a docker.
One file (in particular
dcma-batch.properties
in
Ephesoft/Application/WEB-INF/classes/META-INF/dcma-batch/
is reset after the docker exits and it is relaunched. Is there any way of stopping this?
If you restart a docker container, all data is lost, and it is reset to the initial image.
If you want persistent storage, look into docker volumes here:
https://docs.docker.com/engine/admin/volumes/