I'd like to ask a question about how to dynamically move a running container along with the data produced by the container. I understand that the image of a docker container can be created and moved to other places, and docker volumes allow me to obtain data produced by a container on host machine. Here is my question. Assume that I have a container running that performs an iterative calculation and writes results to a docker volume after each iteration, and I want to move the container along with intermediate results (whatever is available in the docker volume) to a remote host machine that has a docker engine. After migration I will load the saved docker image and data to continue the calculation on the remote host machine. The command "docker save" allows me to get the image of the container, and the intermediate results produced by the container can be accessed from the docker volume on host machine. However, how do I know if the container is writing data when I read data from the docker volume? In other words, how to avoid concurrence of writes and reads in this case? Thanks in advance.
Related
Is is possible to share a single docker volume to multiple docker containers with /var/lib/docker destination?
A minimal reproducible example would be like below:
$ docker volume create --name lib
$ docker run --privileged -v lib:/var/lib/docker --name c1 -d docker:dind
$ docker run --privileged -v lib:/var/lib/docker --name c2 -d docker:dind
I want to work with docker inside c1 and c2 containers simultaneously. But if you wait a moment, you'll see it's not possible and the second container (c2) stops. I've checked the error logs:
$ docker logs -f c2
.
.
.
failed to start containerd: timeout waiting for containerd to start
And, I can not make multiple volumes; Because the storage is limited and the size of images are heavy.
UPDATE:
Maybe I'm facing with XY Problem! Actually I want to have my images shared. I want all of my Docker Images inside my host machine, go into all DinD containers ALSO the containers should be able to create a new Docker Image and this new image should be accessible from other containers at the same time.
On the title of the question, yes, multiple containers can mount the same volume. However, your containers are each docker engines, and the second engine is failing to start because there's already a running docker engine on the /var/lib/docker directory. This isn't a volume mounting issue so much as a docker engine design challenge.
Given your requirements, a container image database from the host engine, shared with various DinD instances, while not sharing the docker engine of the host itself (via a docker.sock or mTLS), I don't believe there's a good answer. You're left with two options:
Run your own local registry server. This is keep the layers from being sent outside your network, and could even be on the same host. However, the layers will be copied for each engine, and you'll need to manage GC policies on that registry. This gives you the desired isolation without the desired deduplication of image layers.
Share the docker.sock between the host and trusted containers. The containers would then have direct access to the host engine, effectively root on the host (unless you have setup the engine as rootless), so only do this in environments where you trust it. This would give you the layer deduplication, but none of the isolation.
The reason it's difficult is docker is designed to manage it's own copy of /var/lib/docker, so all the state can be tracked in memory and periodically pushed out as json metadata files on disk to handle restarts. Mutexes are within the one process, and it doesn't need to worry about multiple writers modifying layers, or a reader running while a writer is still creating a layer.
Take a look at this Document:
https://docs.docker.com/storage/bind-mounts/
Is is possible to share a single docker volume to multiple docker containers with /var/lib/docker destination?
A minimal reproducible example would be like below:
$ docker volume create --name lib
$ docker run --privileged -v lib:/var/lib/docker --name c1 -d docker:dind
$ docker run --privileged -v lib:/var/lib/docker --name c2 -d docker:dind
I want to work with docker inside c1 and c2 containers simultaneously. But if you wait a moment, you'll see it's not possible and the second container (c2) stops. I've checked the error logs:
$ docker logs -f c2
.
.
.
failed to start containerd: timeout waiting for containerd to start
And, I can not make multiple volumes; Because the storage is limited and the size of images are heavy.
UPDATE:
Maybe I'm facing with XY Problem! Actually I want to have my images shared. I want all of my Docker Images inside my host machine, go into all DinD containers ALSO the containers should be able to create a new Docker Image and this new image should be accessible from other containers at the same time.
On the title of the question, yes, multiple containers can mount the same volume. However, your containers are each docker engines, and the second engine is failing to start because there's already a running docker engine on the /var/lib/docker directory. This isn't a volume mounting issue so much as a docker engine design challenge.
Given your requirements, a container image database from the host engine, shared with various DinD instances, while not sharing the docker engine of the host itself (via a docker.sock or mTLS), I don't believe there's a good answer. You're left with two options:
Run your own local registry server. This is keep the layers from being sent outside your network, and could even be on the same host. However, the layers will be copied for each engine, and you'll need to manage GC policies on that registry. This gives you the desired isolation without the desired deduplication of image layers.
Share the docker.sock between the host and trusted containers. The containers would then have direct access to the host engine, effectively root on the host (unless you have setup the engine as rootless), so only do this in environments where you trust it. This would give you the layer deduplication, but none of the isolation.
The reason it's difficult is docker is designed to manage it's own copy of /var/lib/docker, so all the state can be tracked in memory and periodically pushed out as json metadata files on disk to handle restarts. Mutexes are within the one process, and it doesn't need to worry about multiple writers modifying layers, or a reader running while a writer is still creating a layer.
Take a look at this Document:
https://docs.docker.com/storage/bind-mounts/
I have an odoo:13 container and its corresponding postgres container running in a server. I want to move them to a test server so my test server's data be similar to my original server. how can I do it? I have mounted volumes by the way. (I created docker volumes)
As you have already mounted the data on to volumes, you will be able to start the containers on a new server using the same docker images using the backup from the old container.
Refer the official guide for backup-restore-or-migrate-data-volumes.
I am running Docker Swarm with 3-Masters and 3-Worker nodes.
On this Swarm, I have an Elastic search container which reads data from multiple log files and then writes the data into a directory. Later it reads data from this directory and shows me the logs on a UI.
Now the problem is that I am running only 1 instance of this Elastic Search Container and say for some reason it goes down then docker swarm start this on another machine. Since I have 6 machines I have created the particular directory on all the machines, but whenever I start the docker stack the ES container starts and starts reading/writing directory on the machine where it is running.
Is there a way that we can
Force docker swarm to run a container on a particular machine
or
Map volume to shared/network drive
Both are available.
Force docker swarm to run a container on a particular machine
Add --constraint flag when executing docker service create. Some introduction.
Map volume to shared/network drive
Use docker volume with a driver that supports writing files to an external storage system like NFS or Amazon S3. More introduction.
I am able to run a docker image and can connect log file in a container into a host machine volume .
I wonder how I can differentiate two containers of same running images at same host?
May be container ids can be used but I donot know how I can pass container id into log4j.xml file as a parameter?