I have an application that saves the data in the container directory /var/lib/app-data.
This container directory should be backed up since the application makes changes to the /var/lib/app-data content regularly as it runs.
But need to backup only specific directories and files under /var/lib/app-data/
For ex Below are the files and directories should be backed up regularly under /var/lib/app-data/,so even if the container stopped, the new container still get this files and directories.
/var/lib/app-data/ui_log/
/var/lib/app-data/node_log/
/var/lib/app-data/users/mailer.xml
/var/lib/app-data/plugins/
Other files and directories should not be backed up to volumes.
Currently I could able to find out only one way to map docker volume like below.
docker run -v forkuidatavolume:/var/lib/app-data mohan/forkui:1.0
But above step backing up directories other than one mentioned above.
Is it possible to backup specific container files and directories using -v or any other way?
Thanks,
Mohan
You can put a list of files that you do not want to be added to your docker image at .dockerignore. For example:
.git
/tmp/*
/var/lib/app-data/ui_log/*
/var/lib/app-data/node_log/
/var/lib/app-data/users/mailer.xml
/var/lib/app-data/plugins/*
or you can use multiple volumes and link them to your tmp like this:
docker run -v /var/lib/app-data/ui_log/:/tmp/ui_log -v /var/lib/app-data/node_log/:/tmp/node_log ... mohan/forkui:1.0
Related
My situation is the following:
I am having a docker image/container in which I am compiling. I had to install some components to $HOME via the Dockerfile (so while creating the image).
Let's say one of those components is in ~/.config, but also other folders.
I would like to have the possibility to override the files in .config by mounting a home folder from the host on top of the one inside docker. Whenever you place a file in the mounted folder, it overrides the one which is already inside the container.
So in theory, this is exactly what an OverlayFS does, right? While the lower directory would be the one inside the Docker container, the upper directory would be the one on my Host.
Is there a way to accomplish that?
Until now I found the following related topics:
https://serverfault.com/questions/841238/how-to-use-overlayfs-with-docker-volumes
Drawback: The answer does only show how to use overlayfs on the host, but getting acccess to the lower container/image directory is not that self-explaining and also feels dirty.
Can I mount docker host directory as copy on write/overlay?
Drawback: Using mount -t overlay inside docker does not work on newer kernels because of the disabled overlay-on/over-overlay option
I also thought about manipulating the docker files on host directly, i.e. the directories where docker stores the files, but that feels a bit dirty.
To do so, I would declare VOLUME /home/user at the end of the Dockerfile. Then I would find my files of that directory in /var/lib/docker/volumes/user/_data. I could then create a overlayfs on my host, using that directory as lower, my other folder as upper. I could then remount that new directory using docker run --volume. Unfortunately this would involve su rights to access the /var/lib directory.
The other way around would be to bind-mount single files, but that's maybe a bit hackish too.
I am currently running a docker container with a specific folder shared to it. However, I need to access other folders which are not in the shared folder.
Does docker mount or docker volume help in any way to mount a file into a running docker container?
You can do this in two ways.
just copy the file and paste it the shared location, it will be available for use in docker mount location.
copy the host file to the container where you want, using
docker cp your_local_file containerid:/path/to/container
There are other ways around but that will not be that easy as these two approaches.
You can check this details article but I think it will take your handsome time to that each and every time.
mount-volumes-into-a-running-container
You can also check
docker-mount-dynamic-volumes/
I was thinking one alternative could be to use soft links by placing them inside your mounted volume but it won't work as documented in this post. A suggested alternative is to mount your folders as different volumes:
-v /home/test/:/home/test -v /mnt/mountedfile:/mnt/mountedfile
I have a situation where:
I want to mount a directory ~/tmp/mycode to /mycode readonly
I want to be able to edit the files in the directory, so I can't just run -v /my/local/path/tmp/mycode:/mycode
I want it to not persist changes on the host filesystem though so I can't mount it read/write
~/tmp/mycode is rather large
Basically I want to be able to edit the files in the mounted volume but not have those changes persisted.
My current workflow is to create a dummy container using a dockerfile:
ADD . /mycode
and then execute that container.
However as the repository grows, this step takes longer and longer to perform, because the only way I can think is to make a complete copy of ~/tmp/mycode in order to be able to manipulate the files in the container.
I've also thought about mounting the directory and copying it inside the container and committing that container, but that has the same issue.
Is there a way to run a docker container to allow file edits without persisting them on the host short of copying the whole directory?
I am using the latest docker for mac, currently Version 17.03.1-ce-mac5 (16048).
This is fairly trivial to do with docker and overlay:
docker run --name myenv --privileged -v /my/local/path/tmp/mycode:/mnt/rocode:ro -it ubuntu /bin/bash
docker exec -d myenv /sbin/mount -t overlay overlay -o lowerdir=/mnt/rocode,upperdir=/mycode,workdir=/mnt/code-workdir /mycode
This should mount the code from your directory read only and create the overlay inside the container so that /mnt/rocode is read only, but /mycode is writable.
Make sure that your kernel is 3.18+ and that you have overlay in your /proc/filesystems.
I am trying to capture the state of a docker container as an image, in a way that includes files I have added to a volume within the container. So, if I run the original container in this way:
$ docker run -ti -v /cookbook ubuntu:14.04 /bin/bash
root#b78f3599d936:/# cd cookbook
root#b78f3599d936:/cookbook# touch foo.txt
Now, if I either export, or commit the container as a new docker image, and then run a container from the new image, then the file, foo.txt is never included in the /cookbook directory.
My question is whether there is a way to create an image from a container in a way that allows the image to include file content within its volumes.
whether there is a way to create an image from a container in a way that allows the image to include file content within its volumes?
No, because volume is designed to manage data inside and between your Docker containers, it's used to persist and share data. What's in image is usually your program(artifacts, executables, libs. e.g) with its whole environment, building/updating data to image does not make much sense.
And in docs of volumes, they told us:
Changes to a data volume will not be included when you update an image.
Also in docs of docker commit:
The commit operation will not include any data contained in volumes mounted inside the container.
Well, by putting the changes in a volume, you're excluding them from the actual container. The documentation for docker export includes this:
The docker export command does not export the contents of volumes associated with the container. If a volume is mounted on top of an existing directory in the container, docker export will export the contents of the underlying directory, not the contents of the volume.
Refer to Backup, restore, or migrate data volumes in the user guide for examples on exporting data in a volume.
This points to this documentation. Please follow the steps there to export the information stored in the volume.
You're probably looking for something like this:
docker run --rm --volumes-from <containerId> -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /cookbook
This would create a file backup.tar with the contents of the container's /cookbook directory and store it in the current directory of the host. You could then use this tar file to import it in another container.
Essentially, there are three ways to do persistence in Docker:
You can keep files in a volume, which is a filesystem managed by Docker. This is what happens in your example: because the /cookbook directory is part of a volume, your file does not get commited/exported with the image. It does however get stored in the volume, so if you remount the same volume in a different container, you will find your file there. You can list your volumes using docker volume ls. As you can see, you should probably give your volumes names if you plan to reuse them. You can mount an existing volume, or create a new one, if the name does not exist, with
docker run -v name:/directory ubuntu
You can keep files as part of the image. If you commit the container, all changes to its file hierarchy are stored in the new image except those made to mounted volumes. So if you just get rid of the -v flag, your file shows up in the commit.
You can bind mount a directory from the host machine to the container, by using the -v /hostdir:/targetdir syntax. The container then simply has access to a directory of the host machine.
Docker commit allows you to create an image from a container and its data (mounted volumes will be ignored)
I use the command:
docker run -it -v /myhostpath:/dockerpath $container
so I get a mapping from a host dir to a docker dir, and I do see the files shared by both dirs.
After that, I exit the container, commit and save the image.
However, when I open the container again, the shared files are gone.
Could anyone tell me what is happening? Thanks!
The -v option is like using the mount command to mount an external file system. Directories mapped into a container with -v are not in the container's filesystem. Therefore committing the changes in the container's filesystem to a new image does not include these external files.
If you want to copy some external files to the container, you would need to use -v to temporarily mount the directory and then use a cp command to copy to a directory local to the container, and then commit.
Or, you could include the files using ADD or COPY in your Dockerfile
If the external files can be thought of as essential input or output for the container, that is separate from the container and could be needed elsewhere (log files, calculation results, database data, etc), or are supposed to be kept secret, then you should not copy them to the container. Instead, continue using -v to mount that directory on the container.