I'm reading the book The docker book version v1.2.0, in chapter 6 in page 178 it said that
if you deleted the last container that uses a volume, the volume will disappear. Be carful about deleting containers that might hold volumes with data you need
But when I tried it, the volume isn't deleted.
Is that just because I'm reading old version ??
The newer versions will not delete your volume, unless you're using the parameter --volumes or -v.
https://docs.docker.com/engine/reference/commandline/rm/#options
Related
This issue is really strange, but I've no idea how to resolve it. I'm running Win10 and I've got WSL2 on it. When I start Docker with it using the "WSL 2 Based engine", it retrieves some deleted folders out of nowhere. I delete them every single time and after I've restarted my PC and started up Docker again, the folders are back.
There are absolutely no docker containers running ("docker ps" returns nothing) so it couldn't be that some rogue volume definition is being ran along with some container. The folders also only appear once I start up Docker.
The directory inside of which the zombie folders appear is also the source for a Mutagen volume when the containers are running, but as I said - no containers are running.
I ran into the same problem and resolved it by running Troubleshoot from Docker for Windows and purge data on WSL2.
I've not found a solution to selectively clear just that one folder that keeps getting raised from the dead.
Be there a machine that runs various docker projects. Each docker container is regularly replaced/stopped/started as soon as newer versions arrive from the build system.
How does a backup concept for such a machine look like?
Looking into similar questions [1] the correct path to a working backup/restore procedure is not immediately clear to me. My current understanding is something like:
Backup
Use scripts to create images and containers. Store/Backup scripts in your favorite Version Control System. Use version tags to pull docker images. Don't use latest tag.
Exclude /var/lib/docker/overlay2 from backup (to prevent backing up dangling and temporary stuff)
Use named volumes only. Volumes can be saved and restored from backup. For database stuff extra work has to be done. Eventually consider to tar volumes to extra folder [2].
docker prune daily to remove dangling stuff
Restore
Make sure all named volumes are back in place.
Fetch scripts from version control to recreate images as needed. Use docker run to recreate containers.
Application specific tasks - restore databases from dumps , etc.
[1]
How can I backup a Docker-container with its data-volumes?
[2] https://stackoverflow.com/a/48112996/1485527
Don't use latest tag in your images. Set correct tags (like v0.0.1, v0.0.2, etc) for your images and you can have all of your versions in a docker registry.
You should prefer to use stateless container
What is about docker volume? You can use it https://docs.docker.com/storage/volumes/
If you use bind mount volume you can manually save you files in archive for backup
The Docker docs state:
Warning: Do not directly manipulate any files or directories within /var/lib/docker/. These files and directories are managed by Docker.
Let's say someone hasn't read that hint and deleted some files from /var/lib/docker/aufs/diff to free up some disk space. These files didn't live in a Docker volume and are not part of the original Docker image but have been created in the container writable layer. Restarting the given container frees up the disk space but are there any known side effects?
And for the next time: Does removing that kind of files or directories from within the container (via docker exec .. rm ..) result in a proper removal or are they only marked as deleted? The documentation currently doesn't describe this special case.
Restarting the given container frees up the disk space but are there any known side effects?
As you stated in your question, you should not "manipulate any files or directories within /var/lib/docker/", as any side-effect may appear and no documentation trace anything about this: it's internal Docker plumbing which may highly change other Docker versions, ut's not supposed to be exposed to end-users nor be tempered with. You could look at Docker code for your Docker version and all it's dependencies to understand what happened, but it's not really practical :-)
are there any known side effects?
There maybe be side effects - I insist on the may as anything can happen depending on your Docker version and configuration. Even if it may seem to be working, some things may be broken.
Well known side effect is Docker installation corruption, which may have present itself in various fashions: random container crash, data loss, unexplained bug, etc.
Best case scenario, you just discarded some data in your container and everything will work fine in the future.
Not-so-good scenario: you actually broke something in your installation and corrupted it, you'll be better of re-installing Docker entirely.
Does removing that kind of files or directories from within the container (via docker exec .. rm ..) result in a proper removal or are they only marked as deleted?
Deleting a file in the container will not always remove it from the system, it depends on the drive your are using. Doc has a section about writing files for all of them:
AUFS - it seemed implied that file is deleted, AUFS will copy the file from the image layer and work on it, it should then delete the copy
When a file is deleted within a container, a whiteout file is created in the container layer. The version of the file in the image layer is not deleted [...] Subsequent writes to the same file operate against the copy of the file already copied up to the container.
BTRFS - deleted and space reclaimed, doc is quite clear:
If a container creates a file and then deletes it, this operation is performed in the Btrfs filesystem itself and the space is reclaimed.
devicemapper - may not be deleted depending on config:
if you are using direct-lvm, the blocks are freed. If you use loop-lvm, the blocks may not be freed
OverlayFS - seemed implied that file is deleted, but the image file is kept
When a file is deleted within a container, a whiteout file is created in the container (upperdir). The version of the file in the image layer (lowerdir) is not deleted
ZFS - deleted:
If you create and then delete a file or directory within the container’s writable layer, the blocks are reclaimed by the zpool.
VFS is using a copy of the previous layer and work directly in a directory representing that layer, a deletion in the container should probably delete it from the related directory on host machine
The documentation currently doesn't describe this special case.
Yes, and it probably won't ;)
I am learning CDH and Docker and didn't have prior experiene in setting up both tools. After reading documentation i managed to run CDH docker in mac environment and also completed example given in quick start guid. But when next day when i started mac book again to learn something new but i didn't find my previous work which i found very strange and even couldn't see container running which seems fine to me.
What i really want to do is i don't want to loose my work even after stoping docker container. could you please guid me how do i configure docker so that i will not loose my work even after restarting docker again?
Every instance of a docker run will allocate a new filesystem, essentially starting from scratch.
If you actually want to "save" your work, then you need to volume mount (using -v docker flag) your local filesystem into the container for at least the following directories.
HDFS Data Directory
NameNode Data Directory
/home/cloudera
I think the hadoop data folders are somewhere under /var/lib/hadoop-*, by default
The better alternative for saving your workloads would be the CDH VM, where it actually has a persistent HDD associated with it.
we have some docker server built, but when the old image/container gets renewed/removed the old subvolumes are not deleted, so it now end up with we have 10 running images/containers but 50+ subvolumes, it fill up the filesystems pretty quick.
we would like to delete those subvolumes which are no longer needed but how do we find these subvoluems not anymore related to the images/containers?
thanks
Eric