I'm developing using Docker on a multiboot setup under both Fedora and Ubuntu on my laptop. I need this to rule out issues with selinux and/or apparmor so my build will work for both red hat(and friends) and debian(and friends).
I'm using devicemapper in thin pool lvm configuration as storage backend. This was configured using docker-storage-setup tool under Fedora.
I would like to share my docker images and containers to both Fedora (/ is formatted as ext4fs on lvm) and Ubuntu environments (/ is formatted as btrfs also on lvm) to save space.
However after one Docker system has started and taken over the docker thinpool, the other Docker system could not use the same docker thinpool.
This is the error:
Error starting daemon: error initializing graphdriver: devmapper: Unable to take ownership of thin-pool ("my docker thin pool") that already has used data blocks
Based on that it seems to have this limitation by design. In that case would anyone elaborate on my particular use case and is there another way to share my docker devicemapper thin pool with several linux systems so I can save space and not have duplicate images/containers?
In a bug report Eric Paris says:
IF you are using device mapper (instead of loopback) /var/lib/docker contains metadata informing docker about the contents of the device mapper storage area. If you delete /var/lib/docker that metadata is lost. Docker is then able to detect that the thin pool has data but docker is unable to make use of that information. The only solution is to delete the thin pool and recreate it so that both the thin pool and the metadata in /var/lib/docker will be empty.
So syncing parts of /var/lib/docker may be a solution.
Related
When trying to remove a docker container (for example when trying to docker-compose down) I always get these errors:
ERROR: for <my_container> container d8424f80ef124c2f3dd8f22a8fe8273f294e8e63954e7f318db93993458bac27: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/64311f2ee553a5d42291afa316db7aa392a29687ffa61971b4454a9be026b3c4/merged: device or resource busy
Common advice like restarting the docker service, pruning or force removing the containers doesn't work. The only thing that I found that works is manually unmounting with sudo umount /home/virtfs/xxxxx/var/lib/docker/overlay2/<container_id>/merged, then I'm able to remove the container.
My OS is CentOS Linux release 7.9.2009 (Core) with kernel version 3.10.0-1127.19.1.el7.x86_64. I thought this was maybe due to overlay2 clashing with CentOS, but according to this page my CentOS/kernel version should work. It would be great to find a solution to this, because I ideally want to docker-compose down without having to use elevated privileges to umount beforehand.
From the error log, it is observed that the file mounting is involved
Execute the following command to view the related processes
grep 64311f2ee553a5d42291afa316db7aa392a29687ffa61971b4454a9be026b3c4 /proc/*/mountinfo
ps -ef | grep "The process ID obtained by the grep command above"
Stop the occupied process
Then delete the container
Often this happens when there are no processes listed as blocking, then you know it's a kernel module blocking it.
The most likely culprits are nfs (not sure why you'd run that in docker), or files inside docker that are bind-mounted, sometimes the automatic ones, such as perhaps ones created by systemd-networkd.
Overlay2 was phased out by Ubuntu for a reason. CentOS is at its end of life, so this problem might be resolved already in your most likely upgrade path, to Rocky Linux. Alternately you can enter the jungle of migrating your docker storage engine.
Or you can just get rid of the package or software hogging it in the first place, if you can. But you'll have to share more info on what it is for help on that.
I have a docker environment with 2 containers (Jenkins and Nexus, both with their own named volume).
I have a daily cron-job which deletes unused containers and images. This is working fine. But the problem is inside my devicemapper:
du -sh /var/lib/docker/
30G docker/
I can each folder in my docker folder:
Volumes (big, but that's normal in my case):
/var/lib/docker# du -sh volumes/
14G volumes/
Containers:
/var/lib/docker# du -sh containers/
3.2M containers/
Images:
/var/lib/docker# du -sh image/
5.8M image/
Devicemapper:
/var/lib/docker# du -sh devicemapper/
16G devicemapper/
/var/lib/docker/devicemapper/mnt is 7.3G
/var/lib/docker/devicemapper/devicemapper is 8.1G
Docker info:
Storage Driver: devicemapper
Pool Name: docker-202:1-xxx-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: ext4
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 5.377 GB
Data Space Total: 107.4 GB
Data Space Available: 28.8 GB
Metadata Space Used: 6.148 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.141 GB
Udev Sync Supported: true
What is this space and am I able to clean this without breaking stuff?
Don't use a devicemapper loop file for anything serious! Docker has big warnings about this.
The /var/lib/docker/devicemapper/devicemapper directory contains the sparse loop files that contain all the data that docker mounts. So you would need to use lvm tools to trawl around them and do things. Have a read though the remove issues with devicemapper, they are kinda sorta resolved but maybe not.
I would move away from devicemapper where possible or use LVM thin pools on anything RHEL based. If you can't change storage drivers, the same procedure will at least clear up any allocated sparse space you can't reclaim.
Changing the docker storage driver
Changing storage driver will require dumping your /var/lib/docker directories which contains all your docker data. There are ways to save portions of it but that involves messing around with Docker internals. Better to commit and export any containers or volumes you want to keep and import them after the change. Otherwise you will have a fresh, blank Docker install!
Export data
Stop Docker
Remove /var/lib/docker
Modify your docker startup to use the new storage driver.
Set --storage-driver=<name> in /lib/systemd/system/docker.service or /etc/systemd/system/docker.service or /etc/default/docker or /etc/sysconfig/docker
Start Docker
Import Data
AUFS
AUFS is not in the mainline kernel (and never will be) which means distro's have to actively include it somehow. For Ubuntu it's in the linux-image-extra packages.
apt-get install linux-image-extra-$(uname -r) linux-image-extra-virtual
Then change the storage driver option to --storage-driver=aufs
OverlayFS
OverlayFS is already available in Ubuntu, just change the storage driver to --storage-driver=overlay2 or --storage-driver=overlay if you are still using a 3.x kernel
I'm not sure how good an idea this is right now. It can't be much worse than the loop file but
The overlay2 driver is pretty solid for dev use but isn't considered production ready yet (e.g. Docker Enterprise don't provide support) but it is being pushed to become the standard driver due to the AUFS/Kernel issues.
Direct LVM Thin Pool
Instead of the devicemapper loop file you can use an LVM thin pool directly. RHEL makes this easy with a docker-storage-setup utility that distributed with their EPEL docker package. Docker have detailed steps for setting up the volumes manually.
--storage-driver=devicemapper \
--storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool \
--storage-opt dm.use_deferred_removal=true
Docker 17.06+ supports managing simple direct-lvm block device setups for you.
Just don't run out of space in the LVM volume, ever. You end up with an unresponsive Docker daemon that needs to be killed and then LVM resources that are still in use that are hard to clean up.
A periodic docker system prune -a works for me on systems where I use devicemapper and not the LVM thinpool. The pattern I use is:
I label any containers, images, etc with label "protected" if I want them to be exempt from cleanup
I then periodically run docker system prune -a --filter=label!=protected (either manually or on cron with -f)
Labeling examples:
docker run --label protected ...
docker create --label=protected=true ...
For images, Dockerfile's LABEL, eg LABEL protected=true
To add a label to an existing image that I cannot easily rebuild, I make a 2 line Dockerfile with the above, build a new image, then switch the new image for the old one (tag).
General Docker label documentation
First, what is devicemapper (official documentation)
Device Mapper has been included in the mainline Linux kernel since version 2.6.9 [in 2005]. It is a core part of RHEL family of Linux distributions.
The devicemapper driver stores every image and container on its own virtual device. These devices are thin-provisioned copy-on-write snapshot devices.
Device Mapper technology works at the block level rather than the file level. This means that devicemapper storage driver's thin provisioning and copy-on-write operations work with blocks rather than entire files.
The devicemapper is the default Docker storage driver on some Linux distributions.
Docker hosts running the devicemapper storage driver default to a configuration mode known as loop-lvm. This mode uses sparse files to build the thin pool used by image and container snapshots
Docker 1.10 [from 2016] and later no longer matches image layer IDs with directory names in /var/lib/docker.
However, there are two key directories.
The /var/lib/docker/devicemapper/mnt directory contains the mount points for image and container layers.
The /var/lib/docker/devicemapper/metadatadirectory contains one file for every image layer and container snapshot.
If your docker info does show your Storage Driver is devicemapper (and not aufs), proceed with caution with those folders.
See for instance issue 18867.
I faced the same issue where in my /var/lib/docker/devicemapper/devicemapper/data file has reached ~91% of root volume(~45G of 50G). I tried removing all the unwanted images, deleted volumes, nothing helped in reducing this file.
Did a few googling and understood that the "data" files is loopback-mounted sparse files and docker uses it to store the mount locations and other files we would have stored inside the containers.
Finally I removed all the images which were run before and stopped
Warning: Deletes all docker containers
docker rm $(docker ps -aq)
The reduced the devicemapper file significantly. Hope this may help you
.
I've been tasked with working with the CentOS Atomic Host distribution which comes preinstalled with Docker. My problem is I can pull from a host registry without a problem (but I don't know where it's stored), but what I'd really like to do is just scp/sftp an image over to my client PC and "docker load" the image.
If I choose auto installation, I get a "cah-docker--pool_tdata" that's 48.9 and a "cah-docker--pool_tmeta" and the /dev/mapper/cash-root is only 3 GB which can't hold an image.
Where should I be transferring the files to and could anyone give me a rundown on why these partitions are like this? I couldn't find anything in the docs about it.
Atomic Host's only purpose is to run Docker. From the Project Atomic website:
Atomic Host is a lightweight, immutable platform, designed with the sole purpose of running containerized applications.
The only place where such a host needs a lot of disk space is the Docker pool. Therefore, the installer gives the pool as much disk space as possible and keeps the rest of the partitions very small.
To solve your problem, try to pipe the output of docker save directly into docker load. From the host where you want to copy the image from:
docker save <image> | ssh <target_host> 'docker load'
This can be improved by following this answer by kolypto:
docker save <image> | bzip2 | pv | \
ssh <target_host> 'bunzip2 | docker load'
I was debugging my dockerfile and downloaded some huge packages too many times. Now everytime i run any docker instructions, there's always "Cannot connect to the Docker daemon". I checked a bit and I think it's because the storage is full. But I have no idea how to fix this. Can anyone help me out? Here's a snapshot of the error.
A snapshot
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
WARN[0000] You are running linux kernel version 2.6.32-431.el6.x86_64, which might be unstable running docker. Please upgrade your kernel to 3.10.0.
INFO[0000] [graphdriver] using prior storage driver "devicemapper"
WARN[0000] Running modprobe bridge nf_nat failed with message: , error: exit status 1
ERRO[0001] Error during graph storage driver.Cleanup(): device or resource busy
FATA[0001] Error starting daemon: Some kind of disk I/O error occurred: disk I/O error
Filesystem Size Used Avail Use% Mounted on
/dev/xvde 7.9G 7.9G 0 100% /
You should upgrade your kernel, Docker does not play too well with 2.6.
Now, assuming it was working anyway, you partition as only 8Gb, which is very low. Once you run out of space, you can endup in a situation where you can't cleanup and reduce your disk usage. I know Docker did a lot of work over the past couple of years towards it, but Devicemapper (your storage driver) used to require some disk space in order to remove anything.
In your current situation, you probably want to call it a day and destroy everything and start again with a bigger partition (and a newer kernel if possible).
I want to test docker in my CentOS 7.1 box, I got this warning:
[root#docker1 ~]# docker run busybox /bin/echo Hello Docker
Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
Hello Docker
I want to know the reason and how to suppress this warning.
The CentOS instance is running in virtualbox created by vagrant.
The warning message occurs because your Docker storage configuration is using a "loopback device" -- a virtual block device such as /dev/loop0 that is actually backed by a file on your filesystem. This was never meant as anything more than a quick hack to get Docker up and running quickly as a proof of concept.
You don't want to suppress the warning; you want to fix your storage configuration such that the warning is no longer issued. The easiest way to do this is to assign some local disk space for use by Docker's devicemapper storage driver and use that.
If you're using LVM and have some free space available on your volume group, this is relatively easy. For example, to give docker 100G of space, first create a data and metadata volume:
# lvcreate -n docker-data -L 100G /dev/my-vg
# lvcreate -n docker-metadata -L1G /dev/my-vg
And then configure Docker to use this space by editing /etc/sysconfig/docker-storage to look like:
DOCKER_STORAGE_OPTIONS=-s devicemapper --storage-opt dm.datadev=/dev/my-vg/docker-data --storage-opt dm.metadatadev=/dev/my-vg/docker-metadata
If you're not using LVM or don't have free space available on your VG, you could expose some other block device (e.g., a spare disk or partition) to Docker in a similar fashion.
There are some interesting notes on this topic here.
Thanks. This was driving me crazy. I thought bash was outputting this message. I was about to submit a bug against bash. Unfortunately, none of the options presented are viable on a laptop or such where disk is fully utilized. Here is my answer for that scenario.
Here is what I used in the /etc/sysconfig/docker-storage on my laptop:
DOCKER_STORAGE_OPTIONS="--storage-opt dm.no_warn_on_loop_devices=true"
Note: I had to restart the docker service for this to have an effect. On Fedora the command for that is:
systemctl stop docker
systemctl start docker
There is also just a restart command (systemctl restart docker), but it is a good idea to check to make sure stop really worked before starting again.
If you don't mind disabling SELinux in your containers, another option is to use overlay. Here is a link that describes that fully:
http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docker-storage-drivers/
In summary for /etc/sysconfig/docker:
OPTIONS='--selinux-enabled=false --log-driver=journald'
and for /etc/sysconfig/docker-storage:
DOCKER_STORAGE_OPTIONS=-s overlay
When you change a storage type, restarting docker will destroy your complete image and container store. You may as well everything up in the /var/lib/docker folder when doing this:
systemctl stop docker
rm -rf /var/lib/docker
dnf reinstall docker
systemctl start docker
In RHEL 6.6 any user with docker access can access my private keys, and run applications as root with the most trivial of hacks via volumes. SELinux is the one thing that prevents that in Fedora and RHEL 7. That said, it is not clear how much of the additional RHEL 7 security comes from SELinux outside the container and how much inside the container...
Generally, loopback devices are fine for instances where the limit of 100GB maximum and a slightly reduced performance are not a problem. The only issue I can find is the docker store can be corrupt if you have a disk full error while running... That can probably be avoided with quotas, or other simple solutions.
However, for a production instance it is definitely worth the time and effort to set this up correctly.
100G may excessive for your production instance. Containers and images are fairly small. Many organizations are running docker containers within VM's as an additional measure of security and isolation. If so, you might have a fairly small number of containers running per VM. In which case even 10G might be sufficient.
One final note. Even if you are using direct lvm, you probable want a additional filesystem for /var/lib/docker. The reason is the command "docker load" will create an uncompressed version of the images being loaded in this folder before adding it to the data store. So if you are trying to keep it small and light then explore options other than direct lvm.
#Igor Ganapolsky Feb and #Mincă Daniel Andrei
Check this:
systemctl edit docker --full
If directive EnvironmentFile is not listed in [Service] block, then no luck (I also have this problem on Centos7), but you can extend standard systemd unit like this:
systemctl edit docker
EnvironmentFile=-/etc/sysconfig/docker
ExecStart=
ExecStart=/usr/bin/dockerd $OPTIONS
And create a file /etc/sysconfig/docker with content:
OPTIONS="-s overlay --storage-opt dm.no_warn_on_loop_devices=true"