Is it possible to mount an overlay fs inside a (privileged) docker container? At least my intuitive approach, which works fine outside of a container, fails:
> mkdir /tmp/{up,low,work,merged}
> mount -t overlay overlay -o lowerdir=/tmp/low/,upperdir=/tmp/up/,workdir=/tmp/work/ /tmp/merged/
mount: /tmp/merged: wrong fs type, bad option, bad superblock on overlay, missing codepage or helper program, or other error.
Additional information:
Docker version 18.09.1, build 4c52b90
Kernel 4.19.0-8-amd64
Debian 10 (host and docker-image)
Found something that worked! Mounting the workdir and upperdir as tmpfs does the trick for me.
Like so:
> mkdir /tmp/overlay
> mkdir /tmp/{low,merged}
> mount -t tmpfs tmpfs /tmp/overlay
> mkdir /tmp/overlay/{up,work}
> mount -t overlay overlay -o lowerdir=/tmp/low/,upperdir=/tmp/overlay/up/,workdir=/tmp/overlay/work/ /tmp/merged/
I'd still be interested in an explanation why creating an overlay w/o tmpfs fails within a docker container?
How to mount an overlayfs inside a docker container:
https://gist.github.com/detunized/7c8fc4c37b49c5475e68ef9574587eee
Basically, you need to run the container with either --privileged or the more secure --cap-add=SYS_ADMIN.
This is a bit of a guess but I suspect it is because docker is already using overlayfs and overlayfs is refusing to use upperdir as another overlayfs.
I suspect this may be due to whiteout files:
In order to support rm and rmdir without changing the lower
filesystem, an overlay filesystem needs to record in the upper
filesystem that files have been removed. This is done using whiteouts
and opaque directories (non-directories are always opaque).
A whiteout is created as a character device with 0/0 device number.
When a whiteout is found in the upper level of a merged directory, any
matching name in the lower level is ignored, and the whiteout itself
is also hidden.
To delete a file that exists in a lowerdir, overlayfs will create a whiteout file and hides all whiteout files (device number 0,0). This logically means that you cannot create a character device file with number 0,0 inside an overlayfs because that must be hidden by overlayfs itself.
If you were allowed to use an overlayfs as an upperdir it wouldn't be able to create blackout files and therefore wouldn't be able to rm or rmdir any files from the lower layers. Because it can't create a character device file with number 0,0 on another overlayfs.
Related
I am very new to docker so please pardon if anything stupid :P
I have docker running on my cloud server and was facing issue of running out of space because of docker overlay files. So I mounted 100GB of storage to the server at
/home/<user>/data
and in daemon.json configured the docker root directory to this newly mounted storage and copied all the old files but after that also when I check
df -h
overlay file shows size 36G. Am I doing something wrong
How can I increase this overlay to completely utilize the storage ?
PS: Also when it starts filling up it doesn't increase space it just fills up and all the apps stop working
Docker stores images, containers, and volumes under /var/lib/docker by default. If you haven't mounted another filesystem there, you are likely looking at the free space on your root filesystem.
When mounting another filesystem in this location, you likely want to move the current directory aside so you can copy it into the new filesystem. If you do restore the content, be sure to use a command that preserves ownership, permissions, and symlinks (I believe cp -a and tar both do this).
Also, make sure the docker engine is not running when you replace this directory, and be sure the filesystem type matches your current root filesystem type, or is compatible with your graph driver.
I'm running a Mac-native Docker (no virtualbox/docker-machine).
I have a huge image with a lot of infrastructure in it (Postgres, etc.).
I have run cleanup scripts to get rid of a lot of cruft--unused images and so forth.
When I run my image I get an error like:
could not create directory "/var/lib/postgresql/data/pg_xlog": No space left on device
On my host Mac /var is sitting at 60% space available and generally my disk has lots of storage free.
Is this some Docker configuration I need to bump up to give it more resources?
Relevant lines from mount inside docker:
none on / type aufs (rw,relatime,si=5b19fc7476f7db86,dio,dirperm1)
/dev/vda1 on /data type ext4 (rw,relatime,data=ordered)
/dev/vda1 on /etc/resolv.conf type ext4 (rw,relatime,data=ordered)
/dev/vda1 on /etc/hostname type ext4 (rw,relatime,data=ordered)
/dev/vda1 on /etc/hosts type ext4 (rw,relatime,data=ordered)
/dev/vda1 on /var/lib/postgresql/data type ext4 (rw,relatime,data=ordered)
Here’s df:
[11:14]
Filesystem 1K-blocks Used Available Use% Mounted on
none 202054928 4333016 187269304 3% /
tmpfs 1022788 0 1022788 0% /dev
tmpfs 1022788 0 1022788 0% /sys/fs/cgroup
/dev/vda1 202054928 4333016 187269304 3% /data
shm 65536 4 65532 1% /dev/shm
tmpfs 204560 284 204276 1% /run/docker.sock
I haven't found many options for this, the main issue in github is https://github.com/docker/for-mac/issues/371
Some of the options suggested there are:
If you can remove all images/containers, you can follow these instructions:
docker rm $(docker ps -a -q)
docker rmi $(docker images -q)
docker volume rm $(docker volume ls |awk '{print $2}')
rm -rf ~/Library/Containers/com.docker.docker/Data/*
You can try to prune all unused images/containers but this has proven not very effective:
docker system prune
Use a template image that is larger, install qemu using homebrew and move the image around, see this specific comment: https://github.com/docker/for-mac/issues/371#issuecomment-242047368 but you need to have at least 2x of space left to do this without losing containers/images.
See also: How do you get around the size limitation of Docker.qcow2 in the Docker for Mac?
And https://forums.docker.com/t/no-space-left-on-device-error/10894/26
I ran into the same issue, running docker system prune --volumes resolved the problem.
"Volumes are not pruned by default, and you must specify the --volumes flag for docker system prune to prune volumes."
See: https://docs.docker.com/config/pruning/#prune-everything
I ran into this recently on with a docker installation on linux that uses the devicemapper storage driver (default). There was indeed a docker configuration I needed to change to fix this.
Docker images are made of read-only layers of filesystem snapshots, each layer created by a command in your Dockerfile, which are built on top of a common base storage snapshot. The base snapshot is shared by all your images and has a file system with a default size of 10GB. When you run your image you get a new writable layer on top of all the layers in the image, so you can add new files in your running container but it's still eventually based on the same base snapshot with the 10GB filesystem. This is at least true for devicemapper, not sure about other drivers. Here is the relevant documentation from docker.
To change this default value to something else, there's a daemon parameter you can set, e.g. docker daemon --storage-opt dm.basesize=100G. Since you probably don't run the daemon manually need to edit the docker daemon options in some file, depending on how you run the docker daemon. With docker for mac you can edit the daemon parameters as JSON in the preferences under Daemon->Advanced. You probably need to add something like this:
{
"storage-opts": ["dm.basesize=100G"]
}
(but like I said, I had this problem on linux, so didn't try the above).
Anyway in order for this to take effect, you'll need to remove all your existing images (so that they're re-created on top of the new base snapshot with the new size). See storage driver options.
I'd like to control whether docker operates on a persistent storage or on a persistent storage overlayed with a volatile one.
It is because I have the filesystem on an SD card (Raspberry Pi) and it needs to last long. I mostly want to operate on a read-only filesystem (ext4) overlayed with tmpfs (run containers on it), but when I detect that an update is available I want to unmount overlayfs, switch filesystem as read-write, update the image, then switch everything back to the tmpfs-overlayed read-only filesystem.
# mv /var/lib/docker /var/lib/docker~
# mkdir -p /var/lib/docker /tmp/docker /tmp/work
# mount -t overlay -o lowerdir=/var/lib/docker~,upperdir=/tmp/docker,workdir=/tmp/work overlay /var/lib/docker
# docker daemon --storage-driver devicemapper
I tried two storage drivers: overlay2 and devicemapper (loop). The former refused to work on overlayfs underlying filesystem (it is also mentioned in the documentation that it is not supported), the latter consumes all my memory and then Docker gets killed by OS. The behaviour is the same for Raspberry Pi and my PC.
The only storage that should work is vfs, but from what I have read, it is very inefficient for storage (no Copy-on-Write), so it is of no use for me.
Now I'm giving a try to do it with aufs storage driver and overlayfs backing filesystem (Docker documentation doesn't state that it is disabled). I hope it will work but it has some disadvantages: aufs is not supported by mainline Linux kernel.
Is there some other way to switch between the two filesystems? Or could the SD card saving be done by some completely different way (e.g. running in-memory containers)?
EDIT: Sorry, this finally DOES NOT WORK!!!. Docker daemon starts but is unable to create containers. This is the error:
Handler for POST /v1.24/containers/create returned error: error creating aufs mount to /var/lib/docker/aufs/mnt c549130a63857658f8675fd84296afae46293a9f7ae54e9ee04e83c231db600f-init: invalid argument
aufs storage driver with overlayfs backing filesystem works. For now it seems like the only option, however I'm not satisfied with the solution, because it looks like a hack to me and because aufs is not in mainline kernel so I needed to compile the kernel myself.
This is how I did it (it's quite a hack, please advice me to do it better):
on my PC:
$ git clone https://github.com/p4l1ly/rpi-kernel
$ cd rpi-kernel
$ vagrant up
...wait some quite long time...
$ vagrant ssh
$ cp /var/kernel_build/results/kernel-20161003-100112/rpi2_3/kernel7.img /vagrant/
$ exit
$ sudo cp kernel7.img /mnt
then on the SD card:
# mv /var/lib/docker /var/lib/docker~
# mkdir -p /var/lib/docker /tmp/docker /tmp/work
# mount -t overlay -o lowerdir=/var/lib/docker~,upperdir=/tmp/docker,workdir=/tmp/work overlay /var/lib/docker
# docker daemon --storage-driver aufs
Is it possible to convert an .img file containing an OS (Arch Linux) into a Docker image? More precisely I want to dockerize a RuneAudio Raspberry Pi image.
Producing a Docker image from a full operating system image is often a sub-optimal process. The operating system image is going to include a variety of things that are simply not necessary in the Docker environment, which simply means that the resulting image is going to be unnecessarily large.
That said, if you want to try this anyway, the guestfish command from the libguestfs package makes this very simple:
guestfish --ro -a RuneAudio_rpi_0.3-beta_20141029_2GB.img -m /dev/sda5:/ tar-out / - |
docker import - runeaudio
That will create a runeaudio docker image with the contents of the RuneAudio_rpi_0.3-beta_20141029_2GB.img disk image. Note that this will, of course, only run under Docker running on a Raspberry Pi, and the resulting image isn't necessarily going to work without further modification.
You can also accomplish the same thing by mounting the disk image locally:
losetup -P /dev/loop0 RuneAudio_rpi_0.3-beta_20141029_2GB.img
mount /dev/loop0p5 /mnt
tar -C /mnt -cf - | docker import - runeaudio
umount /mnt
losetup -d /dev/loop0
I like guestfish because it doesn't require root access, and doesn't require mucking about with loop devices and mountpoints, so there's less setup and cleanup.
Is there a way to install Docker on a specific volume ?
When I install Docker on Amazon Linux with the following command :
sudo yum install docker
and then start the docker service using :
sudo service docker start
It creates two Data Spaces :
Data file: /dev/loop0
Metadata file: /dev/loop1
How can I have those spaces be on a given volume such as /mnt/docker for example ?
Those are device files. They will always be in /dev (actually not, but let's just assume for sake of simplicity, here). loop0 and loop1 are loop devices that are backed by the actual Docker volume files. You can easily see this using losetup -l:
> losetup -l
NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
/dev/loop0 0 0 1 0 /var/lib/docker/devicemapper/devicemapper/data
/dev/loop1 0 0 1 0 /var/lib/docker/devicemapper/devicemapper/metadata
What you might want to do (depending on your file system layout) is moving the Docker runtime directory somewhere else (default is /var/lib/docker; all Docker volumes and images are stored there). For this, you can supply the -g flag to the Docker daemon.
In CentOS/Fedora/RHEL (and probably, because it's based on RHEL, also Amazon Linux), you can modify the /etc/sysconfig/docker file for this (look for an OPTIONS variable). In Ubuntu/Debian /etc/default/docker would be the place to look.
I was able to get docker to store all of its data (containers and their data volumes) at a different place in the file system (my EBS volume) by editing /etc/sysconfig/docker
which has the line:
OPTIONS="--default-ulimit nofile=1024:4096"
I added the -g option, as documented here
OPTIONS="--default-ulimit nofile=1024:4096 --graph=/home/ec2-user/myvolume"
where myvolume is the directory where I mounted my EBS volume. Of course you need to stop and restart the docker daemon for this to take effect.
This is on Amazon Linux. Apparently the docker config file is /etc/default/docker on some Linuxes.