So far, I really only understand VOLUME as a way to
Specify a directory inside of a data container that will be persistent
Specify a location that will link to your host container
What I am failing to understand is why I see so many Dockerfiles that use VOLUME /path/to/app or even worse VOLUME /var/lib/mysql. I understand that you might want to create a container that has this volume and then use --volumes-from to link to that container for persistence. But why would you make that specification inside the container that is actually using that data. How does it help? As far as I can see, VOLUME /var/data is not any different than just saying RUN mkdir /var/data. How are volumes beneficial when they are not inside a data container, shared with the host, or being used by other containers?
Docker images and Docker containers have a layered file system which is slow. By defining directories as data volumes you instruct docker to let those directories outside of the slow layered file system. This as multiple consequences among which:
fast file system
ability to share a volume between multiple containers
persistence (as long as at least one container that use that volume exists)
This is why volumes are not only a commodity but a necessity for directories for which good I/O performances is expected.
As far as I can see, VOLUME /var/data is not any different than just saying RUN mkdir /var/data.
The difference is that with volumes the directory /var/data is a mount point on a different (and faster) file system. You can witness that /var/data is not just another directory by running the mount command in a container:
$ docker run --rm -v /var/data busybox mount
rootfs on / type rootfs (rw)
none on / type aufs (rw,relatime,si=6c354c296f850c3c)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,mode=755)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
/dev/mapper/vg0-root on /etc/resolv.conf type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/vg0-root on /etc/hostname type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/vg0-root on /etc/hosts type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/vg0-root on /var/data type ext4 (rw,relatime,errors=remount-ro,data=ordered)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime)
tmpfs on /proc/kcore type tmpfs (rw,nosuid,mode=755)
/ is on a aufs layered (and slow) file system
/var/data is on a ext4 (and fast) file system
Related
I'm running a Mac-native Docker (no virtualbox/docker-machine).
I have a huge image with a lot of infrastructure in it (Postgres, etc.).
I have run cleanup scripts to get rid of a lot of cruft--unused images and so forth.
When I run my image I get an error like:
could not create directory "/var/lib/postgresql/data/pg_xlog": No space left on device
On my host Mac /var is sitting at 60% space available and generally my disk has lots of storage free.
Is this some Docker configuration I need to bump up to give it more resources?
Relevant lines from mount inside docker:
none on / type aufs (rw,relatime,si=5b19fc7476f7db86,dio,dirperm1)
/dev/vda1 on /data type ext4 (rw,relatime,data=ordered)
/dev/vda1 on /etc/resolv.conf type ext4 (rw,relatime,data=ordered)
/dev/vda1 on /etc/hostname type ext4 (rw,relatime,data=ordered)
/dev/vda1 on /etc/hosts type ext4 (rw,relatime,data=ordered)
/dev/vda1 on /var/lib/postgresql/data type ext4 (rw,relatime,data=ordered)
Here’s df:
[11:14]
Filesystem 1K-blocks Used Available Use% Mounted on
none 202054928 4333016 187269304 3% /
tmpfs 1022788 0 1022788 0% /dev
tmpfs 1022788 0 1022788 0% /sys/fs/cgroup
/dev/vda1 202054928 4333016 187269304 3% /data
shm 65536 4 65532 1% /dev/shm
tmpfs 204560 284 204276 1% /run/docker.sock
I haven't found many options for this, the main issue in github is https://github.com/docker/for-mac/issues/371
Some of the options suggested there are:
If you can remove all images/containers, you can follow these instructions:
docker rm $(docker ps -a -q)
docker rmi $(docker images -q)
docker volume rm $(docker volume ls |awk '{print $2}')
rm -rf ~/Library/Containers/com.docker.docker/Data/*
You can try to prune all unused images/containers but this has proven not very effective:
docker system prune
Use a template image that is larger, install qemu using homebrew and move the image around, see this specific comment: https://github.com/docker/for-mac/issues/371#issuecomment-242047368 but you need to have at least 2x of space left to do this without losing containers/images.
See also: How do you get around the size limitation of Docker.qcow2 in the Docker for Mac?
And https://forums.docker.com/t/no-space-left-on-device-error/10894/26
I ran into the same issue, running docker system prune --volumes resolved the problem.
"Volumes are not pruned by default, and you must specify the --volumes flag for docker system prune to prune volumes."
See: https://docs.docker.com/config/pruning/#prune-everything
I ran into this recently on with a docker installation on linux that uses the devicemapper storage driver (default). There was indeed a docker configuration I needed to change to fix this.
Docker images are made of read-only layers of filesystem snapshots, each layer created by a command in your Dockerfile, which are built on top of a common base storage snapshot. The base snapshot is shared by all your images and has a file system with a default size of 10GB. When you run your image you get a new writable layer on top of all the layers in the image, so you can add new files in your running container but it's still eventually based on the same base snapshot with the 10GB filesystem. This is at least true for devicemapper, not sure about other drivers. Here is the relevant documentation from docker.
To change this default value to something else, there's a daemon parameter you can set, e.g. docker daemon --storage-opt dm.basesize=100G. Since you probably don't run the daemon manually need to edit the docker daemon options in some file, depending on how you run the docker daemon. With docker for mac you can edit the daemon parameters as JSON in the preferences under Daemon->Advanced. You probably need to add something like this:
{
"storage-opts": ["dm.basesize=100G"]
}
(but like I said, I had this problem on linux, so didn't try the above).
Anyway in order for this to take effect, you'll need to remove all your existing images (so that they're re-created on top of the new base snapshot with the new size). See storage driver options.
I have installed openstack cinder module. Everything works great.
but i want to mount a volume in order to see its contents.
I typed lvscan command on cinder storage computer. I can see my volume list.
ACTIVE '/dev/cinder-volumes/volume-XXXXX-XXXX-XXXX-XXX-XXXX' [1.00 GiB] inherit
but when i type
mount /dev/cinder-volumes/volume-XXXXX-XXXX-XXXX-XXX-XXXX /mnt
i get this error message
mount: wrong fs type, bad option, bad superblock on /dev/mapper/cinder--volumes-volume--XXXXXXXXXXXXXXXX,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
Is it because volume is already mounted by openstack instance ?
What should i do for mounting or reading/writing data in this volume ?
Thanks
We can have a data volume in docker:
$ docker run -v /path/to/data/in/container --name test_container debian
$ docker inspect test_container
...
Mounts": [
{
"Name": "fac362...80535",
"Source": "/var/lib/docker/volumes/fac362...80535/_data",
"Destination": "/path/to/data/in/container",
"Driver": "local",
"Mode": "",
"RW": true
}
]
...
But if the data volume lives in /var/lib/docker/volumes/fac362...80535/_data, is it any different from having the data in a folder mounted using -v /path/to/data/in/container:/home/user/a_good_place_to_have_data?
Although using volumes and bind mounts feels the same (with the only change being the location of the directory), there are differences in behavior.
Volumes vs Bind Mounts
With Bind Mount, a file or directory on the host machine is mounted into a container. The file or directory is referenced by its full or relative path on the host machine.
With Volume, a new directory is created within Docker's storage directory on the host machine, and Docker manages that directory's content.
Volumes advantages over bind mounts:
Volumes are easier to back up or migrate than bind mounts.
You can manage volumes using Docker CLI commands or the Docker API.
Volumes work on both Linux and Windows containers.
Volumes can be more safely shared among multiple containers.
Volume drivers allow you to store volumes on remote hosts or cloud providers, to encrypt the contents of volumes, or to add other functionality.
A new volume’s contents can be pre-populated by a container.
EDIT (9.9.2019):
According to #Sebi2020 comment, Bind mounts are much easier to backup. Docker doesn't provide any command to backup volumes. You have to use temporary containers with a bind mount to create backups.
Volumes
Created and managed by Docker. You can create a volume explicitly
using the docker volume create command, or Docker can create a volume
during container or service creation.
When you create a volume, it is stored within a directory on the
Docker host. When you mount the volume into a container, this
directory is what is mounted into the container. This is similar to
the way that bind mounts work, except that volumes are managed by
Docker and are isolated from the core functionality of the host
machine.
A given volume can be mounted into multiple containers simultaneously.
When no running container is using a volume, the volume is still
available to Docker and is not removed automatically. You can remove
unused volumes using docker volume prune.
When you mount a volume, it may be named or anonymous. Anonymous
volumes are not given an explicit name when they are first mounted
into a container, so Docker gives them a random name that is
guaranteed to be unique within a given Docker host. Besides the name,
named and anonymous volumes behave in the same ways.
Volumes also support the use of volume drivers, which allow you to
store your data on remote hosts or cloud providers, among other
possibilities.
Bind mounts
Available since the early days of Docker. Bind mounts have limited
functionality compared to volumes. When you use a bind mount, a file
or directory on the host machine is mounted into a container. The file
or directory is referenced by its full path on the host machine. The
file or directory does not need to exist on the Docker host already.
It is created on demand if it does not yet exist. Bind mounts are very
performant, but they rely on the host machine’s filesystem having a
specific directory structure available. If you are developing new
Docker applications, consider using named volumes instead. You can’t
use Docker CLI commands to directly manage bind mounts.
There is also tmpfs mounts.
tmpfs mounts
A tmpfs mount is not persisted on disk, either on the Docker host or
within a container. It can be used by a container during the lifetime
of the container, to store non-persistent state or sensitive
information. For instance, internally, swarm services use tmpfs mounts
to mount secrets into a service’s containers.
Reference:
https://docs.docker.com/storage/
is it any different from having the data in a folder mounted using -v /path/to/data/in/container:/home/user/a_good_place_to_have_data?
It is because, as mentioned in "Mount a host directory as a data volume"
The host directory is, by its nature, host-dependent. For this reason, you can’t mount a host directory from Dockerfile because built images should be portable. A host directory wouldn’t be available on all potential hosts.
If you have some persistent data that you want to share between containers, or want to use from non-persistent containers, it’s best to create a named Data Volume Container, and then to mount the data from it.
You can combine both approaches:
docker run --volumes-from dbdata -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /dbdata
Here we’ve launched a new container and mounted the volume from the dbdata container.
We’ve then mounted a local host directory as /backup.
Finally, we’ve passed a command that uses tar to backup the contents of the dbdata volume to a backup.tar file inside our /backup directory. When the command completes and the container stops we’ll be left with a backup of our dbdata volume.
Yes, this is quite different from a few perspectives. Like you wrote in the question's title, it is about understanding why we need data volumes vs bind mount to host.
Part 1 - Basic scenarios with examples
Lets take 2 scenarios.
Case 1: Web server
We want to provide our web server a configuration file that might change frequently. For example: exposing ports according to the current environment.
We can rebuild the image each time with the relevant setup or create 2 different images for each environment. Both of this solutions aren’t very efficient.
With Bind mounts Docker mounts the given source directory into a location inside the container.
(The original directory / file in the read-only layer inside the union file system will simply be overridden).
For example - binding a dynamic port to nginx:
version: "3.7"
services:
web:
image: nginx:alpine
volumes:
- type: bind #<-----Notice the type
source: ./mysite.template
target: /etc/nginx/conf.d/mysite.template
ports:
- "9090:8080"
environment:
- PORT=8080
command: /bin/sh -c "envsubst < /etc/nginx/conf.d/mysite.template >
/etc/nginx/conf.d/default.conf && exec nginx -g 'daemon off;'"
(*) Notice that this example could also be solved using Volumes.
Case 2 : Databases
Docker containers do not store persistent data -- any data that will be written to the writable layer in container’s union file system will be lost once the container stops running.
But what if we have a database running on a container, and the container stops - that means that all the data will be lost?
Volumes to the rescue.
Those are named file system trees which are managed for us by Docker.
For example - persisting Postgres SQL data:
services:
db:
image: postgres:latest
volumes:
- "dbdata:/var/lib/postgresql/data"
volumes:
- type: volume #<-----Notice the type
source: dbdata
target: /var/lib/postgresql/data
volumes:
dbdata:
Notice that in this case, for named volumes, the source is the name of the volume
(for anonymous volumes, this field is omitted).
Part 2 - Comparison
Differences in management and isolation on the host
Bind mounts exist on the host file system and being managed by the host maintainer. Applications / processes outside of Docker can also modify it.
Volumes can also be implemented on the host, but Docker will manage them for us and they can not be accessed outside of Docker.
Volumes are a much wider solution
Although both solutions help us to separate the data lifecycle from containers,
by using Volumes you gain much more power and flexibility over your system.
With Volumes we can design our data effectively and decouple it from other parts of the system by storing it in dedicated remote locations (e.g., in the cloud) and integrate it with external services like backups, monitoring, encryption and hardware management.
The difference between host directory and a data volume is in that that Docker manages the latter by placing it into the $DOCKER-DATA-DIR/volumes directory and attaching a reference to it (names or randomly generated ids). That is you get a little bit of convenience.
Both host directories and data volumes are directories on the host. Both are host dependent. You can't reference either of them in a Dockerfile; the VOLUME directive creates a new nameless (with randomly generated id) volume every time you launch a new container and cannot reference an existing volume.
* $DOCKER-DATA-DIR is /var/lib/docker here unless you changed the defaults.
I'm working on a NUMA server which has two memory nodes.
I want to create a file system which will be loaded in main memory like tmpfs or ramfs and I want to bind it to a specific memory node. In other words I don't want the ramfs contents to be interleaved across the two memory nodes.
So how can I achieve this?
I have tried the numactl command with the --file option but it seems to work only for single files (I need to load a directory).
thanks
I found out that the mpol option of the mount command does what I want.
For example the command:
mount -t tmpfs -o size=4g,mpol=bind:0 tmpfs pathToTheDir
will create a 4GB filesystem which will be allocated on memory node 0
I want to list all container directories that are mounted volumes.
I.e. to be able to get similar info I get from
docker inspect --format "{{ .Volumes }}" <self>
But from within the container and without having docker installed in there.
I tried cat /proc/mounts, but I couldn't find a proper filter for it.
(EDIT - this may no longer work on Mac) If your Docker host is OS X, the mounted volumes will be type osxfs (or fuse.osxfs). You can run a
mount | grep osxfs | awk '{print $3}'
and get a list of all the mounted volumes.
If your Docker host is Linux (at least Ubuntu 14+, maybe others), the volumes appear to all be on /dev, but not on a device that is in your container's /dev filesystem. The volumes will be alongside /etc/resolv.conf, /etc/hostname, and /etc/hosts. If you do a mount | grep ^/dev to start, then filter out any of the files in ls /dev/*, then filter out the three files listed above, you should be left with host volumes.
mount | grep ^/dev/ | grep -v /etc | awk '{print $3}'
My guess is the specifics may vary from Linux to Linux. Not ideal, but at least possible to figure out.
Assuming you want to check what volumes are mounted from inside a linux based container you can look up entries beginning with "/dev" in /etc/mtab, removing the /etc entries
$ grep "^/dev" /etc/mtab | grep -v " \/etc/"
/dev/nvme0n1p1 /var/www/site1 ext4 rw,relatime,discard,data=ordered 0 0
/dev/nvme0n1p1 /var/www/site2 ext4 rw,relatime,discard,data=ordered 0 0
As you can read from many of the comments you had, a container is initially nothing but a restricted, reserved part of resources that is totally cut away from the rest of your machine. It is not aware of being a Docker, and inside the container everything behaves as if it were a separate machine. Sort of like the matrix, I guess ;)
You get access to the host machine's kernel and its resources, but yet again restricted as just a filtered out set. This is done with the awesome "cgroups" functionality that comes with Unix/Linux kernels.
Now the good news: There are multiple ways for you to provide the information to your Docker, but that is something that you are going to have to provide and build yourself.
The easiest ad most powerful way is to mount the Unix socket located on your host at /var/run/docker.sock to the inside of your container at the same location. That way, when you use the Docker client inside your container you are directly talking to the docker engine on your host.
However, with great power comes great responsibility. This is a nice setup, but it is not very secure. Once someone manages to get into your docker it has root access to your host system this way.
A better way would be to provide a list of mounts through the environment settings, or clinging on to some made-up conventions to be able to predict the mounts.
(Do you realize that there is a parameter for mounting, to give mounts an alias for inside your Docker?)
The docker exec command is probably what you are looking for.
This will let you run arbitrary commands inside an existing container.
For example:
docker exec -it <mycontainer> bash
Of course, whatever command you are running must exist in the container filesystem.
#docker cp >>>> Copy files/folders between a container and the local filesystem
docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH
docker cp [OPTIONS] SRC_PATH CONTAINER:DEST_PATH
to copy full folder:
docker cp ./src/build b081dbbb679b:/usr/share/nginx/html
Note – This will copy build directory in container’s …/nginx/html/ directory to copy only files present in folder:
docker cp ./src/build/ b081dbbb679b:/usr/share/nginx/html
Note – This will copy contents of build directory in container’s …./nginx/html/ directory
Docker Storage options:
Volumes are stored in a part of the host filesystem which is managed by Docker(/var/lib/docker/volumes/ on Linux). Non-Docker processes should not modify this part of the filesystem. Volumes are the best way to persist data in Docker.
When you create a volume, it is stored within a directory on the Docker host. When you mount the volume into a container, this directory is what is mounted into the container. This is similar to the way that bind mounts work, except that volumes are managed by Docker and are isolated from the core functionality of the host machine.
A given volume can be mounted into multiple containers simultaneously. When no running container is using a volume, the volume is still available to Docker and is not removed automatically. You can remove unused volumes using docker volume prune.
When you mount a volume, it may be named or anonymous. Anonymous volumes are not given an explicit name when they are first mounted into a container, so Docker gives them a random name that is guaranteed to be unique within a given Docker host. Besides the name, named and anonymous volumes behave in the same ways.
Volumes also support the use of volume drivers, which allow you to store your data on remote hosts or cloud providers, among other possibilities.
Bind mounts may be stored anywhere on the host system. They may even be important system files or directories. Non-Docker processes on the Docker host or a Docker container can modify them at any time.
Available since the early days of Docker. Bind mounts have limited functionality compared to volumes. When you use a bind mount, a file or directory on the host machine is mounted into a container. The file or directory is referenced by its full path on the host machine. The file or directory does not need to exist on the Docker host already. It is created on demand if it does not yet exist. Bind mounts are very performant, but they rely on the host machine’s filesystem having a specific directory structure available. If you are developing new Docker applications, consider using named volumes instead. You can’t use Docker CLI commands to directly manage bind mounts.
One side effect of using bind mounts, for better or for worse, is that you can change the host filesystem via processes running in a container, including creating, modifying, or deleting important system files or directories. This is a powerful ability which can have security implications, including impacting non-Docker processes on the host system.
tmpfs mounts are stored in the host system’s memory only, and are never written to the host system’s filesystem.
A tmpfs mount is not persisted on disk, either on the Docker host or within a container. It can be used by a container during the lifetime of the container, to store non-persistent state or sensitive information. For instance, internally, swarm services use tmpfs mounts to mount secrets into a service’s containers.
If you need to specify volume driver options, you must use --mount.
-v or --volume: Consists of three fields, separated by colon characters (:). The fields must be in the correct order, and the meaning of each field is not immediately obvious.
o In the case of named volumes, the first field is the name of the volume, and is unique on a given host machine. For anonymous volumes, the first field is omitted.
o The second field is the path where the file or directory will be mounted in the container.
o The third field is optional, and is a comma-separated list of options, such as ro. These options are discussed below.
• --mount: Consists of multiple key-value pairs, separated by commas and each consisting of a = tuple. The --mount syntax is more verbose than -v or --volume, but the order of the keys is not significant, and the value of the flag is easier to understand.
o The type of the mount, which can be bind, volume, or tmpfs. This topic discusses volumes, so the type will always be volume.
o The source of the mount. For named volumes, this is the name of the volume. For anonymous volumes, this field is omitted. May be specified as source or src.
o The destination takes as its value the path where the file or directory will be mounted in the container. May be specified as destination, dst, or target.
o The readonly option, if present, causes the bind mount to be mounted into the container as read-only.
o The volume-opt option, which can be specified more than once, takes a key-value pair consisting of the option name and its value.