I'm struggling to understand how exactly does the named volume work in the following example from docker docs:
version: "3"
services:
db:
image: db
volumes:
#1
- data-volume:/var/lib/db
backup:
image: backup-service
volumes:
#2
- data-volume:/var/lib/backup/data
volumes:
data-volume:
My guess is, that the first occurrence of the named volume (#1) defines what is contained inside the volume, while subsequent occurrences (#2) simply share the volume's content with whatever containers they are referenced from.
Is this guess correct?
Listing data-volume: under the top-level volumes: key creates a named volume on the host if it doesn't exist yet. This behaves the following way according to this source
If you create a named volume by running a new container from image by docker run -v my-precious-data:/data imageName, the data within the image/container under /data will be copied into the named volume.
If you create another container binds to an existing named volume, no files from the new image/container will be copied/overwritten, it will use the existing data inside the named volume.
They don’t have a docker command to backup / export a named volume. However you can find out the actual location of the file by “docker volume inspect [volume-name]”.
In case the volume is empty and both containers have data in the target directory the first container to be run will mount its data into the volume and the other container will see that data (and not its own). I don't know which container will run first (although I expect it executes from top to bottom) however you can force an order with depends_on as shown here
------------------- Update
The depends_on option is ignored when deploying a stack in swarm mode with a version 3 Compose file.
The way that I understand your guess, you are not completely correct.
Declaring and referencing a named volume in a docker-compose file will create an empty volume which may then be accessed and shared by the services saying so in their volumes section.
If you want to share a named volume, you have to declare this volume in the top-level volume section of your docker-compose file. Example (as in the docker docs already linked by yourself):
version: "3"
services:
db:
image: db
volumes:
#1 uses the named and shared volume 'data-volume' created with #3
- data-volume:/var/lib/db
backup:
image: backup-service
volumes:
#2 uses the named and shared volume 'data-volume' created with #3
- data-volume:/var/lib/backup/data
volumes:
#3 creates the named volume 'data-volume'
data-volume:
The volume will be empty on start (and therefore the folders in the containers where that volume is mounted to). Its content will be a result of the services acions on runtime.
Hope that made it a bit more clear.
Related
I am writing a docker-compose.yaml file for my project. I have checked the volumes documentation here .
I also understand the concept of volume in docker that I can mount a volume e.g. -v my-data/:/var/lib/db where my-data/ is a directory on my host machine while /var/lib/db is the path inside database container.
My confuse is with the link I put above. There it has the following sample:
version: "3.9"
services:
db:
image: db
volumes:
- data-volume:/var/lib/db
backup:
image: backup-service
volumes:
- data-volume:/var/lib/backup/data
volumes:
data-volume:
I wonder does it mean that I have to create a directory named data-volume on my host machine? What if I have a directory on my machine with path temp/my-data/ and I want to mount that path to the database container /var/lib/db ? Should I do something like below?
version: "3.9"
services:
db:
image: db
volumes:
- temp/my-data/:/var/lib/db
volumes:
temp/my-data/:
My main confusion is the volumes: section at the bottom, I am not sure whether the volume name should be the path of my directory or should be just literally a name I give & if it is the latter case then how could the given name be mapped with temp/my-data/ on my machine? The sample doesn't indicate that & is ambiguous to clarify that.
Could someone please clarify it for me?
P.S. I tried with above docker-compose I guessed, ended up with the error:
ERROR: The Compose file './docker-compose.yaml' is invalid because:
volumes value 'temp/my-data/' does not match any of the regexes: '^[a-zA-Z0-9._-]+$'
Mapped volumes can either be files/directories on the host machine (sometimes called bind mounts in the documentation) or they can be docker volumes that can be managed using docker volume commands.
The volumes: section in a docker-compose file specify docker volumes, i.e. not files/directories. The first docker-compose in your post uses such a volume.
If you want to map a file or directory (like in your last docker-compose file), you don't need to specify anything in the volumes: section.
Docker volumes (the ones specified in the volumes: section or created using docker volume create) are of course also stored somewhere on your host computer, but docker manages that and you shouldn't normally need to know where or what the format is.
This part of the documentation is pretty good about explaining it, I think https://docs.docker.com/storage/volumes/
As #HansKilian mentions, you don't need both volumes and services.volumes. To use services.volumes, map the host directory to the container directory like this:
services:
db:
image: db
volumes:
- /host/path/lib/db:/container/path/lib/db
With that, the directory /host/path/lib/db on the host machine will be used by the container and available at /container/path/lib/db.
Now, if you're like me, I get really confused with fake examples, so let's say the real directory on your host machine is /var/lib/db and you just want to see it at /db when you run a shell in Docker (i.e., docker exec -it /bin/bash container-id).
docker-compose.yaml would look like this:
services:
db:
image: db
volumes:
- /var/lib/db:/db
Now when you run the shell, cd /logs and ls, you'll see the same results as if you'd cd /var/lib/db on the host.
If you want to use the volumes section to indicate a global volume to use, you first have to create that volume using docker volume create. The documentation Hans linked includes steps to do this. The syntax of /host/path:/container/path is replaced by volume-name:/container/path. Then, once defined, you'd alter your docker-compose.yaml to be more like this:
services:
db:
image: db
volumes:
- your-global-volume-name:/db
volumes:
your-global-volume-name:
external: true
Note that I have not tested or used the this configuration. I'm assuming it's correct based on the other method working and the few changes I can identify in the docs.
I am experimenting with docker on windows and creating a stack for the same.
I just found that when i use docker-compose up -d, docker volume are created with the name like foldername_volumename.
I have a working app for the stack under one folder and just want to change the folder name. But found that while I changed it, it prevent me to use the same volume that was previously used.
I have some configurations and data that I will lose if i will move to another volume name.
Is there any way to reuse the same volume but still able to change the folder name?
What is the best practice?
You can use external: true to let docker compose know that it does not need to create the volume, it already exists (and therefore, the folder name will not be prepended).
version: '3.2'
volumes:
mydata:
external: true
services:
test:
image: alpine
volumes:
- mydata:/data
External volumes documentation
The volume name is based on the project name. By default project name is based on the containing folder's name, but you can override it by doing docker-compose -p yourprojectname. So if you do that you can get consistent volume names regardless of containing folder name.
I have for example this service and volume defined in my docker-compose file
postgres:
image: postgres:9.4
volumes:
- db_data:/var/lib/postgresql/data
volumes:
blue_prod_db:
driver: rancher-nfs
Then. if you define a volume inside a Dockerfile like this:
RUN mkdir /stuff
COPY ./stuff/* /stuff/
VOLUME /stuff
How can you later access it through the docker-compose configuration and add it to a container?
When configured in the Dockerfile, a volume will result in any container started from that image, including temporary containers later in the build process from the RUN command, to have a volume defined at the specified location, e.g. /stuff. If you do not define a source for that volume at run time, you will get an anonymous volume created by docker for you at that location. However, you can always define a volume with a source at run time (even without the volume being defined) by specifying the location in your compose file:
version: "3"
services:
app:
image: your_image
volumes:
- data:/stuff
volumes:
data:
Note that there are two volumes sections, one for a specific service that specifies where the volume is mounted inside the container, and another at the top level where you can specify the source of the volume. Without specifying a source, you'll get a local volume driver with a directory under /var/lib/docker bind mounted into the container.
I do not recommend specifying volumes inside the Dockerfile in general, it breaks the ability to extend the image in later steps for child images, and clutters the filesystem with anonymous volumes that are not easy to track back to their origin. It's best to define them at runtime with something like a compose file.
I am new to docker, and stumbled upon a docker-compose file. I get the gist of all other properties but I have no idea what below line is doing:
volumes:
- ./data:/data/db
Can anyone please help me with this.
multiple volumes can be attached to your container ... each are defined as a pair
volumes:
- /parent/host/path01:/inside/container/path_one
- /parent/host/path02:/inside/container/path_another
of each pair the left side is a pre-existing volume reachable on host before container is created ... right side is what the freshly launched container views that left side as from inside the container
in your example, in same dir where you launch docker-compose from, there evidently exists a dir called data ... using ./data will reach it using a relative path ... the right side /data/db is what the code in your container calls that same dir
/full/path/to/reach/data:/data/db
is using the absolute path to reach that same ./data dir which lives on the parent host which docker-compose is executed on
This volume mapping allows permanent storage on parent host to become visible (read/writable) to the container ... since the container filesystem is ephemeral and so goes away when container exits this volume mapping gives the container access to permanent storage for specified paths which must appear in your yaml file ... especially important for database containers like mongo ... all files used in your container not mapped in the volumes yaml disappear once the container exists
Here is a typical yaml snippet for mongo where it gains access to permanent storage on parent host
loudmongo:
image: mongo
container_name: loud_mongo
restart: always
ports:
- 127.0.0.1:27017:27017
volumes:
- /cryptdata7/var/data/db:/data/db
The dash symbol is probably what is throwing you off, because it is poorly formatted YAML syntax for a YAML list element.
The volume syntax after the dash is just following the so-called "short" syntax for a host-to-container bind-mounted volume mapping.
I saw the docker-compose patterns but I'm confused. What is the best way to make composed containers.
When should I use link, or volumes_from.
When should I use volumes_from, volumes
#1 app-db-data
app:
image: someimage
link:
- db // data volume container name
db:
image: mysql
volumes_from:
- data // data volume name
data:
image: someimage
volumes:
- {host data}:{guest data}
#2 app-db+data
app:
image: someimage
link:
- db // data volume container name
db:
image: mysql
volumes:
- data // data file name
app
#1 app-service-data
app:
image: someimage
volumes_from:
- service // service container name
service:
image: mysql
volumes_from:
- data // image container name
data:
image: someimage
volumes:
- {host data}:{guest data}
#2 app-service+data
app:
image: someimage
volumes_from:
- service // service container name
service:
image: mysql
volumes:
- data // mounted file
Thanks
In short:
volumes_from mounts from other containers.
volumes mounts defined inline.
links connects containers.
A little bit more explained:
volumes_from mounts volumes from other containers. For example if you have data only containers and you want to mount these data only containers in the container that has your application code.
volumes is a the inline way to define and mount volumes. If you read #17798 you can see that named volumes can replace data only containers in most cases.
The simplest is then to use volumes. Since you can reuse them by naming them.
links is different. Because it does not mount. Instead it connects containers. So if you do:
app:
container_name: app_container
links:
- db
That means that if you connect to app_container with docker exec -it app_container bash and try ping db you will see that container is able to resolve ip for db.
This is because docker creates a network between containers.
Link and volumes_from are different concepts. Links are used when you need to connect (by network) two containers. In this case if you want to connect an App to the Database, the way to do this is by using a link, since applications use a port and host to connect to a database (not a directory on the filesystem).
Volumes and volumes_from differ in that the first one only declares volumes that docker will make persistent or host:guest mounts, but volumes_from tells docker to use a volumes that is already declared on another host (making it available to this host).
Of those 4 cases that you present, I think that the first and second are good choices. In the first you are creating a data only container, and make the mysql container use it. In the second case the data and the mysql container are the same.
Links and volumes are perfectly explained in the docker documentation.
Hope it helps.
Addition: Volumes_from is used when you want to mount all anon-volumes of a container - named volumes could have been mounted directly since the early days.
AFAICs https://docs.docker.com/compose/compose-file/#volumes . docker-compose has removed this functionality entirely, not sure how and why and if there is an alternative. But assume, you have an app container and you have a httpd container. Usually you would define the codebase folder, /var/www, as an anon volume and then mount it in httpd to be to serve static files using the httpd service, while passing all dynamic files like ruby/php/java to an upstream backend on app.
The point in using a anon volume and not a named volume is, that actually you want to be able to redeploy app and change the codebase ( app update ) which would not work, if app would have a named volume. That said, anon volumes are doing exactly that and thats why volumes_from is used here - using named volumes is no option is this case ( as it is very practical in a a lot of other cases ).
For the reference the upgrade guides for volumes_from:
https://docs.docker.com/compose/compose-file/compose-versioning/#upgrading
So volumes_from usually is used in a different context / scenario and named-volumes are the standard in a ll other cases as explained above. A brief post about that is https://stackoverflow.com/a/44744861/3625317