What is the difference between volumes-from and volumes? - docker

I saw the docker-compose patterns but I'm confused. What is the best way to make composed containers.
When should I use link, or volumes_from.
When should I use volumes_from, volumes
#1 app-db-data
app:
image: someimage
link:
- db // data volume container name
db:
image: mysql
volumes_from:
- data // data volume name
data:
image: someimage
volumes:
- {host data}:{guest data}
#2 app-db+data
app:
image: someimage
link:
- db // data volume container name
db:
image: mysql
volumes:
- data // data file name
app
#1 app-service-data
app:
image: someimage
volumes_from:
- service // service container name
service:
image: mysql
volumes_from:
- data // image container name
data:
image: someimage
volumes:
- {host data}:{guest data}
#2 app-service+data
app:
image: someimage
volumes_from:
- service // service container name
service:
image: mysql
volumes:
- data // mounted file
Thanks

In short:
volumes_from mounts from other containers.
volumes mounts defined inline.
links connects containers.
A little bit more explained:
volumes_from mounts volumes from other containers. For example if you have data only containers and you want to mount these data only containers in the container that has your application code.
volumes is a the inline way to define and mount volumes. If you read #17798 you can see that named volumes can replace data only containers in most cases.
The simplest is then to use volumes. Since you can reuse them by naming them.
links is different. Because it does not mount. Instead it connects containers. So if you do:
app:
container_name: app_container
links:
- db
That means that if you connect to app_container with docker exec -it app_container bash and try ping db you will see that container is able to resolve ip for db.
This is because docker creates a network between containers.

Link and volumes_from are different concepts. Links are used when you need to connect (by network) two containers. In this case if you want to connect an App to the Database, the way to do this is by using a link, since applications use a port and host to connect to a database (not a directory on the filesystem).
Volumes and volumes_from differ in that the first one only declares volumes that docker will make persistent or host:guest mounts, but volumes_from tells docker to use a volumes that is already declared on another host (making it available to this host).
Of those 4 cases that you present, I think that the first and second are good choices. In the first you are creating a data only container, and make the mysql container use it. In the second case the data and the mysql container are the same.
Links and volumes are perfectly explained in the docker documentation.
Hope it helps.

Addition: Volumes_from is used when you want to mount all anon-volumes of a container - named volumes could have been mounted directly since the early days.
AFAICs https://docs.docker.com/compose/compose-file/#volumes . docker-compose has removed this functionality entirely, not sure how and why and if there is an alternative. But assume, you have an app container and you have a httpd container. Usually you would define the codebase folder, /var/www, as an anon volume and then mount it in httpd to be to serve static files using the httpd service, while passing all dynamic files like ruby/php/java to an upstream backend on app.
The point in using a anon volume and not a named volume is, that actually you want to be able to redeploy app and change the codebase ( app update ) which would not work, if app would have a named volume. That said, anon volumes are doing exactly that and thats why volumes_from is used here - using named volumes is no option is this case ( as it is very practical in a a lot of other cases ).
For the reference the upgrade guides for volumes_from:
https://docs.docker.com/compose/compose-file/compose-versioning/#upgrading
So volumes_from usually is used in a different context / scenario and named-volumes are the standard in a ll other cases as explained above. A brief post about that is https://stackoverflow.com/a/44744861/3625317

Related

Re-using existing volume with docker compose

I have setup two standalone docker containers, one runs a webserver another one runs a mysql for it.
Right now I was attempting to have it working with docker-compose. All is nice and it runs well, but I was wondering how could I re-use existing volumes from the existing standalone containers that I have previously created (since I want to retain the data from them).
I saw people suggesting to use external: true command for this, but could not get the right syntax so far.
Is external: true the correct way approach for this, or should I approach this differently?
Or can I just specify the path to the volume within docker-compose.yml and make it use the old existing volume?
Yes you can do it normally, just an example below:
Set external to true and set name to the name of the volume you want to mount.
version: "3.5"
services:
transmission:
image: linuxserver/transmission
container_name: transmission
volumes:
- transmission-config:/config
- /path/to/downloads:/downloads
ports:
- 51413:51413
- 51413:51413/udp
networks:
- rede
restart: always
networks:
rede:
external: true
name: rede
volumes:
transmission-config:
external: true
name: transmission-config
Per the documentation, using the external flag allows you to use volumes created outside the scope of the docker-compose file.
However, it is advisable to create a fresh volume via the docker-compose file and copy the existing data from the old volumes to the new volumes
You can create a volume explicitly using the docker volume create command, or Docker can create a volume during container or service creation. When you create a volume, it is stored within a directory on the Docker host. When you mount the volume into a container, this directory is what is mounted into the container.
If your system is running, you can exec into the mysql container, copy and move it outside.
docker cp "${container_id}":/path_to_folder /path_to_server

docker volume type - bind vs volume

TLDR
In docker-compose, what's the difference between
volumes:
- type: volume
source: mydata
target: /data
and
volumes:
- type: bind
source: mydata
target: /data
?
The question in long:
When you specify the volumes option in your docker-compose file, you can use the long-syntax style
According to the docs, the type option accepts 3 different values: volume, bind and tmpfs:
I understand the tmpfs option - it means that the volume will not be saved after the container is down..
But I fail to find any reference in the docs about the difference between the other 2 options: bind and volume, could someone enlighten me about that?
When bind mounts are files coming from your host machine, volumes are something more like the nas of docker.
Bind mounts are files mounted from your host machine (the one that runs your docker daemon) onto your container.
Volumes are like storage spaces totally managed by docker.
You will find, in the literature, two types of volumes:
named volumes (you provide the name of it)
anonymous volumes (usual UUID names from docker, like you can find them on container or untagged images)
Those volumes come with their own set of docker commands; you can also consult this list via
docker volume --help
You can see your existing volumes via
docker volume ls
You can create a named volume via
docker volume create my_named_volume
But you can also create a volume via a docker-compose file
version: "3.3"
services:
mysql:
image: mysql
volumes:
- type: volume
source: db-data
target: /var/lib/mysql/data
volumes:
db-data:
Where this is the part saying please docker, mount me the volume named db-data on top of the container directory /var/lib/mysql/data
- type: volume
source: db-data
target: /var/lib/mysql/data
And this is the part saying to docker please create me a volume named db-data
volumes:
db-data:
Docker documentation about the three mount types:
https://docs.docker.com/storage/bind-mounts/
https://docs.docker.com/storage/volumes/
https://docs.docker.com/storage/tmpfs/
If I understood you correctly, you're asking in other words: What is the difference between Volumes and bind mounts?
Differences in management and isolation on the host
Bind mounts exist on the host file system and being managed by the host maintainer. Applications / processes outside of Docker can also modify it.
Volumes can also be implemented on the host, but Docker will manage them for us and they can not be accessed outside of Docker.
Volumes are a much wider solution
Although both solutions help us to separate the data lifecycle from containers,
by using Volumes you gain much more power and flexibility over your system.
With Volumes we can design our data effectively and decouple it from the host and other parts of the system by storing it dedicated remote locations (Cloud for example) and integrate it with external services like backups, monitoring, encryption and hardware management.
More Volumes advantages over bind mounts:
No host concerns.
Can be managed using Docker CLI.
Volumes can save you some uid/gid issues related permissions which occur in cases like when a container user's uid does not match the host gid.
A new volume’s contents can be pre-populated by a container.
Examples
Lets take 2 scenarios.
Case 1: Web server.
We want to provide our web server a configuration file that might change frequently. For example: exposing ports according to the current environment.
We can rebuild the image each time with the relevant setup or create 2 different images for each environment. Both of this solutions aren’t very efficient.
With Bind mounts Docker mounts the given source directory into a location inside the container.
(The original directory / file in the read-only layer inside the union file system will simply be overridden).
For example - binding a dynamic port to nginx:
version: "3.7"
services:
web:
image: nginx:alpine
volumes:
- type: bind #<-----Notice the type
source: ./mysite.template
target: /etc/nginx/conf.d/mysite.template
ports:
- "9090:8080"
environment:
- PORT=8080
command: /bin/sh -c "envsubst < /etc/nginx/conf.d/mysite.template >
/etc/nginx/conf.d/default.conf && exec nginx -g 'daemon off;'"
(*) Notice that this example could also be solved using Volumes.
Case 2 : Databases.
Docker containers do not store persistent data: any data that will be written to the writable layer in container’s union file system will be lost once the container stop running.
But what if we have a database running on a container, and the container stops - that means that all the data will be lost?
Volumes to the rescue.
Those are named file system trees which are managed for us by Docker.
For example - persisting Postgres SQL data:
services:
db:
image: postgres:latest
volumes:
- "dbdata:/var/lib/postgresql/data"
volumes:
- type: volume #<-----Notice the type
source: dbdata
target: /var/lib/postgresql/data
volumes:
dbdata:
Notice that in this case, for named volumes, the source is the name of the volume
(For anonymous volumes, this field is omitted).
Feature
Bind
Volume                                 
Internal soul
Bind mounts attach a user-specified location on host filesystem to a specific point in a container file tree.
Volume attach with disk storage on the host filesystem or cloud storage.
command
--mount type=bind,src="",dst=""
Docker CLI docker volume command
Dependency
dependent on location on to the host filesystem.
Container-independent data management
Separation of concerns
No
Yes
Conflict with other containers
Yes Example: multiple instances of Cassandra that all use the same host location as a bind mount for data storage. In that case, each of the instances would compete for the same set of files. Without other tools such as file locks, that would likely result in corruption of the database.
No. By default, Docker creates volumes by using the local volume plugin.
When to choose
1- Bind mounts are useful when the host provides a file or directory that is needed by a program running in a container, or when that containerized program produces a file or log that is processed by users or programs running outside containers. 2- appropriate tools for workstations, machines with specialized concerns 3- systems with more traditional configuration management tooling.
Working with Persistent storage 1. Databases 2. Cloud storage
When not to choose
Better to avoid these kinds of specific bindings in generalized platforms or hardware pools.
To be written

Understanding volumn in docker compose

The following is an example given in https://docker-curriculum.com/
version: "3"
services:
es:
image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
container_name: es
environment:
- discovery.type=single-node
ports:
- 9200:9200
volumes:
- esdata1:/usr/share/elasticsearch/data
web:
image: prakhar1989/foodtrucks-web
command: python app.py
depends_on:
- es
ports:
- 5000:5000
volumes:
- ./flask-app:/opt/flask-app
volumes:
esdata1:
driver: local
and it says The volumes parameter specifies a mount point in our web container where the code will reside about the /opt/flask-app
I think it means, /opt/flask-app is a mount point and it points to the host machines ./flask-app
However it doesn't say anything about esdata1 and I can't apply the same explanation as given to /opt/flask-app since there's no esdata1 directory/file in the host machine.
What is happening for the esdata1 ?
My guess is that it means creating a volume (The closest thing I can think of is a disk partition) and name it esdata1 and mount it on /usr/share/elasticsearch/data, am I correct on this guess?
These are a bit different things: volumes and bind mounts. Bind mounts let you specify folder on host machine, which would serve as a storage. Volumes (at lease for local driver) also have folders on host machines, but their location is managed by Docker and is sometimes a bit more difficult to find.
When you specify volume in docker-compose.yml, if your path starts with / or . it becomes a bind mount, like in web service. Otherwise, if it is a single verb, it is a volume, like for es service.
You can inspect all volumes on your host machine by running docker volume ls.
What is happening for the esdata1 ? My guess is that it means creating
a volume (The closest thing I can think of is a disk partition) and
name it esdata1 and mount it on /usr/share/elasticsearch/data, am I
correct on this guess?
That's all correct.
I do not pretend on setting up the rules, but in general, volumes are more suitable for sharing common data between several containers, like in docker-compose, while bind mounts suite better for sharing data from host to container, like some initial configs for services.

Moving volume between containers docker-composer

I have image A (some_laravel_project) and B (laravel_module). Image A is a Laravel project that looks like this.
app
modules
core
Volume Image b here
config
As the list above suggests I want to share a volume from Image B in Image A using docker-compose. I want to access the files in container B.
This is the docker-compose I tried and didn't receive any errors creating those images in gitlab ci. I checked and the volume and its files are in stored in the module_user:latest container.
I think I made a mistake mounting the volume to some_laravel_project.
version: '3'
services:
laravel:
image: some_laravel_project
working_dir: /var/www
volumes:
- /var/www/storage
- userdata:/var/www/Modules
user:
image: laravel_module
volumes:
- userdata:/user
volumes:
userdata:
webroot:
The method you used to share volumes across container in docker compose is the correct one. You can find this documented under docker-compose volumes
if you want to reuse a volume across multiple services, then define a
named volume in the top-level volumes key. Use named volumes with
services,
In you case, the directory /var/www/Modules in laravel will have the same content as that in /user inside user service. You can verify that by going into the containers and checking each directoty by running;
docker exec -it <container-name> bash

How do named volumes work in docker?

I'm struggling to understand how exactly does the named volume work in the following example from docker docs:
version: "3"
services:
db:
image: db
volumes:
#1
- data-volume:/var/lib/db
backup:
image: backup-service
volumes:
#2
- data-volume:/var/lib/backup/data
volumes:
data-volume:
My guess is, that the first occurrence of the named volume (#1) defines what is contained inside the volume, while subsequent occurrences (#2) simply share the volume's content with whatever containers they are referenced from.
Is this guess correct?
Listing data-volume: under the top-level volumes: key creates a named volume on the host if it doesn't exist yet. This behaves the following way according to this source
If you create a named volume by running a new container from image by docker run -v my-precious-data:/data imageName, the data within the image/container under /data will be copied into the named volume.
If you create another container binds to an existing named volume, no files from the new image/container will be copied/overwritten, it will use the existing data inside the named volume.
They don’t have a docker command to backup / export a named volume. However you can find out the actual location of the file by “docker volume inspect [volume-name]”.
In case the volume is empty and both containers have data in the target directory the first container to be run will mount its data into the volume and the other container will see that data (and not its own). I don't know which container will run first (although I expect it executes from top to bottom) however you can force an order with depends_on as shown here
------------------- Update
The depends_on option is ignored when deploying a stack in swarm mode with a version 3 Compose file.
The way that I understand your guess, you are not completely correct.
Declaring and referencing a named volume in a docker-compose file will create an empty volume which may then be accessed and shared by the services saying so in their volumes section.
If you want to share a named volume, you have to declare this volume in the top-level volume section of your docker-compose file. Example (as in the docker docs already linked by yourself):
version: "3"
services:
db:
image: db
volumes:
#1 uses the named and shared volume 'data-volume' created with #3
- data-volume:/var/lib/db
backup:
image: backup-service
volumes:
#2 uses the named and shared volume 'data-volume' created with #3
- data-volume:/var/lib/backup/data
volumes:
#3 creates the named volume 'data-volume'
data-volume:
The volume will be empty on start (and therefore the folders in the containers where that volume is mounted to). Its content will be a result of the services acions on runtime.
Hope that made it a bit more clear.

Resources