I have compose file with two dependent services and all works fine. But one has mount volume as
volumes:
- "/web/logs:/usr/src/app/log"
and
deploy:
mode: replicated
replicas: 8 # may be different
Thus all replicas use one directory and file name. It is possible to write into different directory or files?
you can add an extra folder under /log,use some variable for the folder name eg: ip address so it will be unique and can do it automatic.
Related
I am experimenting with docker on windows and creating a stack for the same.
I just found that when i use docker-compose up -d, docker volume are created with the name like foldername_volumename.
I have a working app for the stack under one folder and just want to change the folder name. But found that while I changed it, it prevent me to use the same volume that was previously used.
I have some configurations and data that I will lose if i will move to another volume name.
Is there any way to reuse the same volume but still able to change the folder name?
What is the best practice?
You can use external: true to let docker compose know that it does not need to create the volume, it already exists (and therefore, the folder name will not be prepended).
version: '3.2'
volumes:
mydata:
external: true
services:
test:
image: alpine
volumes:
- mydata:/data
External volumes documentation
The volume name is based on the project name. By default project name is based on the containing folder's name, but you can override it by doing docker-compose -p yourprojectname. So if you do that you can get consistent volume names regardless of containing folder name.
I am new to docker and volumes and is confused about how named volumes are working. I have two scenarios in which I want to know how the named volumes will work
First Scenario
I have to setup two projects with docker and both have separate databases. Now how the database volumes will be mapped with /var/lib/mysql? Does it maintain separate data based on db name?
Second Scenario
I have two services using same named volume. In both the services, the path of container mapped to the named volume is different. How this will work?
services:
s1:
volume:
- vol:/var/lib/s1
s2:
volume:
- vol:/var/lib/s2
volumes:
vol:
Since you are using docker-compose, it does some things for you. If your composed "project name" is project_a, the docker-compose vol volume will be named project_a_vol. Verify this by running docker volume ls. By a "composed project name" I mean the name of the project which usually equals to the name of the directory in which the docker-compose was run, or custom one if the --project-name parameter was set (eg. docker-compose --project-name xxx up)
I assume you're using the default docker volume filesystem storage driver. A named volume is nothing more than a directory inside the /var/lib/docker/volumes folder (try it sudo ls -l /var/lib/docker/volumes). By mounting a volume using vol:/var/lib/s1 you tell docker to synchronize directories:
Local /var/lib/docker/volumes/project_a_vol with container directory /var/lib/s1.
If you compose your services this way:
services:
s1:
volumes:
- vol:/var/lib/s1
s2:
volumes:
- vol:/var/lib/s2
The same directory will be mounted to 2 services: s1 and s2 and you most probably will have a problem because 2 services will try to read & write to the same directory at the same time. Unless those services can handle such case.
It's better to have separate volumes though. In such case a volume for one service can be purged leaving the other one intact.
Some hints to your questions.
First Scenario : Two docker containers with a DB each.
- For this scenario since the databases are different, they run on their own container space.
- You can create a Docker Volume or use Docker bind mounts to attach disk to your Database `/var/lib/mysql'
- If you use volumes, you create one volume per database and the data are isolated.
- If you use bind mounts, make sure you mount different disk locations, if you use same location, the second database container data will overwrite the first database data.
Second Scenario : As per this scenario, since the Volume label is same, the second coming up service data would replace the already running Service data every time the services start.
Example (many options omitted for brevity):
version: "3"
volumes:
traefik:
driver: local
driver_opts:
type: nfs
o: "addr=192.168.1.100,soft,rw,nfsvers=4,async"
device: ":/volume/docker/traefik"
services:
traefik:
volumes:
- traefik/traefik.toml:/traefik.toml
This errors out as there is no volume with the name traefik/traefik.toml meaning that the volume name must be the full path to the file (i.e. you can't append a path to the volume name)?
Trying to set device: ":/volume/docker/traefik/traefik.toml" just returns a not a directory error.
Is there a way to take a single file and mount it into a container?
You cannot mount a file or sub-directory within a named volume, the source is either the named volume or a host path. NFS itself, along with most filesystems you'd mount in Linux, require you to mount an entire filesystem, not a single file, and when you get down to the inode level, this is often a really good thing.
The options remaining that I can think of are to mount the entire directory somewhere else inside your container, and symlink to the file you want. Or to NFS mount the directory to the host and do a host mount (bind mount) to a specific file.
However considering the example you presented, using a docker config would be my ideal solution, removing the NFS mount entirely, and getting a read only copy of the file that's automatically distributed to whichever node is running the container.
More details on configs: https://docs.docker.com/engine/swarm/configs/
I believe I found the issue!
Wrong:
volumes:
- traefik/traefik.toml:/traefik.toml
Correct:
volumes:
- /traefik/traefik.toml:/traefik.toml
Start the volume with "/"
I'm configuring a Docker compose file in order to start and integrate several services.
version: '3'
services:
db:
...
volumes:
- ./data/mariadb:/var/lib/mysql
...
solr:
...
volumes:
- ./data/solr/:/opt/solr/server/solr/mycores
...
When doing this, all nessessary folders are created within the folder I run docker-compose up.
Problem 1: This folders have no generic rights and docker is not able to store/write all information and the containers throw errors. Which rights do those folders need (Windows, Linux)?
As a second approach I used the volumes definition to create the data storage.
version: '3'
services:
db:
...
volumes:
- mariadb:/var/lib/mysql
...
solr:
...
volumes:
- solr:/opt/solr/server/solr/mycores
...
volumes:
mariadb:
solr:
In this setting the data folders for the volumes are created within a virtual machine and the containers can write/read all data without a problem. But the folders are somehow hidden in the Hyper-V Environment in Windows.
Problem 2: How can i define the named volumes in this section in a way that they point to a defined folder outside a VM on my physical drives (Windows, Linux)?
Problem 1: This folders have no generic rights and docker is not able to store/write all information and the containers throw errors. Which rights do those folders need (Windows, Linux)?
The directory on the host needs the permission bits set so that the user inside the container can write to it. The mapping of usernames and group names inside the container and on the host may be different due to different /etc/passwd and /etc/group files. Therefore you need to set the permissions based on the uid and gid of the user inside the container when configuring permissions on the host.
Problem 2: How can i define the named volumes in this section in a way that they point to a defined folder outside a VM on my physical drives (Windows, Linux)?
There's no workaround for setting the permissions on the directory on the host if you don't use named volumes. The only reason named volumes don't have this issue is because they get initialized to the permissions and ownership (along with the directory contents) of the image when they are empty or created and attached to a container. Host volumes (aka bind mounts) do not have this initialization feature.
If you need to refer to a volume by name instead of by path for a specific use case, then there is a plugin called local persist that may solve your issue. I've yet to encounter a scenario where this was needed so I have no feedback on how well it works.
I'm struggling to understand how exactly does the named volume work in the following example from docker docs:
version: "3"
services:
db:
image: db
volumes:
#1
- data-volume:/var/lib/db
backup:
image: backup-service
volumes:
#2
- data-volume:/var/lib/backup/data
volumes:
data-volume:
My guess is, that the first occurrence of the named volume (#1) defines what is contained inside the volume, while subsequent occurrences (#2) simply share the volume's content with whatever containers they are referenced from.
Is this guess correct?
Listing data-volume: under the top-level volumes: key creates a named volume on the host if it doesn't exist yet. This behaves the following way according to this source
If you create a named volume by running a new container from image by docker run -v my-precious-data:/data imageName, the data within the image/container under /data will be copied into the named volume.
If you create another container binds to an existing named volume, no files from the new image/container will be copied/overwritten, it will use the existing data inside the named volume.
They don’t have a docker command to backup / export a named volume. However you can find out the actual location of the file by “docker volume inspect [volume-name]”.
In case the volume is empty and both containers have data in the target directory the first container to be run will mount its data into the volume and the other container will see that data (and not its own). I don't know which container will run first (although I expect it executes from top to bottom) however you can force an order with depends_on as shown here
------------------- Update
The depends_on option is ignored when deploying a stack in swarm mode with a version 3 Compose file.
The way that I understand your guess, you are not completely correct.
Declaring and referencing a named volume in a docker-compose file will create an empty volume which may then be accessed and shared by the services saying so in their volumes section.
If you want to share a named volume, you have to declare this volume in the top-level volume section of your docker-compose file. Example (as in the docker docs already linked by yourself):
version: "3"
services:
db:
image: db
volumes:
#1 uses the named and shared volume 'data-volume' created with #3
- data-volume:/var/lib/db
backup:
image: backup-service
volumes:
#2 uses the named and shared volume 'data-volume' created with #3
- data-volume:/var/lib/backup/data
volumes:
#3 creates the named volume 'data-volume'
data-volume:
The volume will be empty on start (and therefore the folders in the containers where that volume is mounted to). Its content will be a result of the services acions on runtime.
Hope that made it a bit more clear.