I have a problem that I just can't understand. I am using docker to run certain containers, but I have problems with at least one Volume, where I't like to ask if anybody can give me a hint what I am doing wrong. I am using Nifi-Ingestion as example, but it affects even more container volumes.
First, let's talk about the versions I use:
Docker version 19.03.8, build afacb8b7f0
docker-compose version 1.27.4, build 40524192
Ubuntu 20.04.1 LTS
Now, let's show the volume in my working docker-compose-file:
In my container, it is configured as followed:
volumes:
- nifi-ingestion-conf:/opt/nifi/nifi-current/conf
Below my docker-compose file it is defined as a normal named volume:
volumes:
nifi-ingestion-conf:
This is a snippet from the docker-compose that I'd like to get working
In my container, it is configured in this case as followed (having my STORAGE_VOLUME_PATH defined as /mnt/storage/docker_data):
volumes:
- ${STORAGE_VOLUME_PATH}/nifi-ingestion-conf:/opt/nifi/nifi-current/conf
On the bottom I guess there is something to do but I don't know what I could need to do here. In this case it is the same as in the working docker-compose:
volumes:
nifi-ingestion-conf:
So, now whats my problem?
I have two docker-compose files. One uses the normal named volumes, and one uses the volumes in my extra mount path. When I run the containers, the volumes seem to work different since files are written in the first style, but not in the second. My mount paths are generated in the second version so there is nothing wrong with my environment variables in the .env-file.
Hint: the /mnt/storage/docker_data is an NFS-mount but my machine has the full privileges on that share.
Here is my fstab-entry to mount that volume (maybe I have to set other options):
10.1.0.2:/docker/data /mnt/storage/docker_data nfs auto,rw
Bigger snippets
Here is a bigger snipped if the docker-compose (i need to cut and remove confident data, my problem is not that it does not work, it is only that the volume acts different. Everything for this one volume is in the code.):
version: "3"
services:
nifi-ingestion:
image: my image on my personal repo
container_name: nifi-ingestion
ports:
- 0000
labels:
- app-specivic
volumes:
- ${STORAGE_VOLUME_PATH}/nifi-ingestion-conf:/opt/nifi/nifi-current/conf
#working: - nifi-ingestion-conf:/opt/nifi/nifi-current/conf
environment:
- app-specivic
networks:
- cnetwork
volumes:
nifi-ingestion-conf:
networks:
cnetwork:
external: false
ipam:
driver: default
config:
- subnet: 192.168.1.0/24
And here of the env (only the value we are using)
STORAGE_VOLUME_PATH=/mnt/storage/docker_data
if i understand your question correctly, you wonder why the following docker-compose snippet works for you
version: "3"
services:
nifi-ingestion:
volumes:
- nifi-ingestion-conf:/opt/nifi/nifi-current/conf
volumes:
nifi-ingestion-conf:
and the following docker-compose snippet does not work for you
version: "3"
services:
nifi-ingestion:
volumes:
- ${STORAGE_VOLUME_PATH}/nifi-ingestion-conf:/opt/nifi/nifi-current/conf
what makes them different is how you use volumes. you need to differentiate between mount host paths and mount named volumes
You can mount a host path as part of a definition for a single service, and there is no need to define it in the top level volumes key.
But, if you want to reuse a volume across multiple services, then define a named volume in the top-level volumes key.
named volumes are managed by docker
If you start a container with a volume that does not yet exist, Docker creates the volume for you.
also, would advise you to read this answer
update:
you might also want to read about docker nfs volumes
Related
I am writing a docker-compose.yaml file for my project. I have checked the volumes documentation here .
I also understand the concept of volume in docker that I can mount a volume e.g. -v my-data/:/var/lib/db where my-data/ is a directory on my host machine while /var/lib/db is the path inside database container.
My confuse is with the link I put above. There it has the following sample:
version: "3.9"
services:
db:
image: db
volumes:
- data-volume:/var/lib/db
backup:
image: backup-service
volumes:
- data-volume:/var/lib/backup/data
volumes:
data-volume:
I wonder does it mean that I have to create a directory named data-volume on my host machine? What if I have a directory on my machine with path temp/my-data/ and I want to mount that path to the database container /var/lib/db ? Should I do something like below?
version: "3.9"
services:
db:
image: db
volumes:
- temp/my-data/:/var/lib/db
volumes:
temp/my-data/:
My main confusion is the volumes: section at the bottom, I am not sure whether the volume name should be the path of my directory or should be just literally a name I give & if it is the latter case then how could the given name be mapped with temp/my-data/ on my machine? The sample doesn't indicate that & is ambiguous to clarify that.
Could someone please clarify it for me?
P.S. I tried with above docker-compose I guessed, ended up with the error:
ERROR: The Compose file './docker-compose.yaml' is invalid because:
volumes value 'temp/my-data/' does not match any of the regexes: '^[a-zA-Z0-9._-]+$'
Mapped volumes can either be files/directories on the host machine (sometimes called bind mounts in the documentation) or they can be docker volumes that can be managed using docker volume commands.
The volumes: section in a docker-compose file specify docker volumes, i.e. not files/directories. The first docker-compose in your post uses such a volume.
If you want to map a file or directory (like in your last docker-compose file), you don't need to specify anything in the volumes: section.
Docker volumes (the ones specified in the volumes: section or created using docker volume create) are of course also stored somewhere on your host computer, but docker manages that and you shouldn't normally need to know where or what the format is.
This part of the documentation is pretty good about explaining it, I think https://docs.docker.com/storage/volumes/
As #HansKilian mentions, you don't need both volumes and services.volumes. To use services.volumes, map the host directory to the container directory like this:
services:
db:
image: db
volumes:
- /host/path/lib/db:/container/path/lib/db
With that, the directory /host/path/lib/db on the host machine will be used by the container and available at /container/path/lib/db.
Now, if you're like me, I get really confused with fake examples, so let's say the real directory on your host machine is /var/lib/db and you just want to see it at /db when you run a shell in Docker (i.e., docker exec -it /bin/bash container-id).
docker-compose.yaml would look like this:
services:
db:
image: db
volumes:
- /var/lib/db:/db
Now when you run the shell, cd /logs and ls, you'll see the same results as if you'd cd /var/lib/db on the host.
If you want to use the volumes section to indicate a global volume to use, you first have to create that volume using docker volume create. The documentation Hans linked includes steps to do this. The syntax of /host/path:/container/path is replaced by volume-name:/container/path. Then, once defined, you'd alter your docker-compose.yaml to be more like this:
services:
db:
image: db
volumes:
- your-global-volume-name:/db
volumes:
your-global-volume-name:
external: true
Note that I have not tested or used the this configuration. I'm assuming it's correct based on the other method working and the few changes I can identify in the docs.
Due to space limitations on my local machine I need to ensure my docker containers store their data on my external hard drive.
The docker project I am using has a docker compose file and it specifies a number of volumes like so:
version: "2"
volumes:
pgdata:
cache:
services:
postgres:
image: "openmaptiles/postgis:${TOOLS_VERSION}"
volumes:
- pgdata:/var/lib/postgresql/data
Those volumes ultimately exist on my local machine. I'd like to change their location to somewhere on my external drive e.g /Volumes/ExternalDrive/docker/
How do I go about this?
I have read the docker documentation on volumes and and docker-compose but can't find a way to specify the path of where a volume should exist.
If anyone could point the way I would be most grateful.
You can explore and test more features related to volumes using the CLI help and then transpose to compose.
docker volume create --help
https://docs.docker.com/engine/reference/commandline/volume_create/
Note that example below might not work on Windows hence the built-in local driver on Windows does not support any options, but if you're running docker on Linux, the compose file below should do the job:
version: "2"
volumes:
pgdata:
driver: local
driver_opts:
type: 'none'
o: 'bind'
device: '/path/to/the/external/storage'
cache:
services:
postgres:
image: "openmaptiles/postgis:${TOOLS_VERSION}"
volumes:
- pgdata:/var/lib/postgresql/data
You might also consider to change the docker launch options to store its data in a location of your choice. Here's a guide: https://linuxconfig.org/how-to-move-docker-s-default-var-lib-docker-to-another-directory-on-ubuntu-debian-linux
Alternatively, if you're more comfortable trying to solve this in the OS rather than docker, you could try some tricks at the filesystem level to create a symbolic link at /var/lib/docker/volumes/ to point at your external storage, but be careful and backup everything. I personally never tried this, but I believe it should be transparent for the docker storage driver.
I have setup two standalone docker containers, one runs a webserver another one runs a mysql for it.
Right now I was attempting to have it working with docker-compose. All is nice and it runs well, but I was wondering how could I re-use existing volumes from the existing standalone containers that I have previously created (since I want to retain the data from them).
I saw people suggesting to use external: true command for this, but could not get the right syntax so far.
Is external: true the correct way approach for this, or should I approach this differently?
Or can I just specify the path to the volume within docker-compose.yml and make it use the old existing volume?
Yes you can do it normally, just an example below:
Set external to true and set name to the name of the volume you want to mount.
version: "3.5"
services:
transmission:
image: linuxserver/transmission
container_name: transmission
volumes:
- transmission-config:/config
- /path/to/downloads:/downloads
ports:
- 51413:51413
- 51413:51413/udp
networks:
- rede
restart: always
networks:
rede:
external: true
name: rede
volumes:
transmission-config:
external: true
name: transmission-config
Per the documentation, using the external flag allows you to use volumes created outside the scope of the docker-compose file.
However, it is advisable to create a fresh volume via the docker-compose file and copy the existing data from the old volumes to the new volumes
You can create a volume explicitly using the docker volume create command, or Docker can create a volume during container or service creation. When you create a volume, it is stored within a directory on the Docker host. When you mount the volume into a container, this directory is what is mounted into the container.
If your system is running, you can exec into the mysql container, copy and move it outside.
docker cp "${container_id}":/path_to_folder /path_to_server
The following is an example given in https://docker-curriculum.com/
version: "3"
services:
es:
image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
container_name: es
environment:
- discovery.type=single-node
ports:
- 9200:9200
volumes:
- esdata1:/usr/share/elasticsearch/data
web:
image: prakhar1989/foodtrucks-web
command: python app.py
depends_on:
- es
ports:
- 5000:5000
volumes:
- ./flask-app:/opt/flask-app
volumes:
esdata1:
driver: local
and it says The volumes parameter specifies a mount point in our web container where the code will reside about the /opt/flask-app
I think it means, /opt/flask-app is a mount point and it points to the host machines ./flask-app
However it doesn't say anything about esdata1 and I can't apply the same explanation as given to /opt/flask-app since there's no esdata1 directory/file in the host machine.
What is happening for the esdata1 ?
My guess is that it means creating a volume (The closest thing I can think of is a disk partition) and name it esdata1 and mount it on /usr/share/elasticsearch/data, am I correct on this guess?
These are a bit different things: volumes and bind mounts. Bind mounts let you specify folder on host machine, which would serve as a storage. Volumes (at lease for local driver) also have folders on host machines, but their location is managed by Docker and is sometimes a bit more difficult to find.
When you specify volume in docker-compose.yml, if your path starts with / or . it becomes a bind mount, like in web service. Otherwise, if it is a single verb, it is a volume, like for es service.
You can inspect all volumes on your host machine by running docker volume ls.
What is happening for the esdata1 ? My guess is that it means creating
a volume (The closest thing I can think of is a disk partition) and
name it esdata1 and mount it on /usr/share/elasticsearch/data, am I
correct on this guess?
That's all correct.
I do not pretend on setting up the rules, but in general, volumes are more suitable for sharing common data between several containers, like in docker-compose, while bind mounts suite better for sharing data from host to container, like some initial configs for services.
I have Docker commands to create a container and then use that container's name with --volumes-from to run another container and it works fine -
docker create -v /home/dev/docker/my/config:/home/myuser/4.0/config --name shared-config my/configurator:4.0.0
The above would create a new container by name shared-config from image my/configurator:4.0.0
and when trying to run any other container (say my/oms:4.0.0) I can simply use volume from container named shared-config using --volumes-from
docker run --volumes-from shared-config -p 8083:8080 -d my/oms:4.0.0
using --volumes-from we can use the volume multiple times in which ever container it is required.
Till here everything seems fine.
Now, I am trying to do the above in docker-compose using file-format version "3" and not able to understand how will I be able to re-use data-volume once it is created. Since as per docker-compose in version 3 they have discontinued use of --volumes-from.
They say -
To share a volume between services, define it using the top-level volumes option and reference it from each service that shares it using the service-level volumes option.
In above statement they are referring to named volumes, please refer from here.
But I just want to mount a host directory as a data volume and re-use that data volume. My question is how do I reuse this data-volume through docker-compose file version "3".
To the simplest, for each service I want to run through docker-compose I can use volume key at service level
version: "3"
services:
my-oms:
image: my/oms:4.0.0
ports:
- "8083:8080"
volumes:
- /home/dev/docker/my/config:/home/myuser/4.0/config
But what if I want to use my host's directory (/home/dev/docker/my/config) as a data volume in different services. Should I have the volume key for each service or actually there is a better way in docker-compose version "3" where I can re-use the data-volume in other services (how we did using --volumes-from).
Any pointers or suggestions or something that I missed?
The best option to avoid repeating syntax is to extend your docker-compose.yml using the extends option:
So you can have a common-services.yml that looks like:
version: "3"
services:
generic-vol:
volumes:
- /home/dev/docker/my/config:/home/myuser/4.0/config
And then your docker-compose.yml gets updated to look like:
version: "3"
services:
my-oms:
extends:
file: common-services.yml
service: generic-vol
image: my/oms:4.0.0
ports:
- "8083:8080"
Note that docker stack deploy -c docker-compose.yml may not support all these options, I've encountered issues using variables and multiple docker-compose files for my project. The solution to that is to use docker-compose to parse the file into something the stack deploy can use with docker-compose config >docker-compose.stack.yml and then pass that yml file to your stack deploy.
A second option is to utilize the features of the yml syntax itself. It allows anchors and references to those anchors. That syntax looks like:
version: "3"
services:
my-oms:
image: my/oms:4.0.0
ports:
- "8083:8080"
volumes: &common-vol
- /home/dev/docker/my/config:/home/myuser/4.0/config
my-xyz:
image: my/xyz:4.0.0
ports:
- "8888:8080"
volumes: *common-vol
The first &common-vol creates an anchor, and the later *common-vol is a reference to that same part of yml data.