I have setup two standalone docker containers, one runs a webserver another one runs a mysql for it.
Right now I was attempting to have it working with docker-compose. All is nice and it runs well, but I was wondering how could I re-use existing volumes from the existing standalone containers that I have previously created (since I want to retain the data from them).
I saw people suggesting to use external: true command for this, but could not get the right syntax so far.
Is external: true the correct way approach for this, or should I approach this differently?
Or can I just specify the path to the volume within docker-compose.yml and make it use the old existing volume?
Yes you can do it normally, just an example below:
Set external to true and set name to the name of the volume you want to mount.
version: "3.5"
services:
transmission:
image: linuxserver/transmission
container_name: transmission
volumes:
- transmission-config:/config
- /path/to/downloads:/downloads
ports:
- 51413:51413
- 51413:51413/udp
networks:
- rede
restart: always
networks:
rede:
external: true
name: rede
volumes:
transmission-config:
external: true
name: transmission-config
Per the documentation, using the external flag allows you to use volumes created outside the scope of the docker-compose file.
However, it is advisable to create a fresh volume via the docker-compose file and copy the existing data from the old volumes to the new volumes
You can create a volume explicitly using the docker volume create command, or Docker can create a volume during container or service creation. When you create a volume, it is stored within a directory on the Docker host. When you mount the volume into a container, this directory is what is mounted into the container.
If your system is running, you can exec into the mysql container, copy and move it outside.
docker cp "${container_id}":/path_to_folder /path_to_server
Related
I am writing a docker-compose.yaml file for my project. I have checked the volumes documentation here .
I also understand the concept of volume in docker that I can mount a volume e.g. -v my-data/:/var/lib/db where my-data/ is a directory on my host machine while /var/lib/db is the path inside database container.
My confuse is with the link I put above. There it has the following sample:
version: "3.9"
services:
db:
image: db
volumes:
- data-volume:/var/lib/db
backup:
image: backup-service
volumes:
- data-volume:/var/lib/backup/data
volumes:
data-volume:
I wonder does it mean that I have to create a directory named data-volume on my host machine? What if I have a directory on my machine with path temp/my-data/ and I want to mount that path to the database container /var/lib/db ? Should I do something like below?
version: "3.9"
services:
db:
image: db
volumes:
- temp/my-data/:/var/lib/db
volumes:
temp/my-data/:
My main confusion is the volumes: section at the bottom, I am not sure whether the volume name should be the path of my directory or should be just literally a name I give & if it is the latter case then how could the given name be mapped with temp/my-data/ on my machine? The sample doesn't indicate that & is ambiguous to clarify that.
Could someone please clarify it for me?
P.S. I tried with above docker-compose I guessed, ended up with the error:
ERROR: The Compose file './docker-compose.yaml' is invalid because:
volumes value 'temp/my-data/' does not match any of the regexes: '^[a-zA-Z0-9._-]+$'
Mapped volumes can either be files/directories on the host machine (sometimes called bind mounts in the documentation) or they can be docker volumes that can be managed using docker volume commands.
The volumes: section in a docker-compose file specify docker volumes, i.e. not files/directories. The first docker-compose in your post uses such a volume.
If you want to map a file or directory (like in your last docker-compose file), you don't need to specify anything in the volumes: section.
Docker volumes (the ones specified in the volumes: section or created using docker volume create) are of course also stored somewhere on your host computer, but docker manages that and you shouldn't normally need to know where or what the format is.
This part of the documentation is pretty good about explaining it, I think https://docs.docker.com/storage/volumes/
As #HansKilian mentions, you don't need both volumes and services.volumes. To use services.volumes, map the host directory to the container directory like this:
services:
db:
image: db
volumes:
- /host/path/lib/db:/container/path/lib/db
With that, the directory /host/path/lib/db on the host machine will be used by the container and available at /container/path/lib/db.
Now, if you're like me, I get really confused with fake examples, so let's say the real directory on your host machine is /var/lib/db and you just want to see it at /db when you run a shell in Docker (i.e., docker exec -it /bin/bash container-id).
docker-compose.yaml would look like this:
services:
db:
image: db
volumes:
- /var/lib/db:/db
Now when you run the shell, cd /logs and ls, you'll see the same results as if you'd cd /var/lib/db on the host.
If you want to use the volumes section to indicate a global volume to use, you first have to create that volume using docker volume create. The documentation Hans linked includes steps to do this. The syntax of /host/path:/container/path is replaced by volume-name:/container/path. Then, once defined, you'd alter your docker-compose.yaml to be more like this:
services:
db:
image: db
volumes:
- your-global-volume-name:/db
volumes:
your-global-volume-name:
external: true
Note that I have not tested or used the this configuration. I'm assuming it's correct based on the other method working and the few changes I can identify in the docs.
I have a problem that I just can't understand. I am using docker to run certain containers, but I have problems with at least one Volume, where I't like to ask if anybody can give me a hint what I am doing wrong. I am using Nifi-Ingestion as example, but it affects even more container volumes.
First, let's talk about the versions I use:
Docker version 19.03.8, build afacb8b7f0
docker-compose version 1.27.4, build 40524192
Ubuntu 20.04.1 LTS
Now, let's show the volume in my working docker-compose-file:
In my container, it is configured as followed:
volumes:
- nifi-ingestion-conf:/opt/nifi/nifi-current/conf
Below my docker-compose file it is defined as a normal named volume:
volumes:
nifi-ingestion-conf:
This is a snippet from the docker-compose that I'd like to get working
In my container, it is configured in this case as followed (having my STORAGE_VOLUME_PATH defined as /mnt/storage/docker_data):
volumes:
- ${STORAGE_VOLUME_PATH}/nifi-ingestion-conf:/opt/nifi/nifi-current/conf
On the bottom I guess there is something to do but I don't know what I could need to do here. In this case it is the same as in the working docker-compose:
volumes:
nifi-ingestion-conf:
So, now whats my problem?
I have two docker-compose files. One uses the normal named volumes, and one uses the volumes in my extra mount path. When I run the containers, the volumes seem to work different since files are written in the first style, but not in the second. My mount paths are generated in the second version so there is nothing wrong with my environment variables in the .env-file.
Hint: the /mnt/storage/docker_data is an NFS-mount but my machine has the full privileges on that share.
Here is my fstab-entry to mount that volume (maybe I have to set other options):
10.1.0.2:/docker/data /mnt/storage/docker_data nfs auto,rw
Bigger snippets
Here is a bigger snipped if the docker-compose (i need to cut and remove confident data, my problem is not that it does not work, it is only that the volume acts different. Everything for this one volume is in the code.):
version: "3"
services:
nifi-ingestion:
image: my image on my personal repo
container_name: nifi-ingestion
ports:
- 0000
labels:
- app-specivic
volumes:
- ${STORAGE_VOLUME_PATH}/nifi-ingestion-conf:/opt/nifi/nifi-current/conf
#working: - nifi-ingestion-conf:/opt/nifi/nifi-current/conf
environment:
- app-specivic
networks:
- cnetwork
volumes:
nifi-ingestion-conf:
networks:
cnetwork:
external: false
ipam:
driver: default
config:
- subnet: 192.168.1.0/24
And here of the env (only the value we are using)
STORAGE_VOLUME_PATH=/mnt/storage/docker_data
if i understand your question correctly, you wonder why the following docker-compose snippet works for you
version: "3"
services:
nifi-ingestion:
volumes:
- nifi-ingestion-conf:/opt/nifi/nifi-current/conf
volumes:
nifi-ingestion-conf:
and the following docker-compose snippet does not work for you
version: "3"
services:
nifi-ingestion:
volumes:
- ${STORAGE_VOLUME_PATH}/nifi-ingestion-conf:/opt/nifi/nifi-current/conf
what makes them different is how you use volumes. you need to differentiate between mount host paths and mount named volumes
You can mount a host path as part of a definition for a single service, and there is no need to define it in the top level volumes key.
But, if you want to reuse a volume across multiple services, then define a named volume in the top-level volumes key.
named volumes are managed by docker
If you start a container with a volume that does not yet exist, Docker creates the volume for you.
also, would advise you to read this answer
update:
you might also want to read about docker nfs volumes
I have for example this service and volume defined in my docker-compose file
postgres:
image: postgres:9.4
volumes:
- db_data:/var/lib/postgresql/data
volumes:
blue_prod_db:
driver: rancher-nfs
Then. if you define a volume inside a Dockerfile like this:
RUN mkdir /stuff
COPY ./stuff/* /stuff/
VOLUME /stuff
How can you later access it through the docker-compose configuration and add it to a container?
When configured in the Dockerfile, a volume will result in any container started from that image, including temporary containers later in the build process from the RUN command, to have a volume defined at the specified location, e.g. /stuff. If you do not define a source for that volume at run time, you will get an anonymous volume created by docker for you at that location. However, you can always define a volume with a source at run time (even without the volume being defined) by specifying the location in your compose file:
version: "3"
services:
app:
image: your_image
volumes:
- data:/stuff
volumes:
data:
Note that there are two volumes sections, one for a specific service that specifies where the volume is mounted inside the container, and another at the top level where you can specify the source of the volume. Without specifying a source, you'll get a local volume driver with a directory under /var/lib/docker bind mounted into the container.
I do not recommend specifying volumes inside the Dockerfile in general, it breaks the ability to extend the image in later steps for child images, and clutters the filesystem with anonymous volumes that are not easy to track back to their origin. It's best to define them at runtime with something like a compose file.
I have image A (some_laravel_project) and B (laravel_module). Image A is a Laravel project that looks like this.
app
modules
core
Volume Image b here
config
As the list above suggests I want to share a volume from Image B in Image A using docker-compose. I want to access the files in container B.
This is the docker-compose I tried and didn't receive any errors creating those images in gitlab ci. I checked and the volume and its files are in stored in the module_user:latest container.
I think I made a mistake mounting the volume to some_laravel_project.
version: '3'
services:
laravel:
image: some_laravel_project
working_dir: /var/www
volumes:
- /var/www/storage
- userdata:/var/www/Modules
user:
image: laravel_module
volumes:
- userdata:/user
volumes:
userdata:
webroot:
The method you used to share volumes across container in docker compose is the correct one. You can find this documented under docker-compose volumes
if you want to reuse a volume across multiple services, then define a
named volume in the top-level volumes key. Use named volumes with
services,
In you case, the directory /var/www/Modules in laravel will have the same content as that in /user inside user service. You can verify that by going into the containers and checking each directoty by running;
docker exec -it <container-name> bash
I have Docker commands to create a container and then use that container's name with --volumes-from to run another container and it works fine -
docker create -v /home/dev/docker/my/config:/home/myuser/4.0/config --name shared-config my/configurator:4.0.0
The above would create a new container by name shared-config from image my/configurator:4.0.0
and when trying to run any other container (say my/oms:4.0.0) I can simply use volume from container named shared-config using --volumes-from
docker run --volumes-from shared-config -p 8083:8080 -d my/oms:4.0.0
using --volumes-from we can use the volume multiple times in which ever container it is required.
Till here everything seems fine.
Now, I am trying to do the above in docker-compose using file-format version "3" and not able to understand how will I be able to re-use data-volume once it is created. Since as per docker-compose in version 3 they have discontinued use of --volumes-from.
They say -
To share a volume between services, define it using the top-level volumes option and reference it from each service that shares it using the service-level volumes option.
In above statement they are referring to named volumes, please refer from here.
But I just want to mount a host directory as a data volume and re-use that data volume. My question is how do I reuse this data-volume through docker-compose file version "3".
To the simplest, for each service I want to run through docker-compose I can use volume key at service level
version: "3"
services:
my-oms:
image: my/oms:4.0.0
ports:
- "8083:8080"
volumes:
- /home/dev/docker/my/config:/home/myuser/4.0/config
But what if I want to use my host's directory (/home/dev/docker/my/config) as a data volume in different services. Should I have the volume key for each service or actually there is a better way in docker-compose version "3" where I can re-use the data-volume in other services (how we did using --volumes-from).
Any pointers or suggestions or something that I missed?
The best option to avoid repeating syntax is to extend your docker-compose.yml using the extends option:
So you can have a common-services.yml that looks like:
version: "3"
services:
generic-vol:
volumes:
- /home/dev/docker/my/config:/home/myuser/4.0/config
And then your docker-compose.yml gets updated to look like:
version: "3"
services:
my-oms:
extends:
file: common-services.yml
service: generic-vol
image: my/oms:4.0.0
ports:
- "8083:8080"
Note that docker stack deploy -c docker-compose.yml may not support all these options, I've encountered issues using variables and multiple docker-compose files for my project. The solution to that is to use docker-compose to parse the file into something the stack deploy can use with docker-compose config >docker-compose.stack.yml and then pass that yml file to your stack deploy.
A second option is to utilize the features of the yml syntax itself. It allows anchors and references to those anchors. That syntax looks like:
version: "3"
services:
my-oms:
image: my/oms:4.0.0
ports:
- "8083:8080"
volumes: &common-vol
- /home/dev/docker/my/config:/home/myuser/4.0/config
my-xyz:
image: my/xyz:4.0.0
ports:
- "8888:8080"
volumes: *common-vol
The first &common-vol creates an anchor, and the later *common-vol is a reference to that same part of yml data.