Is this possible from docker-compose - or do I have identify the volumes location and delete them?
I can only find functionality to delete ALL the volumes associated with the docker-compose.yaml config file.
Compose can only remove all volumes defined in a yml file, or none
$ docker-compose down -v
If you don't want to investigate which volumes are used for the current compose file and remove them with docker volume rm X, then create a cutdown compose file that only lists the volumes you want to remove
docker-compose.yml
version: "2.1"
volumes:
one:
two:
three:
docker-compose-cleanup.yml
version: "2.1"
volumes:
two:
Then you can act on different sets depending on the compose file
$ docker-compose up
Creating volume "composevolumes_three" with default driver
Creating volume "composevolumes_two" with default driver
Creating volume "composevolumes_one" with default driver
Attaching to
$ docker-compose -f docker-compose-cleanup.yml down -v
Removing volume composevolumes_two
Related
I have a docker-compose file which describes several services. All services have volumes attached to them, however only one has the volume named. When I run docker compose down I want to automatically delete the not named volumes while at the same time create all volumes that are missing.
services:
service1:
image: some/image:1
volumes:
- named-volume:/home/user1
service2:
image: some/image:2
#volumes: not declared volumes that are named automatically with a hash
volumes:
named-volume:
name: volume-for-service1
The first time I run docker compose up I want to automatically create all volumes (named and unnamed) and when I run docker compose down I want that unnamed volumes to be deleted while the named one (volume-for-service1) to be preserved. Next time I run docker compose up it should only create the unnamed volumes as the named one already exists.
I have tried:
docker compose down -v which removed no volume
docker compose down --remove-orphans which removed no volume
docker compose down --rmi local which removed no volume
docker-compose down -v which removed the named volume
docker-compose down --remove-orphans which removed no volume
docker-compose down --rmi local which removed no volume
OS: Windows 10 x64
I don't quite get it. What command should I run to achieve desired results?
Try using --renew-anon-volumes flag when bringing up the services
and use --volumes when bringing down the services
> docker-compose --renew-anon-volumes up
> docker-compose --volumes down
Refer the docker compose documentation
-V, --renew-anon-volumes Recreate anonymous volumes instead of retrieving
data from the previous containers.
-v, --volumes Remove named volumes declared in the `volumes`
section of the Compose file and anonymous volumes
attached to containers.
https://docs.docker.com/compose/reference/down/
To prevent removing named volumes, you should define them as external in the config file:
volumes:
volume-for-service1:
name: volume-for-service1
external: true
But you have to initially create them outside the config file somewhere else, either through:
docker volume create volume-for-service-1
or in a separate config file.
Reference: https://docs.docker.com/compose/compose-file/#external-1
I'm not aware of a way to remove unnamed volumes automatically, but you can match its hash and remove it with a small script.
To reuse your docker-compose.yml example, first you get the container name given the service name with:
docker-compose ps service2 # this is the one with unnamed volume in your example
Output could be something like:
NAME COMMAND SERVICE STATUS
project-service2-1 "docker-entrypoint.s…" service2 exited (0)
Then using the container name you can find its unamed volume hash:
docker inspect -f '{{ (index .Mounts 0).Name }}' project-service2-1
Now before deleting the volume you need to bring the container down or the volume would be in use.
docker-compose down
docker volume rm $volume # replace the "volume" var with the inspect output
Now that we saw the steps, let's try to make it a little script (slightly adjusted):
service_name=service2 # set the variable accordingly
container_id=$(docker-compose ps $service_name --quiet)
volume_name=$(docker inspect -f '{{ (index .Mounts 0).Name }}' $container_id)
docker-compose down
docker volume rm -f $volume_name
In my Docker environment I have always to run the command to create volumes manually like
docker volume create --name= ...
I would like a way to speed up this process with a script shell which could help me to run at once.
If I could see a possible solution would be great as I have many volumes to create manually
A possible solution would be to use docker-compose and have a docker_compose.yml file composed only of volumes but no services:
version: "3.8"
volumes:
logvolume01: {}
logvolume02: {}
logvolume03: {}
When run, this creates the volumes accordingly:
$ docker-compose up
Creating volume "docker_logvolume01" with default driver
Creating volume "docker_logvolume02" with default driver
Creating volume "docker_logvolume03" with default driver
Attaching to
$ docker volume ls
DRIVER VOLUME NAME
local docker_logvolume01
local docker_logvolume02
local docker_logvolume03
If you need a more complex set of options while creating your volumes, you can find them in the documentation.
Just a little quirk to note here: per default, when you are using docker-compose, the volumes will be prefixed with the name of the folder you are in, this is done by Docker so there is no collision between different Docker projects.
This is the reason why, in the example above, the volumes are starting with docker_, because the folder I am in, is called docker.
To fix this, just give a name to your volumes:
version: "3.8"
volumes:
logvolume01:
name: logvolume01
logvolume02:
name: logvolume02
logvolume03:
name: logvolume03
Running this modified version gives:
$ docker-compose up
Creating volume "logvolume01" with default driver
Creating volume "logvolume02" with default driver
Creating volume "logvolume03" with default driver
Attaching to
$ docker volume ls
DRIVER VOLUME NAME
local logvolume01
local logvolume02
local logvolume03
When I create a volume manually and include it in docker-compose, if I don't prefix the volume tag with docker_, docker compose creates a new volume prefixed with docker_
For example:
I create a volume with:
docker volume create myvolume
It's visible at /var/lib/docker/volumes/myvolume.
I include it in my docker-compose yaml file, but when I run docker-compose, a new volume is created at /var/lib/docker/volumes/docker_myvolume
If I call my volume docker_myvolume and include that in my docker-compose yaml, it uses it and doesn't create it's own.
Is this normal behavior?
Yes, this is normal behavior. When you specify a volume in your docker-compose.yml file without a leading driver_ prefix, Docker Compose will create a new volume with a name that is prefixed with driver_. This is because Docker Compose uses a default driver for creating and managing volumes, which is the local driver.
You can specify a volume in your docker-compose.yml file with the external option to tell Docker Compose to use an existing volume instead of creating a new one. For example:
version: '3'
services:
myservice:
volumes:
- type: volume
source: myvolume
target: /app/data
volume:
external: true
This will tell Docker Compose to use the existing volume myvolume instead of creating a new one.
Yes, Compose generally prefixes things with its project name. This includes containers, networks, and named volumes. In general, if you actually need to interact with these things, there is an equivalent docker-compose command that chooses the correct name (e.g., docker-compose exec).
In general you shouldn't be directly modifying things inside /var/lib/docker. That directory tree is Docker's private state and there are no particular guarantees about the format of files there. If your use case involves directly interacting with the volume files from the host, either use a /host/path:/container/path bind mount or explicitly specify the storage location using volume options.
The docker-compose.yml contains a volume definition and a service that uses it.
#(This is a reduced but representative example)
services:
mongo:
image: mongo:3.6.11
...
volumes:
- "mongodb:/data/db/"
volumes:
mongodb:
driver: local
driver_opts:
type: "none"
o: "bind"
device: "$PWD/.storage/mongodb"
I've removed the entire .storage/mongodb directory, wanting to force the next docker-compose up not reuse any volume.
Instead, I received an error: docker was missing its volume data.
Creating mongo ... error
ERROR: for mongo Cannot create container for service mongo: failed to mount local volume: mount <...>/.storage/mongodb:/var/lib/docker/volumes/mongodb/_data, flags: 0x1000: no such file or directory
How I tried to resolve that:
docker-build up --build --force-recreate. Container's build and restart don't work.
docker volume ls -q | while read id; do docker volume rm "$id"; done. Removing volumes didn't seem to help either.
I assume that docker is able to recover from the removed volumes data, since the very first docker-compose up runs without initialised .storage directory.
Basically, I want docker-compose up to work.
It also would be fantastic to understand the issue.
Look forward to your help!
Have you tried docker-compose up with -V option?
-V, --renew-anon-volumes Recreate anonymous volumes instead of retrieving
data from the previous containers.
When using docker compose, I find a lot of volume instances:
› docker volume ls
DRIVER VOLUME NAME
local 4a34b9a352a459171137aac4c046a83f61e6e325b1df4b67dc2ddda8439a6427
local 6ce3e52ea363441b2c9d4b04c26b283d8b4cf631a137987da88db812a9a2d223
local a7af289b29c833510f2201647266001e4746e206128dc63313fe894821fa044d
local fb09475f75fe943671a4e73d76c09c27a4f592b8ddf62224fc4b20afa0095809
I'd like to tag or name them, then reuse them if possible rather recreating them each time.
Is that possible?
Those are anonymous container volumes that happen when you define a volume without a name or bind it to a host folder. This may be with the VOLUME definition in your Dockerfile, a docker run -v /dir ... rather than name:/dir, or a volumes entry in your docker-compose.yml with only the directory. An example of a compose file that does a named mount is:
version: '2'
volumes:
my-vol:
driver: local
services:
my-container:
image: my-image
volumes:
- my-vol:/container/path
Once the anonymous volume has been created, there's no easy way to rename it. Easiest solution is to mount the anonymous volume along with the your target named volume and do a copy, e.g.:
docker run -v 123456789:/source -v my-vol:/target --rm \
busybox cp -av /source/. /target/
Where 123456789 is the long name of your anonymous volume.