I want to recreate a service, including its volumes. The closest I got was the following commands:
docker-compose stop foo
docker-compose rm -f foo
docker-compose up --renew-anon-volumes -d foo
docker-compose start foo
The issue here is --renew-anon-volumes recreates all services that have anonymous volumes, not just foo's volumes. If I don't use --renew-anon-volumes, then I think I need a named volume to do docker volume rm myvolume. However, with named volumes, Docker Compose always prepends a project name. Since my script doesn't know the project name, I can't programmatically delete the volume. I can't enforce that the user uses a particular project name. I know I can set the project name using an environment variable, but there's no guarantee that the user won't run Docker Compose with a different project name.
I think there are 2 potential solutions:
Make --renew-anon-volumes only recreate the volumes for the service I specified
Use a named volume and somehow figure out the correct prefix
Are either of these doable, or is there another solution?
Many roads leading to Rome, depending on your prerequisites:
Do docker volume ls and regex the result for your named volume (just working if volume name is unique)
Use external volumes and volume create them with known names by bootstrap script before running docker-compose up (not working if volumes must be instantiated)
Set project name to a known value. Normally it takes the folder name, but can explicitly given in docker-compose command (-p NAME) or by environment variable (COMPOSE_PROJECT_NAME=NAME).
Setup a dummy compose-file just containing this single service with its volumes for your script. Doing a docker-compose -f 'your-down-file.yml' down -v which removes all named and anonymous volumes belonging to this service and docker-compose -f .. up on this file.
Edit (#DavidMaze):
You're right, docker compose recognizes that fact. But it does NOT remove it, just warning. If you want to remove all "orphans" you need the flag --remove-orphans.
But for some reasons the down does not remove volumes then, even if flag -v is given. This could be reported because it is not behaving like described.
And errata: the flag -f must go before up/down and not after!
docker-compose rm has a -v option to delete anonymous volumes attached to a container, and also a -s option to stop the container. For your particular use case it should be enough to:
docker-compose rm -s -f -v foo
docker-compose up -d foo
This will only help for anonymous volumes, that is, where the Compose file has volumes: with only a container path and there is no corresponding top-level volumes: entry. I don't immediately see a Compose option to list, remove, or otherwise manage named volumes that Compose created.
Related
TLDR: When using docker compose, I can simply recreate a container by changing its configuration and/or image in the docker-compose.yml file along with running docker-compose up. Is there any generic equivalent for recreating a container (to apply changes) which was created by a bare docker create/run command?
Elaborating a bit:
The associated docker compose documentation states:
If there are existing containers for a service, and the service’s configuration or image was changed after the container’s creation, docker-compose up picks up the changes by stopping and recreating the containers (preserving mounted volumes).
I'm having troubles to understand which underlaying steps are actually performed during this recreation, as e.g. the docker (without compose) documentation doesn't really seem to use the recreate term at all.
Is it safe to simply run docker container rm xy and then docker container create/run (along with passing the full and modified configuration)? Or is docker compose actually doing more under the hood?
I already found answers about applying specific configuration changes like e.g. this one about port mappings, but I'm still wondering whether there is a more general answer to this.
I'm having troubles to understand which underlaying steps are actually performed during this recreation, as e.g. the docker (without compose) documentation doesn't really seem to use the recreate term at all.
docker-compose is a high level tool; it performs in a single operation what would require multiple commands using the docker cli. When docker-compose says, "docker-compose up picks up the changes by stopping and recreating the containers", it means it is doing the equivalent of:
docker stop <somecontainer>
docker rm <somecontainer>
docker run ...
(Where ... represents whatever configuration is implied by the service definition in your docker-compose.yaml).
Let's say it recognizes a change in container1 it does (not really, working via API):
docker compose rm -fs container1
docker compose create (--build) container1
docker compose start container1
What is partially close to (depending on your compose-config):
docker rm -f projectname_container1
(docker build --flags)
docker create --allDozensOfAttributes projectname_container1
docker start projectname_container1
docker network connect (--flags) projectname_networkname projectname_container1
and maybe more..
so i would advise to use the docker compose commands for single services instead of docker cli if suitable..
The issue is that the variables and settings are not exposed through any docker apis. It may be possible by way of connecting directly to the docker socket, parsing the variables, and then stopping/removing the container and recreating it.
This would be prone to all kinds of errors and would require lots of debugging to get these values.
What I do is to simply store my docker commands in a shell script. You can just save the command you need to run into a text file, name it .sh, set the -x on the file, then run it. Then when you stop/delete the container, you can just rerun the shell script.
Another thing you can do would be to replace the docker command with a function (in something like your ~/.bashrc) that stores the arguments to a text file and rechecks that text file with a passed argument (like "recreate" followed by a name). However, I'm more a fan of doing docker containers in their own shell scripts as its more portable.
I have multiple containers running in my machine. Each of them belongs to a docker-compose.yaml file and they create different volumes.
Now I'd like to delete the volumes of containers created from a specific docker-compose.yaml. I can not find anything online.
docker ps -a, docker ps -aq, docker volume ls => None of this gives an association.
I'd like to delete the volumes [...] created from a specific docker-compose.yaml
docker-compose down -v deletes containers, networks, and volumes. (Without -v it keeps volumes for future use, since they're presumed to contain user data that's hard to recreate.) You don't need to know the specific Docker volume ID for this to work.
There are equivalent docker-compose commands to run most common tasks. These are generally driven off of the service name (block heading) in the docker-compose.yml, so you don't need to know or specify the Docker name of a container or network.
My team is using docker-compose for our project's container.
Previously, while we were learning how Docker works, we were just using docker. In docker, I could instantly see my local changes on my local deployment by attaching a bind mount to the container when I run it on the command line.
Now, using docker-compose, there doesn't seem to be any such option - my workflow is docker-compose up with little opportunity for deviation. I believe I can specify a bind mount as a volume in our docker-compose.yaml. But that's not really what I'm looking for.
I'd like to be able to specify a local, personal, temporary bind mount without having to modify my team's docker-compose.yaml or commit my personal preferences to vc. How can I specify my bind mount from command line, or equivalent?
Looking at the docs, there doesn't seem to be such an option from the commandline.
One way to get similar behavior to what you want is to make a second docker-compose file with the personal options. If you name it docker-compose.override.yaml, it will be picked up automatically. Otherwise, you can use the -f flag to load it, like so
docker-compose -f docker-compose.yaml -f docker-compose.user.yaml up -d
More details for using multiple docker-compose files are documented here (thanks #Sysix).
You could put this filename into gitignore.
-V, --renew-anon-volumes Recreate anonymous volumes instead of retrieving
data from the previous containers.
Does docker-compose up -V not apply to named volumes?
I have a service that, at image build time, pulls some files from SVN. It then creates a named volume.
I can docker-compose build --no-cache to recreate the image and pull latest files from SVN. But the volume doesn't get updated on docker-compose up -V, unless I remove it beforehand.
I just want a clean and simple way to update files in a named volume
Sure, I could manually remove the volume, and then run everything, but I really wanted it to be compose-driven. This leads me to a second problem.
There is a docker-compose down -v that will also remove volumes, but I cannot run that against a single service (only all or nothing).
So I need to somehow figure out the named volumes of just 1 service from compose-file, and then use some extra command (docker volume rm?) to remove just that one volume.
If you don't care about the content of a named volume, either don't create it in the first place (remove the named volume line in the compose file) or delete it when you stop the project with:
docker-compose down -v
I decided to change the project name of a docker composition:
$ docker-compose -p old_name up -d # Before
Starting old_name_web_1
$ docker-compose -p new_name up -d # After
Creating new_name_web_1
But I don't wanted to delete my containers, so I renamed them:
$ docker rename old_name_web_1 new_name_web_1
...
I thought docker-compose was based on container names, but it does not seem to be the case:
$ docker-compose -p new_name up -d
ERROR: for web Cannot create container for service web: Conflict. The name "/new_name_web_1" is already in use by container 4930deaabb[...]. You have to remove (or rename) that container to be able to reuse that name.
ERROR: Encountered errors while bringing up the project.
How can I relink my old containers to the new composition ?
It looks like you are using one of the newer versions of docker compose which tracks containers by labels assigned to them rather than by their names. That is why renaming the container didn't work.
Updating labels
You can check container's labels through the docker inspect command.
$ docker inspect --format='{{json .Config.Labels }}' container_name
The project name is the value of the 'com.docker.compose.project' label.
Moving an existing container to a new project is as easy as changing the value of that label. However it is not yet supported by Docker CLI. There is an open issue requesting that feature.
Workaround
It still can be achieved by directly editing the configuration file of that particular container. There you will find labels currently assigned to that container.
$ nano /var/lib/docker/containers/$container_id/config.v2.json
Assign the new project name to the 'com.docker.compose.project' label and save the file. Next you have to restart the daemon. Otherwise the changes will not be visible to docker.
$ systemctl daemon-reload
While it is true docker-compose reuse existing containers, this comment mentions:
docker-compose by default uses the folder name of the yml file as the project name, and prefix that name to all container names.
This could explain why docker-compose up did not pick up the new container name.