I had a container running from an image, sebp/lighttpd, that had a volume mounted on it at /var/www/localhost. In my docker-compose file, I changed the config for that service to
build:
context: web/
image: myproject-web
Where neither the dockerfile at web/Dockerfile nor its from: image (bistenes/lighttpd) have any volumes defined. I ran docker-compose build and then docker-compose up, but the new container still has the old volume mounts defined, wiping the data I'm trying to COPY into /var/www/localhost when the container runs.
See this issue, which concludes with "It's not a bug, it's a feature!"
The issue indicates that docker-compose rm will do the trick.
I used docker inspect ${CONTAINER} to find the volume IDs, did docker rm ${CONTAINER}, then docker volume rm ${VOLUMES}.
Related
I'm trying to understand volumes.
When I build and run this image with docker build -t myserver . and docker run -dp 8080:80 myserver, the web server on it prints "Hallo". When I change "Hallo" to "Huhu" in the Dockerfile and rebuild & run the image/container, it shows "Huhu". So far, no surprises.
Next, I added a docker-compose.yaml file that has two volumes. One volume is mounted on an existing path of where the Dockerfile creates the index.html. The other is mounted on a new and unused path. I build and run everything with docker compose up --build.
On the first build, the web server prints "Hallo" as expected. I can also see the two volumes in Docker GUI and its contents. The index.html that was written to the image, is now present in the volume. (I guess the volume gets mounted before the Dockerfile can write to it.)
On the second build (swap "Hallo" with "huhu" and run docker compose up --build again) I was expecting the webserver to print "Huhu". But it prints "Hallo". So I'm not sure why the data on the volume was not overwritten by the Dockerfile.
Can you explain?
Here are the files:
Dockerfile
FROM nginx
# First build
RUN echo "Hallo" > /usr/share/nginx/html/index.html
# Second build
# RUN echo "Huhu" > /usr/share/nginx/html/index.html
docker-compose.yaml
services:
web:
build: .
ports:
- "8080:80"
volumes:
- html:/usr/share/nginx/html
- persistent:/persistent
volumes:
html:
persistent:
There are three different cases here:
When you build the image, it knows nothing about volumes. Whatever string is in that RUN echo line, it is stored in the image. Volumes are not mounted when you run the docker-compose build step, and the Dockerfile cannot write to a volume at all.
The first time you run a container with the volume mounted, and the first time only, if the volume is empty, Docker copies content from the mount point in the image into the volume. This only happens with named volumes and not bind mounts; it only happens on native Docker and not Kubernetes; the volume content is never updated at all after this happens.
The second time you run a container with the volume mounted, since the volume is already populated, the content from the volume hides the content in the image.
You routinely see various cases that uses named volumes to "pass through" to the image (especially Node applications) or to "share files" with another container (frequently an Nginx server). These only work because Docker (and only Docker) automatically populates empty named volumes, and therefore they only work the first time. If you change your package.json, your Node application that mounts a volume over node_modules won't see updates; if you change your static assets that you're sharing with a Web server, the named volume will hide those changes in both the application and HTTP-server containers.
Since the named-volume auto-copy only happens in this one very specific case, I'd try to avoid using it, and more generally try to avoid mounting anything over non-empty directories in your image.
Is there any proper way of restarting an entire docker compose stack from within one of its containers?
One workaround involves mounting the docker socket:
volumes:
- /var/run/docker.sock:/var/run/docker.sock
and then use the Docker Engine SDKs (https://docs.docker.com/engine/api/sdk/examples/).
However, this solution only allows restarting the containers itselves. There seems to be no way to send compose commands, like docker compose restart, docker compose up, etc.
The only solution I've found to send docker compose commands is to open a terminal on the host from the container using ssh, like this: access host's ssh tunnel from docker container
This is partly related to How to run shell script on host from docker container? , but I'm actually looking for a more specific solution to only send docker compose commands.
I tried with this simple docker-compose.yml file
version: '3'
services:
nginx:
image: nginx
ports:
- 3000:80
Then I started a docker container using
docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd):/work docker
Then, inside the container, I did
cd /work
docker-compose up -d
and it started the container up on the host.
Please note that you have an error in your socket mapping. It needs to be
- /var/run/docker.sock:/var/run/docker.sock
(you have a period instead of a slash at one point)
As mentioned by #BMitch in the comments, compose project name was the reason why I wasn't able to run docker compose commands inside the running container.
By default the compose project name is set to the directory name, so if the docker-compose.yml is run from a host directory named folder1, then the commands inside the container should be run as:
docker-compose -p folder1 ...
So now, for example, restarting the stack works:
docker-compose -p folder1 restart
Just as a reference, a fixed project name for your compose can be set using name: ... as a top-level attribute of the .yml file, but requires docker compose v2.3.3 : Set $PROJECT_NAME in docker-compose file
I'm running a docker-compose setup, and when I want to update files in my image I create a new docker image. Though the problem is; the file I'm editing is located in the persistent volume, meaning the Docker image itself will get the changes, but since I'm not deleting docker-compose volumes the volume will be used by the new image, hence the old file will be used by new image.
Running docker-compose down -v is not an options because I want to keep other existing files in the volume (logs etc.).
I want to know if it possible to do this without too much hacks, since I'm looking to automate this.
Example docker-compose.yml
version: '3.3'
services:
myService:
image: myImage
container_name: myContainer
volumes:
- data_volume:/var/data
volumes:
data_volume
NOTE: The process of doing change in my case:
docker-compose down
docker build -t myImage:t1 .
docker compose up -d
You could start a container, mount the volume and execute a command to delete single files. Something like
docker run -d --rm -v data_volume:/var/data myImage rm /var/data/[file to delete]
I have a docker-compose file which describes several services. All services have volumes attached to them, however only one has the volume named. When I run docker compose down I want to automatically delete the not named volumes while at the same time create all volumes that are missing.
services:
service1:
image: some/image:1
volumes:
- named-volume:/home/user1
service2:
image: some/image:2
#volumes: not declared volumes that are named automatically with a hash
volumes:
named-volume:
name: volume-for-service1
The first time I run docker compose up I want to automatically create all volumes (named and unnamed) and when I run docker compose down I want that unnamed volumes to be deleted while the named one (volume-for-service1) to be preserved. Next time I run docker compose up it should only create the unnamed volumes as the named one already exists.
I have tried:
docker compose down -v which removed no volume
docker compose down --remove-orphans which removed no volume
docker compose down --rmi local which removed no volume
docker-compose down -v which removed the named volume
docker-compose down --remove-orphans which removed no volume
docker-compose down --rmi local which removed no volume
OS: Windows 10 x64
I don't quite get it. What command should I run to achieve desired results?
Try using --renew-anon-volumes flag when bringing up the services
and use --volumes when bringing down the services
> docker-compose --renew-anon-volumes up
> docker-compose --volumes down
Refer the docker compose documentation
-V, --renew-anon-volumes Recreate anonymous volumes instead of retrieving
data from the previous containers.
-v, --volumes Remove named volumes declared in the `volumes`
section of the Compose file and anonymous volumes
attached to containers.
https://docs.docker.com/compose/reference/down/
To prevent removing named volumes, you should define them as external in the config file:
volumes:
volume-for-service1:
name: volume-for-service1
external: true
But you have to initially create them outside the config file somewhere else, either through:
docker volume create volume-for-service-1
or in a separate config file.
Reference: https://docs.docker.com/compose/compose-file/#external-1
I'm not aware of a way to remove unnamed volumes automatically, but you can match its hash and remove it with a small script.
To reuse your docker-compose.yml example, first you get the container name given the service name with:
docker-compose ps service2 # this is the one with unnamed volume in your example
Output could be something like:
NAME COMMAND SERVICE STATUS
project-service2-1 "docker-entrypoint.s…" service2 exited (0)
Then using the container name you can find its unamed volume hash:
docker inspect -f '{{ (index .Mounts 0).Name }}' project-service2-1
Now before deleting the volume you need to bring the container down or the volume would be in use.
docker-compose down
docker volume rm $volume # replace the "volume" var with the inspect output
Now that we saw the steps, let's try to make it a little script (slightly adjusted):
service_name=service2 # set the variable accordingly
container_id=$(docker-compose ps $service_name --quiet)
volume_name=$(docker inspect -f '{{ (index .Mounts 0).Name }}' $container_id)
docker-compose down
docker volume rm -f $volume_name
Host OS: Linux
Container OS: Linux
I'm trying to learn how to use docker. I use docker-compose and I'm successfully building images and running containers.
Now if I want to mount some directory inside the container the documentation says that I should use the COPY command inside Dockerfile.
COPY /path/to/my/addons/ /path/to/directory/inside/container
Sadly when I compose this container the COPY command is ignored and my stuff from /path/to/my/addons doesn't make it to the container.
I've also tried with ADD command, but same problem.
Absolute paths
First, you can't use absolute paths for COPY. All paths must be inside the context of the build, which means relative to the Dockerfile. If the folder structure on your host is like this
my-docker-directory
-- Dockerfile
-- docker-compose.yml
-- addons
then you're able to use COPY addons /path/to/directory/inside/container. For all subsequent explanations, I assume that you have an addons folder relative to the Dockerfile.
Mounting a directory
COPY doesn't simply mount a folder to the container at runtime. It doesn't really mount the directory at all. Instead, addons is copied to /path/to/directory/inside/container inside the image. It's important to understand, that this process happens unidirectional (host > image) and only happens when the image is build.
COPY is designed to add dependencies to the image that were required during buildtime like source code that got compiled to binaries. That's the reason why you can't absolute paths. A Dockerfile usually is placed together with source code/config files at the top level area.
The build process of an image happens only on the first run, except you force it using docker-compose up --build. But it doesn't seem that this is what you want. To mount a directory from the host at runtime, use a volume in the docker-compose file:
version: '3'
services:
test:
build: .
volumes:
- ./addons/:/path/to/directory/inside/container
When to use COPY and when volumes?
It's important to realize that COPY and ADD will copy the stuff into the image at buildtime, where volumes mount them from the host at runtume (without including them in the image). So you usually copy general things to the image, that the users needs like default configuration files.
Volumes are required to include files from the host like customized configuration files. Or persistent things as the data-directory of a database. Without volumes those containers work, but are not persistent. So all content would get lost when the container restarts.
Please note that one doesn't exclude the other. It's fine to COPY a default configuration for some application in the image, where the user may override this with volumes to modify them. Especially during development this can make things easier because you don't have to rebuild the entire image for a single changed config file*
* Altough it's a good practice to optimize Dockerfiles for the integrated caching mechanism. If a Dockerfile is well written, rebuilding small config changes often doesn't take too long. But that's another topic out of this scope.
More detailled explanation with example
Basic setup with COPY in Dockerfile
As simple example, we create a Dockerfile from the nginx webserver image and copy html in it
FROM nginx:alpine
COPY my-html /usr/share/nginx/html
Lets create the folder with demo content
mkdir my-html
echo "Dockerfile content" > my-html/index.html
and add a minimalistic docker-compose.yml
version: '3'
services:
test:
build: .
If we run it for the first time using docker-compose up -d, the image got build and our test page is served:
root#server2:~/docker-so-example# docker-compose up -d
Creating network "docker-so-example_default" with the default driver
Creating docker-so-example_test_1 ... done
root#server2:~/docker-so-example# curl $(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' docker-so-example_test_1)
Dockerfile content
Let's manipulate our testfile:
echo "NEW Modified content" > my-html/index.html
If we request our server with curl again, we get the old response:
root#server2:~/docker-so-example# curl $(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' docker-so-example_test_1)
Dockerfile content
To apply our content, a rebuild is required:
docker-compose down && docker-compose up -d --build
Now we can see our changes:
root#server2:~/docker-so-example# curl $(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' docker-so-example_test_1)
NEW Modified content
Use volumes in docker-compose
To show the difference, we use volumes by modifing our docker-compose.yml file like this:
version: '3'
services:
test:
build: .
volumes:
- ./my-html:/usr/share/nginx/html
Now restart the containers using docker-compose down && docker-compose up -d and try it again:
root#server2:~/docker-so-example# echo "Again changed content" > my-html/index.html
root#server2:~/docker-so-example# curl $(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' docker-so-example_test_1)
NEW Modified content
root#server2:~/docker-so-example# echo "Some content" > my-html/index.html
root#server2:~/docker-so-example# curl $(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' docker-so-example_test_1)
Some content
Notice that we didn't re-build the image and our modifications apply immediately. Using volumes, the files are not included in the image.
COPY command inside a docker file copies the content to the image while building. mounting a volume is a different thing. for mounting you need to use
docker run -v <volume_name>:<volume_name> ...
what exactly you want to achieve ? Do you want to see the folders inside containers in your host ?
Take your addon folder to location where your Dockerfile is and then run
mkdir -p /path/to/directory/inside/container
COPY ./addons/* /path/to/directory/inside/container