Mounted volume is empty inside container - docker

I've got a docker-compose.yml like this:
db:
image: mongo:latest
ports:
- "27017:27017"
server:
image: artificial/docker-sails:stable-pm2
command: sails lift
volumes:
- server/:/server
ports:
- "1337:1337"
links:
- db
server/ is relative to the folder of the docker-compose.yml file. However when I docker exec -it CONTAINERID /bin/bash and check /server it is empty.
What am I doing wrong?

Aside from the answers here, it might have to do with drive sharing in Docker Setting. On Windows, I discovered that drive sharing needs to be enabled.
In case it is already enabled and you recently changed your PC's password, you need to disable drive sharing (and click "Apply") and re-enable it again (and click "Apply"). In the process, you will be prompted for your PC's new password. After this process, run your docker command (run or compose) again

Try using:
volumes:
- ./server:/server
instead of server/ -- there are some cases where Docker doesn't like the trailing slash.

As per docker volumes documentation,
https://docs.docker.com/engine/tutorials/dockervolumes/#/mount-a-host-directory-as-a-data-volume
The host-dir can either be an absolute path or a name value. If you
supply an absolute path for the host-dir, Docker bind-mounts to the
path you specify. If you supply a name, Docker creates a named volume
by that name

I had similar issue when I wanted to mount a directory from command line:
docker run -tid -p 5080:80 -v /d/my_project:/var/www/html/my_project nimmis/apache-php5
The container has been started successfully but the mounted directory was empty.
The reason was that the mounted directory must be under the user's home directory. So, I created a symlink under c:\Users\<username> that mounts to my project folder d:\my_project and mounted that one:
docker run -tid -p 5080:80 -v /c/Users/<username>/my_project/:/var/www/html/my_project nimmis/apache-php5

If you are using Docker for Mac then you need to go to:
Docker Desktop -> Preferences -> Resources -> File Sharing
and add the folder you intend to mount. See the screenshot:

I don't know if other people made the same mistake but the host directory path has to start from /home
So my msitake was that in my docker-compose I was WRONGLY specifying the following:
services:
myservice:
build: .
ports:
- 8888:8888
volumes:
- /Desktop/subfolder/subfolder2:/app/subfolder
When the host path should have been full path from /home. something like:
services:
myservice:
build: .
ports:
- 8888:8888
volumes:
- home/myuser/Desktop/subfolder/subfolder2:/app/subfolder

On Ubuntu 20.04.4 LTS, with Docker version 20.10.12, build e91ed57, I started observing a similar symptom with no apparent preceding action. After a docker-compose -p production-001 -f deploy/docker-compose.yml up -d --build command, with no changes to one of the services (production-001-volumeConsumingService is up-to-date), a part of the volumes stopped mounting.
# deploy/docker-compose.yml
version: "3"
services:
...
volumeConsumingService:
container_name: production-001-volumeConsumingService
hostname: production-001-volumeConsumingService
image: group/production-001-volumeConsumingService
build:
context: .
dockerfile: volumeConsumingService.Dockerfile
depends_on:
- anotherServiceDefinedEarlier
restart: always
volumes:
- ../data/certbot/conf:/etc/letsencrypt # mouning
- ../data/certbot/www:/var/www/certbot # not mounting
- ../data/www/public:/var/www/public # not mounting
- ../data/www/root:/var/www/root # not mounting
command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"
networks:
- default
- external
...
networks:
external:
name: routing
A workaround that seems to be working is to enforce a restart on the failing service immediately after the docker-compose -p production-001 -f deploy/docker-compose.yml up -d --build command:
docker-compose -p production-001 -f deploy/docker-compose.yml up -d --build && docker stop production-001-volumeConsumingService && docker start production-001-volumeConsumingService
In the case when the volumes are not mounted after a host reboot, adding a cron task to restart the service once should do.

In my case, the volume was empty because I did not use the right path format without quotes.
If you have a relative or absolute path with spaces in it, you do not need to use double quotes around the path, you can just use any path with spaces and it will be understood since docker-compose has the ":" as the delimiter and does not check spaces.
Ways that do not work (double quotes are the problem!):
volumes:
- "MY_PATH.../my server":/server
- "MY_PATH.../my server:/server" (I might have missed testing this, not sure!)
- "./my server":/server
- ."/my server":/server
- "./my server:/server"
- ."/my server:/server"
Two ways how you can do it (no double quotes!):
volumes:
- MY_PATH.../my server:/server
- ./my server:/server

Related

docker-compose create a directory instead of mounting the file

I have a very simple docker-compose.yml file where I use nginx and mounting a file as a volume.
But everytime I run the application, it is creating a directory .htpasswd without really mounting the .htpasswd file where I locally.
This is the docker-compose.yml.
version: '3'
services:
reverse:
container_name: reverse
hostname: reverse
restart: unless-stopped
image: nginx
ports:
- 80:80
- 443:443
volumes:
- ./nginx/.htpasswd:/etc/nginx/conf.d/.htpasswd
Can someone help me fix this?
by default if binded to a none existent path, docker will create a folder, the solution would be in your case to create the path before running your docker-compose
How are you running Docker? here's an answer ...
For Mac with Minikube/Hyperkit docker and Docker Compose
Since I'm not using Docker Desktop any longer, I've experienced numerous issues similar to "docker in docker (dind)" paradigm with minikube...
mount minikube
use absolute path
e.g., easiest way was to mount the exact home path...
minikube mount $HOME:/Users/<you>
... keeps running...
docker-compose.yaml
volumes:
- /Users/<you>/path/to/file.yaml:/somedir/file.yaml

Docker persisted volum has no permissions (Apache Solr)

My docker-compose.yml:
solr:
image: solr:8.6.2
container_name: myproject-solr
ports:
- "8983:8983"
volumes:
- ./data/solr:/var/solr/data
networks:
static-network:
ipv4_address: 172.20.1.42
After bringing up the docker with docker-compose up -d --build, the solr container is down and the log (docker logs myproject-solr) shows this:
Copying solr.xml
cp: cannot create regular file '/var/solr/data/solr.xml': Permission denied
I've noticed that if I give full permissions on my machine to the data directory sudo chmod 777 ./data/solr/ -R and I run the Docker again, everything is fine.
I guess the issue comes when the solr user is not my machine, because Docker creates the data/solr folder with root:root. Having my ./data folder gitignored, I cannot manage these folder permissions.
I'd like to know a workaround to manage permissions properly with the purpose of persisting data
It's a known "issue" with docker-compose: all files created by Docker engine are owned by root:root. Usually it's solved in one of the two ways:
Create the volume in advance. In your case, you can create the ./data/solr directory in advance, with appropriate permissions. You might make it accessible to anyone, or, better, change its owner to the solr user. The solr user and group ids are hardcoded inside the solr image: 8983 (Dockerfile.template)
mkdir -p ./data/solr
sudo chown 8983:8983 ./data/solr
If you want to avoid running additional commands before docker-compose, you can create additional container which will fix the permissions:
version: "3"
services:
initializer:
image: alpine
container_name: solr-initializer
restart: "no"
entrypoint: |
/bin/sh -c "chown 8983:8983 /solr"
volumes:
- ./data/solr:/solr
solr:
depends_on:
- initializer
image: solr:8.6.2
container_name: myproject-solr
ports:
- "8983:8983"
volumes:
- ./data/solr:/var/solr/data
networks:
static-network:
ipv4_address: 172.20.1.42
There is docker-compose-only solution :)
Problem
Docker mounts local folders with root permissions.
In Solr's docker image, the default user is solr - for a good reason: Solr commands should be run with this user (you can force to run them with root but that is not recommended).
Most Solr commands require write permissions to /var/solr/, for data and logs storage.
In this context, when you run a solr command as the solr user, you are rejected because you don't have write permission to /var/solr/.
Solution
What you can do is to first start the container as root to change the permissions of /var/solr/. And then switch to solr user to run all necessary solr commands. You can't start our Solr server.
In the example below, we use solr-precreate to create a default core and start solr.
version: '3.7'
services:
solr:
image: solr:8.5.2
volumes:
- ./mnt/solr:/var/solr
ports:
- 8983:8983
user: root # run as root to change the permissions of the solr folder
# Change permissions of the solr folder, create a default core and start solr as solr user
command: bash -c "
chown -R 8983:8983 /var/solr
&& runuser -u solr -- solr-precreate default-core"
Set with a Dockerfile
It's possibly not exactly what you wanted as the files aren't persisted when rebuilding the container, but it solves the 'rights' problem. Copy the files over and chown them with a Dockerfile:
FROM solr:8.7.0
COPY --chown=solr ./data /var/solr/data
This is more useful if you're trying to initialise a single core:
FROM solr:8.7.0
COPY --chown=solr ./core /var/solr/data/someCollection
It also has the advantage that you can create an image for reuse.
With a named volume
For persistence, you can also create a volume (in this case core) and copy the contents of a directory (also called core here), assigning the rights to the files on the way:
docker container create --name temp -v core:/data tianon/true || exit $?
tar -cf - --directory core --owner 8983 --group 8983 . | docker cp - temp:/data
docker rm temp
This was adapted from these answers:
https://github.com/moby/moby/issues/25245#issuecomment-365980572
https://stackoverflow.com/a/52446394
Then you can mount the named volume in your Docker Compose file:
version: '3'
services:
solr:
image: solr:8.7.0
networks:
- internal
ports:
- 8983:8983
volumes:
- core:/var/solr/data/someCollection
volumes:
core:
external: true
This solution persists the data without overriding the data on the host. And it doesn't need the extra build step. And can obviously be adapted for mounting the entire /var/solr/data folder.
It doesn't seem to matter that the mounted volume/directory doesn't have the correct rights (/var/solr/data/someCollection has owner root:root).

Docker named volumes between multiple containers

I want to deploy some services into my server and all of them will use nginx as web server, every project has it own .conf file and I want to share all of then with nginx container. I tried to use named volumes but when it's used by more than one container the data gets replaced. I want to get all this .conf files from diferent containers and put in a volume so it can be read by nginx container. I also tried to use subdirectories in named volumes, but, use namedVolumeName/path do not work.
Obs: I'm using docker-compose in all projects
version: "3.7"
services:
backend:
container_name: jzmimoveis-backend
image: paulomesquita/jzmimoveis-backend
command: uwsgi --socket :8000 --wsgi-file jzmimoveis/wsgi.py
volumes:
- nginxConfFiles:/app/nginx
- jzmimoveisFiles:/app/src
networks:
- jzmimoveis
restart: unless-stopped
expose:
- 8000
frontend:
container_name: jzmimoveis-frontend
image: paulomesquita/jzmimoveis-frontend
command: serve -s build/
volumes:
- nginxConfFiles:/app/nginx
networks:
- jzmimoveis
restart: unless-stopped
expose:
- 5000
volumes:
nginxConfFiles:
external: true
jzmimoveisFiles:
external: true
networks:
jzmimoveis:
external: true
For example, is this case i linked both frontend and backend nginx file to the named volume nginxConfFiles, but, when I do docker-compose up -d in this file, just one of the .conf file appears in volume, I think it gets overwritten by the other container in the same file.
Probably you could have, on the nginx container, the shared volume pointing to /etc/nginx/conf.d, and then use different names for each project conf file.
Below a proof-of-concept, three servers with a config file to be attached on each one, and a proxy (your Nginx) with the shared volume bound to /config:
version: '3'
services:
server1:
image: busybox:1.31.1
volumes:
- deleteme_after_demo:/config
- ./server1.conf:/app/server1.conf
command: sh -c "cp /app/server1.conf /config; tail -f /dev/null"
server2:
image: busybox:1.31.1
volumes:
- deleteme_after_demo:/config
- ./server2.conf:/app/server2.conf
command: sh -c "cp /app/server2.conf /config; tail -f /dev/null"
server3:
image: busybox:1.31.1
volumes:
- deleteme_after_demo:/config
- ./server3.conf:/app/server3.conf
command: sh -c "cp /app/server3.conf /config; tail -f /dev/null"
proxy1:
image: busybox:1.31.1
volumes:
- deleteme_after_demo:/config:ro
command: tail -f /dev/null
volumes:
deleteme_after_demo:
Let's create 3 config files to be included:
➜ echo "server 1" > server1.conf
➜ echo "server 2" > server2.conf
➜ echo "server 3" > server3.conf
then:
➜ docker-compose up -d
Creating network "deleteme_default" with the default driver
Creating deleteme_server2_1 ... done
Creating deleteme_server3_1 ... done
Creating deleteme_server1_1 ... done
Creating deleteme_proxy1_1 ... done
And finally, let's verify the config files are accessible from proxy container:
➜ docker-compose exec proxy1 sh -c "cat /config/server1.conf"
server 1
➜ docker-compose exec proxy1 sh -c "cat /config/server2.conf"
server 2
➜ docker-compose exec proxy1 sh -c "cat /config/server3.conf"
server 3
I hope it helps.
Cheers!
Note: you should see mounting a volume exactly the same way as using Unix mount command. If you already have content inside the mount point, after mount you are not going to see it, but the content of the mounted device (unless it was empty and first created here). Whatever you want to see there needs to be already on the device or you need to move it afterward.
So, I did it by mounting the files because I had no data in the container I used. Then copying these with the starting command. You could address it a different way, eg copying the config file to the mounted volume by the use of an entry point script in your image.
A named volume is initialized when it's empty/new and a container is started using that volume. The initialization is from the image filesystem, and after that, the named volume is persistent and will retain the state from the previous use.
In this case, what you have is a race condition. The volume is sharing the files, but it depends on which container compose starts up first to control which image is used to initialize the volume. The named volume is shared between multiple images, it's just the content that you want to be different.
For your use case, you may be better off putting some logic in the image build and entrypoint to save the files you want to mirror in the volume to a different location in the image on build, and then update the volume on container startup. By moving this out of the named volume initialization steps, you avoid the race condition, and allow the volume to be updated with future changes from the image. An example of this is in my base image with the save-volume you'd run in the Dockerfile, and load-volume you'd run in your entrypoint.
As a side note, it's also a good practice to mount that named volume as read-only in the containers that have no need to write to the config files.

Docker Compose with shell environment variable error

I'm running a Docker deployment for an application. I'm mounting a volume where I want the external path to be provided by a shell environment variable. I get this error:
ERROR: for video-server Cannot create container for service video-server: invalid volume specification: '46b9d2fb3b9b13c9404d31bae571dac3f633122393c4a77f2561afb8aed5c06e:=/opt/videos:rw': invalid mount config for type "volume": invalid mount path: '=/opt/videos' mount path must be absolute
My docker-compose configuration is this:
video-server:
build:
context: .
dockerfile: video-server_Dockerfile
container_name: video-server
networks:
- videoManagerNetwork
environment:
- VIDEO_MANAGER_DIR=/opt/videos
volumes:
- ${VIDEO_MANAGER_DIR_PROD}=/opt/videos
ports:
- 9000:8080
I can see the correct value of the VIDEO_MANAGER_DIR_PROD environment variable by doing both of these commands, so I know it's on my shell:
echo $VIDEO_MANAGER_DIR_PROD
sudo echo $VIDEO_MANAGER_DIR_PROD
What's strange is that, if I do a complete wipe of my docker configurations (sudo docker system prune --all --volumes), and then run the docker-compose for the first time (sudo docker-compose up -d), everything works.
However, if I take the container down, rebuild it, and try to run that same command (sudo docker-compose up -d) again, then I get the error displayed above.
You cannot assign the source volume like a variable, so you will use : for this assignment.
Documentation about Docker Compose volumes: docs.docker.com
video-server:
build:
context: .
dockerfile: video-server_Dockerfile
container_name: video-server
networks:
- videoManagerNetwork
environment:
- VIDEO_MANAGER_DIR: /opt/videos
volumes:
- ${VIDEO_MANAGER_DIR_PROD}:/opt/videos
ports:
- 9000:8080

Docker Compose for Rails

I'm trying to replicate this docker command in a docker-compose.yml file
docker run --name rails -d -p 80:3000 -v "$PWD"/app:/www -w /www -ti rails
My docker-compose.yml file:
rails:
image: rails
container_name: rails
ports:
- 80:3000
volumes:
- ./app:/wwww
When I'm doing docker-compose up -d, the container is created but it does not strat.
When I'm adding tty: true to my docker docker-compose.yml file, the container start well but my volume is not mounted.
How can I replicate excatly my docker command in a docker-compose.yml?
There are some ways to solve your problem.
Solution 1: If you want to use the rails image in your docker-compose.yml, you need to set the command and working directory for it like
rails:
image: rails
container_name: rails
command: bash -c "bundle install && rails server -b 0.0.0.0"
ports:
- 80:3000
volumes:
- ./app:/www
working_dir: /www
This will create a new container from the rails image every time you run docker-compose up.
Solution 2: Move your docker-compose.yml to the same directory with Gemfile, and create Dockerfile in that directory in order to build a docker container in advance (to avoid running bundle installevery time)
#Dockerfile
FROM rails:onbuild
I use rails:onbuild here for simplicity reasons (about the differences between rails:onbuild and rails:<version>, please see the documentation).
After that, modify the docker-compose.yml to
rails:
build: .
container_name: rails
ports:
- 80:3000
volumes:
- .:/www
working_dir: /www
Run docker-compose up and this should work!
If you modify your Gemfile, you may also need to rebuild your container by docker-compose build before running docker-compose up.
Thanks for your answer. It helped me to find the solutions.
It was actually a volume problem. I wanted to mount the volume with the directory /www. But it was not possible.
So I used the directory used by default with the rails images:
/usr/src/app
rails:
image: rails
container_name: rails
ports:
- 80:3000
working_dir: /usr/src/app
volumes:
- ./app:/usr/src/app
tty: true
Now my docker-compose up -d command works

Resources