docker-compose: create named volume as specific user - docker

Here's a minimal docker-compose.yml:
version: "3.6"
services:
foo:
user: "${UID}:${GID}"
image: node:latest
container_name: foo
working_dir: /var/www/foo
volumes:
- bar:/var/www/foo/bar
volumes:
bar:
After running docker-compose up the named volume is created at /var/lib/docker/volumes/project_bar, and the project_bar directory is owned by root - this is fine. Although the _data directory inside it is also own by root - I would like it to be own by a specific, non-root user.
For example:
docker-compose run --rm foo sh -c "mkdir bar/foo"
will fail with mkdir: cannot create directory 'bar/foo': Permission denied.
I was under the impression that specifying user: "${UID}:${GID}" (note: uid/gid are available to docker, exported in shell they are) will solve this. Obviously, I was wrong.
Is there a way to have /var/lib/docker/volumes/project_bar/_data being own by specific user? Without chown'ing anything if possible. And without involving Dockerfile.
EDIT: To clarify, I would like /var/www/foo/bar in the container and its equivalent in /var/lib/docker/volumes to be owned by a specific, not root user.

You are either involving the Dockerfile, or you need to setup the volume with a separate process. The named volume will be initialized with the contents of the image, including files, directories, owners, and permissions. This happens when the volume is empty or non-existent.
So your options include:
update the image to include the desired ownership of that directory. Note if the image defines this as a volume, your change must preced that definition.
create a volume in advance with the desired ownership and mount that in the container
modify the entrypoint of the container to run as root, fix the permissions, and then switch to the desired user (using something like gosu)
Expanding on the second option, you can create a volume with the desired ownership and content by running a temporary container to do the task:
docker container run --rm -v bar-data:/data busybox /bin/sh -c \
"touch /data/.initialized && chown -R ${UID}:${GID} /data"
Then in your compose file you can declare that volume as external:
version: "3.6"
services:
foo:
user: "${UID}:${GID}"
image: node:latest
container_name: foo
working_dir: /var/www/foo
volumes:
- bar:/var/www/foo/bar
volumes:
bar:
# tell compose this volume is externally created with a different name
external: true
name: bar-data
That will create a volume with a single file, needed to prevent docker from reinitializing an empty volume from the image contents. This also means you need to load any data you want in the volume since it will be otherwise empty. A common method to copy data into a volume involves using tar:
tar -cC source_dir . | \
docker run --rm -i -v foo-data:/target busybox tar -xC /target

Related

How to COPY in Dockfiler after volume is mounted

I have a docker-compose.yml file that mounts a couple of volumes. Here is a snippet:
version: '3'
services:
gitlab-runner:
build: '.'
volumes:
- gitlab-config-volume:/etc/gitlab-runner
volumes:
gitlab-config-volume:
external: false
However, my Dockerfile has a COPY action into /etc/gitlab-runner/certs
FROM gitlab/gitlab-runner:latest
COPY files/ca.crt /etc/gitlab-runner/certs/ca.crt
The problem is, that this COPY happens before the mount. Is there a way I can work around this issue?
The easiest approach is to not mount a volume over content configured in your Dockerfile.
services:
gitlab-runner:
build: .
# no volumes:
If the volume actually is configuration, and it includes environment-specific settings like TLS CA certificates, it might not make sense to include this in your image at all. It will usually be easier to inject these files from the host system than to try to copy them into a named Docker volume.
services:
gitlab-runner:
# (could still `build: .`, but don't copy config into the image)
image: gitlab/gitlab-runner:latest
volumes:
# uses a host directory and not a named volume
- ./gitlab-config:/etc/gitlab-runner
cp files/ca.crt gitlab-config/certs/ca.crt
docker-compose up -d
Finally, if you really want the file to be included in your image, but to be able to mount some other content into it, you need to run code in your container to copy the file into the volume. This happens at container startup, so it will happen after the volume mount. This is the most complex option, though.
The shell script is straightforward:
#!/bin/sh
# Copy the CA certificate into the configuration directory if required.
if [ -f /etc/gitlab-runner/ca.crt ]; then
cp /opt/config/ca.crt /etc/gitlab-runner/certs
fi
# Run the main container command.
exec "$#"
In your Dockerfile, you need to make this script be the ENTRYPOINT (with JSON-array syntax); you need to repeat the CMD from the base image; and you need to copy the default file into the filesystem, somewhere other than the volume mount point. Doing this correctly involves knowing some details of the base image, and finding its Dockerfile is all but essential.
FROM gitlab/gitlab-runner:latest
COPY files/ca.crt /opt/config
COPY entrypoint.sh /opt/config
# These last two lines are derived from the base image and are not generic
ENTRYPOINT ["/dumb-init", "/opt/config/entrypoint.sh", "/entrypoint"]
CMD ["run", "--user=gitlab-runner", "--working-directory=/home/gitlab-runner"]

Undefined volume with Docker Compose

I wanted to translate this docker CLI command (from smallstep/step-ca) into a docker-compose.yml file to run with docker compose (version 2):
docker run -d -v step:/home/step \
-p 9000:9000 \
-e "DOCKER_STEPCA_INIT_NAME=Smallstep" \
-e "DOCKER_STEPCA_INIT_DNS_NAMES=localhost,$(hostname -f)" \
smallstep/step-ca
This command successfully starts the container.
Here is the compose file I "composed":
version: "3.9"
services:
ca:
image: smallstep/step-ca
volumes:
- "step:/home/step"
environment:
- DOCKER_STEPCA_INIT_NAME=Smallstep
- DOCKER_STEPCA_INIT_DNS_NAMES=localhost,ubuntu
ports:
- "9000:9000"
When I run docker compose up (again, using v2 here), I get this error:
service "ca" refers to undefined volume step: invalid compose project
Is this the right way to go about this? I'm thinking I missed an extra step with volume creation in docker compose projects, but I am not sure what that would be, or if this is even a valid use case.
The Compose file also has a top-level volumes: block and you need to declare volumes there.
version: '3.9'
services:
ca:
volumes:
- "step:/home/step"
et: cetera
volumes: # add this section
step: # does not need anything underneath this
There are additional options possible, but you do not usually need to specify these unless you need to reuse a preexisting Docker named volume or you need non-standard Linux mount options (the linked documentation gives an example of an NFS-mount volume, for example).
Citing the Compose specification:
To avoid ambiguities with named volumes, relative paths SHOULD always begin with . or ...
So it should be enough to make your VOLUME's host path relative:
services:
ca:
volumes:
- ./step:/home/step
If you don't intend to share the step volume with other containers, you don't need to define it in the top-level volumes key:
If the mount is a host path and only used by a single service, it MAY be declared as part of the service definition instead of the top-level volumes key.
it seems that docker-compose don't know the "volume" you created via command: sudo docker volume create my_xx_volume
so ,just manually mkdir to create a folder and chmod 777 <my_folder>, then your mysql docker will use it very well.
( in production env, don't use chmod but chown to change the file permission )

Mounted directory empty with docker-compose and custom Dockerfile

I am very (read very) new to Docker so experimenting. I have created a very basic Dockerfile to pull in Laravel:
FROM composer:latest
RUN composer_version="$(composer --version)" && echo $composer_version
RUN composer global require laravel/installer
WORKDIR /var/www
RUN composer create-project --prefer-dist laravel/laravel site
My docker-compose.yml file looks like:
version: '3.7'
services:
laravel:
build:
context: .
dockerfile: laravel.dockerfile
container_name: my_laravel
network_mode: host
restart: on-failure
volumes:
- ./site:/var/www/site
When I run docker-compose up, the ./site directory is created but the contents are empty. I've put this in docker-compose as I plan on on including other things like nginx, mysql, php etc
The command:
docker run -v "/where/i/want/data/site:/var/www/site" my_laravel
Results in the same behaviour.
I know the install is successful as I modified my dockerfile with the follwing two lines appended to it:
WORKDIR /var/www/site
RUN ls -la
Which gives me the correct listing.
Clearly misunderstanding something here. Any help appreciated.
EDIT: So, I was able to get this to work... although, it slightly more difficult than just specifying a path..
You can accomplish this by specifying a volume in docker-compose.yml.. The path to the directory (on the host) is labeled as device in the compose file.. It appears that the root of the path has to be an actual volume (possibly a share would work) but the 'destination' of the path can be a directory on the specified volume..
I created a new volume called docker on my machine but I suppose you could do this with your existing disk/volume..
I am on a Mac and this docker-compose.yml file worked for me:
version: '3.7'
services:
nodemon-test:
container_name: my-nodemon-test
image: oze4/nodemon-docker-test
ports:
- "1337:1337"
volumes:
- docker_test_app:/app # see comment below on which name to use here
volumes:
docker_test_app: # use this name under `volumes:` for the service
name: docker_test_app
driver: local
driver_opts:
o: bind
type: none
device: /Volumes/docker/docker_test_app
The container specified exists in my DockerHub.. this is the source code for it, just in case you are worried about anything malicious. I created it like two weeks ago to help someone else on StackOverflow.
Shows files from the container on my machine (the host)..
You can read more about Docker Volume configs here if you would like.
ORIGINAL ANSWER:
It looks like you are trying to share the build directory with your host machine.. After some testing, it appears Docker will overwrite the specified path on the container with the contents of the path on the host.
If you run docker logs my_laravel you should see an error about missing files at /var/www/site.. So, even though the build is successful - once Docker mounts the directory from your machine (./site) onto the container (/var/www/site) it overwrites the path within the container (/var/www/site) with the contents of the path on your host (./site) - which is empty.
To test and make sure the contents of /var/www/site are in fact being overwritten, you can run docker exec -it /bin/bash (you may need to replace /bin/bash with /bash).. This will give you command line access inside of the container. From there you can do ls -a /var/www/site..
Furthermore, you can also pre-stage ./site to have a random test file in it (test.txt or whatever), then docker-compose up -d, then run the same commands from the step above docker exec -it ... and see if the staged test.txt file is now inside the container - this gives you definitive evidence that when you run volumes, the data on your host overwrites data in the container.
With that being said, doing something like this and sharing a log directory will work... the volume path specified on the container is still overwritten, the difference is the container is writing to that path.. it doesn't rely on it for config files/app files.
Hope this helps.

Why does docker-compose up not seem to sync volumes

Here is a simplified version of my docker-compose.yml (it's the volume in buggy-service that does not behave as I expect):
version: '3.4'
services:
local-db:
image: postgres:9.6
environment:
- DB_NAME=${DB_NAME}
# other env vars (not important)
ports:
- 5432:5432
volumes:
- ~/.docker-volumes/${DB_NAME}/postgresql/data:/var/lib/postgresql/data
- postgresql:/docker-entrypoint-initdb.d
buggy-service:
build:
context: .
dockerfile: Dockerfile.test
target: buggy-image
args:
# bunch of args (not important)
volumes:
- /Users/me/temp:/temp
volumes:
postgresql:
driver_opts:
type: none
device: /Users/me/postgresql
o: bind
If I do docker-compose -f docker-compose.yml up -d local-db, a container for it starts up automatically and I find that /Users/me/postgresql on the host machine (Mac OSX) binds correctly to /docker-entrypoint-initdb.d with content synced.
However, if I do docker-compose -f docker-compose.yml up --build -d buggy-service, a container does not start up automatically.
Question: How do I get buggy-service to behave like local-db, i.e., start up automatically with the required volume mounted?
Here's the stripped down version of Dockerfile.test referenced by buggy-service:
FROM microsoft/dotnet:2.1-sdk-alpine AS buggy-image
# Bunch of ARG definitions (not important)
VOLUME /temp
# other stuff (not important)
ENTRYPOINT ["/bin/bash"]
# Other FROMs
Edit 1
A bit more info about what I’m trying to achieve...
The buggy-container I’m trying to get working runs .Net Core as the base image. Its purpose is to run dotnet test and generate coverage reports, which can then be consumed in the host, which may either be a local dev machine or a build server (in this case, BitBucket pipelines).
... followed by docker run -dit --name buggy-container buggy-image
This command creates a new container, not based on anything in the compose yml file. Without a volume specification, it will only get an anonymous volume since you've defined the volume in the Dockerfile (I tend to recommend against defining a volume there). You can see the anonymous volumes with a docker volume ls command, they'll be the ones with a long unique id and no reference to what they belong to.
To define a host volume from docker run, you need the -v flag:
docker run -dit -v /Users/me/temp:/temp --name buggy-container buggy-image
From your now changed question, you have a new issue. Your container specifies a single command to run in the entrypoint:
ENTRYPOINT ["/bin/bash"]
When bash runs, it reads input from stdin. When that input ends, like when you run a container with no input attached, bash will exit. When the process your container runs exits, the container exits. From the details available, I can't tell you what that command should be, but a good starting point is to look at other images on docker hub that perform a similar task that you're trying to run, and look at the Dockerfile they use (many hub images point back to a GitHub repo with the full source).

Docker-compose set user and group on mounted volume

I'm trying to mount a volume in docker-compose to apache image. The problem is, that apache in my docker is run under www-data:www-data but the mounted directory is created under root:root. How can I specify the user of the mounted directory?
I tried to run command setupApacheRights.sh. chown -R www-data:www-data /var/www but it says chown: changing ownership of '/var/www/somefile': Permission denied
services:
httpd:
image: apache-image
ports:
- "80:80"
volumes:
- "./:/var/www/app"
links:
- redis
command: /setupApacheRights.sh
I would prefer to be able to specify the user under which it will be mounted. Is there a way?
To achieve the desired behavior without changing owner / permissions on the host system do the following steps.
get the ID of the desired user and or group you want the permissions to match with executing the id command on your host system - this will show you the uid and gid of your current user and as well all IDs from all groups the user is in.
$ id
add the definition to your docker-compose.yml
user: "${UID}:${GID}"
so your file could look like this
php: # this is my service name
user: "${UID}:${GID}" # we added this line to get a specific user / group id
image: php:7.3-fpm-alpine # this is my image
# and so on
set the values in your .env file
UID=1000
GID=1001
3a. Alternatively you can extend your ~/.bashrc file with:
export UID GID
to define it globally rather than defining it in a .env file for each project.
If this does not work for you (like on my current distro, the GID is not set by this, use following two lines:
export UID=$(id -u)
export GID=$(id -g)
Thanks #SteenSchütt for the easy solution for defining the UID / GID globally.
Now your user in the container has the id 1000 and the group is 1001 and you can set that differently for every environment.
Note: Please replace the IDs I used with the user / group IDs you found on your host system. Since I cannot know which IDs your system is using I gave some example group and user IDs.
If you don't use docker-compose or want to know more different approaches to achieve this have a read through my source of information: https://dev.to/acro5piano/specifying-user-and-group-in-docker-i2e
If the volume mount folder does not exist on your machine, docker will create it (with root user), so please ensure that it already exists and is owned by the userid / groupid you want to use.
I add an example for a dokuwiki container to explain it better:
version: '3.5'
services:
dokuwiki:
user: "${UID}" # set a specific user id so the container can write in the data dir
image: bitnami/dokuwiki:latest
ports:
- '8080:8080'
volumes:
- '/home/manuel/docker/dokuwiki/data:/bitnami/dokuwiki/'
restart: unless-stopped
expose:
- "8080"
The dokuwiki container will only be able to initialize correctly if it has write access to the host directory /home/manuel/docker/dokuwiki/data.
If on startup this directory does not exist, docker will create it for us but it will have root:root as user & group. --> Therefor the container startup will fail.
If we create the folder before starting the container
mkdir -P /home/manuel/docker/dokuwiki/data
and then check with
ls -nla /home/manuel/docker/dokuwiki/data| grep ' \.$'
which uid and gid the folder has - and check that they match the ones we put in our .env file in step 3. above.
The bad news is there's no owner/group/permission settings for volume 😢. The good news is the following trick will let you bake it into your config, so it's fully automated 🎉.
In your Dockerfile, create an empty directory in the right location and with desired settings.
This way, the directory will already be present when docker-compose mounts to the location. When the server mounts during boot (based on docker-compose), the mounting action happily leaves those permissions alone.
Dockerfile:
# setup folder before switching to user
RUN mkdir /volume_data
RUN chown postgres:postgres /volume_data
USER postgres
docker-compose.yml
volumes:
- /home/me/postgres_data:/volume_data
source
First determine the uid of the www-data user:
$ docker exec DOCKER_CONTAINER_ID id
uid=100(www-data) gid=101(www-data) groups=101(www-data)
Then, on your docker host, change the owner of the mounted directory using the uid (100 in this example):
chown -R 100 ./
Dynamic Extension
If you are using docker-compose you may as well go for it like this:
$ docker-compose exec SERVICE_NAME id
uid=100(www-data) gid=101(www-data) groups=101(www-data)
$ chown -R 100 ./
You can put that in a one-liner:
$ chown -R $(docker-compose exec SERVICE_NAME id -u) ./
The -u flag will only print the uid to stdout.
Edit: fixed casing error of CLI flag. Thanks #jcalfee314!
Adding rw to the end of the volume mount worked for me:
services:
httpd:
image: apache-image
ports:
- "80:80"
volumes:
- "./:/var/www/app:rw"
links:
- redis
command: /setupApacheRights.sh
Set user www-data for this compose service
user: "www-data:www-data"
Example:
wordpress:
depends_on:
- db
image: wordpress:5.5.3-fpm-alpine
user: "www-data:www-data"
container_name: wordpress
restart: unless-stopped
env_file:
- .env
volumes:
- ./wordpress/wp-content:/var/www/html/wp-content
- ./wordpress/wp-config-local.php:/var/www/html/wp-config.php
If your volumes create ownership issue then you might need to find your volume mount path by
cmd: docker volume ls
After that identify your volume name then inspect your mount path
cmd: docker volume inspect <volume name>
check your mount point there and go on mount point on your docker host machine.
where check ownership of volume by
cmd: ls -l
if it's suggest root:root then change owneship here to your docker user.
cmd: chown docker_user_id:docker_group_id -R volume_path
Note: you can find your docker user id & user group id by entering into your docker bash & hit "id" cmd.
cmd: docker-compose run --rm <container_name> bash
cmd: id
output: uid=102(www-data) gid=102(www-data) groups=102(www-data)
Find similar thread here. https://www.hamaraweb.com/sms/407/docker-volume-ownership-issue-errno-13-permission-denied-bgb6ld/

Resources