I want to start using Docker for my Rails development, so I'm trying to put together a skeleton I can use for all my apps.
However, I've run into an issue with Docker volumes and permissions.
I want to bind-mount the app's directory into the container, so that any changes get propagated to the container without the need to re-build it.
But if I define it as a volume in my docker-compose.yml, I can't chown the directory anymore. I need the directory and all its contents to be owned by the app user in order for Passenger to work correctly.
I read that it's not possible to chown volumes.
Do you know of any workarounds?
You could try to run the chown instead by CMD. Like:
CMD chown -R app:app /home/app/webapp && /sbin/my_init
RUN statements are only executed during built time of your image. But there you do not have mounted volumes, yet.
CMD instead is executed during runtime of the container when the volumes are mounted already. So that would have the effect that you want.
I use a hacky solution to manage this problem for my development environments. To use on development environments only!
The images I use for development environments contain a script that looks like this:
#!/bin/sh
# In usr/local/bin/change-dev-id
# Change the "dev" user UID and GID
# Retrieve new ids to apply
NEWUID=$1
NEWGID=$1
if [ $# -eq 2 ]
then
NEWGID=$2
elif [ $# -ne 1 ]
then
echo "Usage: change-dev-id NEWUID [NEWGID]"
echo "If NEWGID is not provided, its value will be the same as NEWUID"
exit 1
fi
# Retrieve old ids
OLDUID=`id -u dev`
OLDGID=`id -g dev`
# Change the user ids
usermod -u ${NEWUID} dev
groupmod -g ${NEWGID} dev
# Change the files ownership
find / -not \( -path /proc -prune \) -user ${OLDUID} -exec chown -h ${NEWUID} {} \;
find / -not \( -path /proc -prune \) -group ${OLDGID} -exec chgrp -h ${NEWGID} {} \;
echo "UID and GID changed from ${OLDUID}:${OLDGID} to ${NEWUID}:${NEWGID} for \"dev\""
exit 0
In the Dockerfile of my base image, I add it and make it executable:
# Add a script to modify the dev user UID / GID
COPY change-dev-id /usr/local/bin/change-dev-id
RUN chmod +x /usr/local/bin/change-dev-id
Then, instead of changing the owner of the mounted folder, I change the ID of the container's user to match the ID of my user on the host machine:
# In the Dockerfile of the project's development environment, change the ID of
# the user that must own the files in the volume so that it match the ID of
# the user on the host
RUN change-dev-id 1234
This is very hacky but it can be very convenient. I can own the files of the project on my machine while the user in the container has the correct permissions too.
You can update the code of the script to use the username you want (mine is always "dev") or modify it to pass the username as an argument.
Related
Any help from any source is appreciated.
Server has a Docker container with alpine, nginx, php. This container is able to write in bind mounted host directory, only when I set "chown -R nobody directory" to the host directory (nobody is a user in container).
I am using VSCode's extension "Remote - SSH" to connect to server as user ubuntu. VSCode is able to edit files in that same host directory (being used for bind mount), only when I set "chown -R ubuntu directory".
Problem: if I set "ubuntu" as owner, container can't write (using php to write), if I set "nobody" as owner, VSCode SSH can't write. I am finding a way to allow both to write without changing directory owner user again and again, or similar ease.
Image used: https://hub.docker.com/r/trafex/php-nginx
What I tried:
In Container, I added user "nobody" to group "ubuntu". On host, directory (used as mount) was set "sudo chown -R ubuntu:ubuntu directory", user "ubuntu" was already added to group "ubuntu".
VSCode did edit, container was unable to edit. (Edit: IT WORKED, I changed the directory permission for the group to allow write)
Edit: the container already created without Dockerfile also ran and maybe edited with important changes, so maybe I can't use Dockerfile or entrypoint.sh way to solve problem. Can It be achieved through running commands inside container or without creating container again? This container can be stopped.
Edit: I am wondering, in Triet Doan's answer, an option is to modify UID and GID of already created user in the container, will doing this for the user and group "nobody" can cause any problems inside container, I am wondering because probably many commands for settings already executed inside container, files are already edited by php on mounted directory & container is running for days
Edit: I found that alpine has no usermod & groupmod.
This article wrote about this problem very nicely. I would just summarize the main ideas here.
The easiest way to tackle with this permission problem is to modify UID and GID in the container to the same UID and GID that are used in the host machine.
In your case, we try to get the UID and GID of user ubuntu and use them in the container.
The author suggests 3 ways:
1. Create a new user with the same UID and GID of the host machine in entrypoint.sh.
Here’s the Dockerfile version for Ubuntu base image.
FROM ubuntu:latest
RUN apt-get update && apt-get -y install gosu
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
The entrypoint.sh was created as follows:
#!/bin/bash
USER_ID=${LOCAL_UID:-9001}
GROUP_ID=${LOCAL_GID:-9001}
echo "Starting with UID: $USER_ID, GID: $GROUP_ID"
useradd -u $USER_ID -o -m user
groupmod -g $GROUP_ID user
export HOME=/home/user
exec /usr/sbin/gosu user "$#"
Simply build the container with the docker build command.
docker build -t ubuntu-test1 .
The LOCAL_UID and LOCAL_GID can be passed to the container in the docker run command.
$ docker run -it --name ubuntu-test -e LOCAL_UID=$(id -u $USER) -e LOCAL_GID=$(id -g $USER) ubuntu-test1 /bin/bash
Starting with UID: 1001, GID: 1001
user#1291224a8029:/$ id
uid=1001(user) gid=1001(user) groups=1001(user)
We can see that the UID and GID in the container are the same as those in the host.
2. Mount the host machine’s /etc/passwd and /etc/group to a container
This is also a fine approach and simpler at a glance. One drawback of this approach is that a new user created in a container can’t access the bind-mounted file and directories because UID and GID are different from the host machine’s ones.
One must be careful to have /etc/passwd and /etc/group with read-only access, otherwise the container might access and overwrite the host machine’s /etc/passwd and /etc/group. Therefore, the author doesn't recommend this way.
$ docker run -it --name ubuntu-test --mount type=bind,source=/etc/passwd,target=/etc/passwd,readonly --mount type=bind,source=/etc/group,target=/etc/g
roup,readonly -u $(id -u $USER):$(id -g $USER) ubuntu /bin/bash
ether#903ad03490f3:/$ id
uid=1001(user) gid=1001(user) groups=1001(user)
3. Modify UID and GID with the same UID and GID of the host machine
This is mostly the same approach as No.1, but just modify the UID and GID in case a new user has been created in the container already. Assume you have a new user created in the Dockerfile, then just call these commands in either Dockerfile or entrypoint.sh.
If your username and group name were "test", then you can use usermod and groupmod commands to modify UID and GID in the container. The taken UID and GID as environment variables from the host machine will be used for this "test" user.
usermod -u $USER_ID -o -m -d <path-to-new-home> test
groupmod -g $GROUP_ID test
Problem: if I set "ubuntu" as owner, container can't write (using php to write), if I set "nobody" as owner, VSCode SSH can't write. I am finding a way to allow both to write without changing directory owner user again and again, or similar ease.
First, I'd recommend the container image should create a new username for the files inside the container, rather than reusing nobody since that user may also be used for other OS tasks that shouldn't have any special access.
Next, as Triet suggests, an entrypoint that adjusts the container's user/group to match the volume is preferred. My own version of these scripts can be found in this base image that includes a fix-perms script that makes the user id and group id of the container user match the id's of a mounted volume. In particular, the following lines of that script where $opt_u is the container username, $opt_g is the container group name, and $1 is the volume mount location:
# update the uid
if [ -n "$opt_u" ]; then
OLD_UID=$(getent passwd "${opt_u}" | cut -f3 -d:)
NEW_UID=$(stat -c "%u" "$1")
if [ "$OLD_UID" != "$NEW_UID" ]; then
echo "Changing UID of $opt_u from $OLD_UID to $NEW_UID"
usermod -u "$NEW_UID" -o "$opt_u"
if [ -n "$opt_r" ]; then
find / -xdev -user "$OLD_UID" -exec chown -h "$opt_u" {} \;
fi
fi
fi
# update the gid
if [ -n "$opt_g" ]; then
OLD_GID=$(getent group "${opt_g}" | cut -f3 -d:)
NEW_GID=$(stat -c "%g" "$1")
if [ "$OLD_GID" != "$NEW_GID" ]; then
echo "Changing GID of $opt_g from $OLD_GID to $NEW_GID"
groupmod -g "$NEW_GID" -o "$opt_g"
if [ -n "$opt_r" ]; then
find / -xdev -group "$OLD_GID" -exec chgrp -h "$opt_g" {} \;
fi
fi
fi
Then I start the container as root, and the container runs the fix-perms script from the entrypoint, followed by a command similar to:
exec gosu ${container_user} ${orig_command}
This replaces the entrypoint that's running as root with the application running as the specified user. I've got more examples of this in:
DockerCon presentation
Similar SO questions
What I tried: In Container, I added user "nobody" to group "ubuntu".
On host, directory (used as mount) was set "sudo chown -R
ubuntu:ubuntu directory", user "ubuntu" was already added to group
"ubuntu". VSCode did edit, container was unable to edit.
I'd avoid this and create a new user. Nobody is designed to be as unprivileged as possible, so there could be unintended consequences with giving it more access.
Edit: the container already created without Dockerfile also ran and
maybe edited with important changes, so maybe I can't use Dockerfile
or entrypoint.sh way to solve problem. Can It be achieved through
running commands inside container or without creating container again?
This container can be stopped.
This is a pretty big code smell in containers. They should be designed to be ephemeral. If you can't easily replace them, you're missing the ability to upgrade to a newer image, and creating a lot of state drift that you'll eventually need to cleanup. Your changes that should be preserved need to be in a volume. If there are other changes that would be lost when the container is deleted, they will be visible in docker diff and I'd recommend fixing this now rather than increasing the size of the technical debt.
Edit: I am wondering, in Triet Doan's answer, an option is to modify
UID and GID of already created user in the container, will doing this
for the user and group "nobody" can cause any problems inside
container, I am wondering because probably many commands for settings
already executed inside container, files are already edited by php on
mounted directory & container is running for days
I would build a newer image that doesn't depend on this username. Within the container, if there's data you need to preserve, it should be in a volume.
Edit: I found that alpine has no usermod & groupmod.
I use the following in the entrypoint script to install it on the fly, but the shadow package should be included in the image you build rather than doing this on the fly for every new container:
if ! type usermod >/dev/null 2>&1 || \
! type groupmod >/dev/null 2>&1; then
if type apk /dev/null 2>&1; then
echo "Warning: installing shadow, this should be included in your image"
apk add --no-cache shadow
else
echo "Commands usermod and groupmod are required."
exit 1
fi
fi
I want to do a binding (mount volume) between the jupyterLab and the VM. The only problem is the permissions, the folder that I create (/home/jovyan/work) always have a root permissions. I cannot create anything inside. I tried many solutions: In a docker file I tried this solution:
FROM my_image
RUN if [[ -d "$HOME/toto" ]] ; then echo OK ; else mkdir "$HOME/toto" ; fi
RUN chmod 777 $HOME/toto
==> always I got no permissions on the mounted folder
ARG NB_DIR=/tmp/lab_share
RUN mkdir "$NB_DIR"
RUN chown :100 "$NB_DIR"
RUN chmod g+rws "$NB_DIR"
RUN apt-get install acl
RUN setfacl -d -m g::rwx "$NB_DIR"
USER root
==> The problem here is the setfacl is not recognized in the container, I tried to install it just before, always not acceptable.
I tried to add a GRANT_SUDO in the jupyterHub service
extraEnv:
GRANT_SUDO: "yes"
==> The problem here is the extraEnv is not recognized
I tried to create a method in the jupyterHub_config file, just after the binding code:
notebook_dir = os.environ.get('DOCKER_NOTEBOOK_DIR') or '/home/jovyan/work'
c.DockerSpawner.notebook_dir = notebook_dir
c.DockerSpawner.volumes = { "/tmp/{username}" :notebook_dir}
c.DockerSpawner.remove_containers = True
##### CREATE HOOKER
def create_dir_hook(spawner):
username = spawner.user.name # get the username
logger.info(f"### USERNAME {username}")
volume_path = os.path.join('/tmp', username)
logger.info(f"### USERNAME {volume_path}")
if not os.path.exists(volume_path):
# create a directory with umask 0755
# hub and container user must have the same UID to be writeable
# still readable by other users on the system
logger.info(f"FOLDER FOR USER {username} not existing")
logger.info(f"CREATING A FOLDER IN {volume_path}")
os.mkdir(volume_path, 0o777)
# now do whatever you think your user needs
# ...
logger.info("coucou")
# attach the hook function to the spawner
c.Spawner.pre_spawn_hook = create_dir_hook
In this solutions the compilator don't read all the if bloc, even the else bloc. I found in the docker doc, that is a famous issue in docker, but I didn't find it's solution. Really I need your help please, if you have any solution I will be appreciate. Thanks a lot.
My Dockerfile extends from php:8.1-apache. The following happens while developing:
The application creates log files (as www-data, 33:33)
I create files (as the image's default user root, 0:0) within the container
These files are mounted on my host where I'm acting as user (1000:1000). Of course I'm running into file permission issues now. I'd like to update/delete files created in the container on my host and vice versa.
My current solution is to set the image's user to www-data. In that way, all created files belong to it. Then, I change its user and group id from 33 to 1000. That solves my file permission issues.
However, this leads to another problem:
I'm prepending sudo -E to the entrypoint and command. I'm doing that because they're normally running as root and my custom entrypoint requires root permissions. But in that way the stop signal stops working and the container has to be killed when I want it to stop:
~$ time docker-compose down
Stopping test_app ... done
Removing test_app ... done
Removing network test_default
real 0m10,645s
user 0m0,167s
sys 0m0,004s
Here's my Dockerfile:
FROM php:8.1-apache AS base
FROM base AS dev
COPY entrypoint.dev.sh /usr/local/bin/custom-entrypoint.sh
ARG user_id=1000
ARG group_id=1000
RUN set -xe \
# Create a home directory for www-data
&& mkdir --parents /home/www-data \
&& chown --recursive www-data:www-data /home/www-data \
# Make www-data's user and group id match my host user's ones (1000 and 1000)
&& usermod --home /home/www-data --uid $user_id www-data \
&& groupmod --gid $group_id www-data \
# Add sudo and let www-data execute it without asking for a password
&& apt-get update \
&& apt-get install --yes --no-install-recommends sudo \
&& rm --recursive --force /var/lib/apt/lists/* \
&& echo "www-data ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/www-data
USER www-data
# Run entrypoint and command as sudo, as my entrypoint does some config substitution and both normally run as root
ENTRYPOINT [ "sudo", "-E", "custom-entrypoint.sh" ]
CMD [ "sudo", "-E", "apache2-foreground" ]
Here's my custom-entrypoint.sh
#!/bin/sh
set -e
sed --in-place 's#^RemoteIPTrustedProxy.*#RemoteIPTrustedProxy '"$REMOTEIP_TRUSTED_PROXY"'#' $APACHE_CONFDIR/conf-available/remoteip.conf
exec docker-php-entrypoint "$#"
What do I need to do to make the container catch the stop signal (it is SIGWINCH for the Apache server) again? Or is there a better way to handle the file permission issues, so I don't need to run the entrypoint and command with sudo -E?
What do I need to do to make the container catch the stop signal (it is SIGWINCH for the Apache server) again?
First, get rid of sudo, if you need to be root in your container, run it as root with USER root in your Dockerfile. There's little value add to sudo in the container since it should be an environment to run one app and not a multi-user general purpose Linux host.
Or is there a better way to handle the file permission issues, so I don't need to run the entrypoint and command with sudo -E?
The pattern I go with is to have developers launch the container as root, and have the entrypoint detect the uid/gid of the mounted volume, and adjust the uid/gid of the user in the container to match that id before running gosu to drop permissions and run as that user. I've included a lot of this logic in my base image example (note the fix-perms script that tweaks the uid/gid). Another example of that pattern is in my jenkins-docker image.
You'll still need to either configure root's login shell to automatically run gosu inside the container, or remember to always pass -u www-data when you exec into your image, but now that uid/gid will match your host.
This is primarily for development. In production, you probably don't want host volumes, use named volumes instead, or at least hardcode the uid/gid of the user in the image to match the desired id on the production hosts. That means the Dockerfile would still have USER www-data but the docker-compose.yml for developers would have user: root that doesn't exist in the compose file in production. You can find a bit more on this in my DockerCon 2019 talk (video here).
You can use user namespace to map different user/group in your docker to you on the host.
For example, the group www-data/33 in the container could be the group docker-www-data/100033 on the host, you just have be in the group to access log files.
How do I run docker run and docker-compose up/run commands so that the process inside the docker is run by a user with the same uuid as my local user?
I need to do this so that any files generated by an "inside-docker" process would have ownership permissions of my local user.
To replicate:
Use the alpine:3.9 container, mount in a volume for the file to be written and create the file. Assume my current username is user.
mkdir output_dir #Create an output directory
docker run -it --rm --volume "/path/to/output_dir:/tmp" alpine:3.9 touch /tmp/file.txt
ls -la output_dir/file.txt
Will give the output:
-rw-r--r-- 1 root root 0 Feb 7 19:51 /path/to/output_dir/file.txt
This means I need to sudo chown user:user /path/to/output_dir/file.txt to have access as my current user on my own file system.
How do I do this without this extra step?
Idea that comes to mind:
Add a Docker Entrypoint which will create a user inside the container with the same uuid as my local user and execute any code as that user.
docker-entrypoint.sh
#!/bin/sh
TEMP_UID="${TEMP_UID:-1000}"
set -ux
useradd -s /bin/false --no-create-home -u ${TEMP_UID} temp
#su-exec is an executable which makes it easy to run a process as a specific user.
exec su-exec temp $#
The problem with this is I will have to inject the TEMP_UID=<user_id> as an environment variable at every docker run command or include in my docker-compose.yml file for every docker-compose up/run command. If Docker has an internal variable that keeps track of the uuid of the user that ran it, I would just use that. But I can't seem to find such an internal variable.
Any help would be greatly appreciated!
I think the answer is as simple as
docker run --user ${UID} -it --rm --volume "/path/to/output_dir:/tmp" alpine:3.9 touch /tmp/file.txt
Note I injected --user ${UID} into your example command.
Many of the current options require a change outside of the container to pass in the current user, or rely on variables that may not exist in all environments. My preferred solution, since the goal is to fix file permissions on mounted volumes, is to start the entrypoint as root with a script that changes the container userid to match that of the volume mount's userid. And then the end of the entrypoint launches the application with a exec gosu $app_user_name "$#" to switch from root to that application user that was modified inside of the container.
Scripts to do this are in my base image repo. Take note of the fix-perms script which includes two sections like the following (one for uid and another for gid):
# update the uid
if [ -n "$opt_u" ]; then
OLD_UID=$(getent passwd "${opt_u}" | cut -f3 -d:)
NEW_UID=$(stat -c "%u" "$1")
if [ "$OLD_UID" != "$NEW_UID" ]; then
echo "Changing UID of $opt_u from $OLD_UID to $NEW_UID"
usermod -u "$NEW_UID" -o "$opt_u"
if [ -n "$opt_r" ]; then
find / -xdev -user "$OLD_UID" -exec chown -h "$opt_u" {} \;
fi
fi
fi
The OLD_UID value is from the userid in the image, and NEW_UID is from the volume mount. When those don't match, the usermod command is run, followed by a recursive chown command to fix any files with the old uid/gid.
Note that in production, where user id's on the host can be standardized, I match the host user id to that of the image if a volume is needed, allowing me to run the entrypoint as that user instead of root. The entrypoint checks the current userid and skips the fix-perms script and gosu command if it is not root.
I run docker-compose up -d and then ssh into the container. I can load the site via localhost just fine but when I try to edit the source code on my local it does not let me due to permission errors. This is the ls -la output on container vs local:
Container:
Local:
My dockerfile has the chown command:
My local user is called pwm. I tried running chown -R pwm:pwm ../app from host at which point I am able to edit files but then I get laravel permission denied errors. Then I need to runchown -R www-data:www-data ../app again to fix it.
How can I fix this?
For a development environment, my go-to solution for this is to setup an entrypoint script inside the container that starts as root, changes the user inside the container to match that of the file/directory owner from a volume mount (which will be your user on the host), and then switch to that user to run the app. I've got an example of doing this along with the scripts needed to implement this in your own container in my base image repo: https://github.com/sudo-bmitch/docker-base
In there, the fix-perms script does the heavy lifting, including code like the following:
# update the uid
if [ -n "$opt_u" ]; then
OLD_UID=$(getent passwd "${opt_u}" | cut -f3 -d:)
NEW_UID=$(stat -c "%u" "$1")
if [ "$OLD_UID" != "$NEW_UID" ]; then
echo "Changing UID of $opt_u from $OLD_UID to $NEW_UID"
usermod -u "$NEW_UID" -o "$opt_u"
if [ -n "$opt_r" ]; then
find / -xdev -user "$OLD_UID" -exec chown -h "$opt_u" {} \;
fi
fi
fi
That script is run as root inside the container on startup. The last step of the entrypoints that I run will call something like:
exec gosu ${app_user} "$#"
which runs the container command as the application user as the new pid 1 executable.