I want to give a docker container to my students such that they are able to conduct experiments. I thought I use the following dockerfile:
FROM jupyter/datascience-notebook:latest
ADD ./config.json /home/jovyan/.jupyter/jupyter_notebook_config.json
ADD ./books /home/jovyan/work
So, the standard container will include a few notebooks I have created and stored in the books folder. I then build and run this container locally with
#!/usr/bin/env bash
docker build -t aaa .
docker run --rm -p "8888:8888" -v $(pwd)/books:/home/joyvan/work aaa
I build the container aaa and share again the folder books with it (although books has been copied into the image at compile time). I now open the container on port 8888. I can edit the files in the /home/joyvan/work folder but this stuff is not getting transported back to the host. Something goes terrible wrong. Is it because I add the files during the docker build and then share them again in the -v ...?
I have played with various options. I have added the local user to the users group. I do chown on all files in books. All my files show up as root:root in the container. I am then joyvan in the container and do not have write access to those files. How would I make sure the files are owned by joyvan?
EDIT:
Some other elements :
tom#thomas-ThinkPad-T450s:~/babynames$ docker exec -it cranky_poincare /bin/bash
jovyan#5607ac2bcaae:~$ id
jovyan uid=1000(jovyan) gid=100(users) groups=100(users)
jovyan#5607ac2bcaae:~$ cd work/
jovyan#5607ac2bcaae:~/work$ ls
test.txt text2.txt
jovyan#5607ac2bcaae:~/work$ ls -ltr
total 4
-rw-rw-r-- 1 root root 5 Dec 12 19:05 test.txt
-rw-rw-r-- 1 root root 0 Dec 12 19:22 text2.txt
on the host:
tom#thomas-ThinkPad-T450s:~/babynames/books$ ls -ltr
total 4
-rw-rw-r-- 1 tom users 5 Dez 12 20:05 test.txt
-rw-rw-r-- 1 tom users 0 Dez 12 20:22 text2.txt
tom#thomas-ThinkPad-T450s:~/babynames/books$ id tom
uid=1001(tom) gid=1001(tom) groups=1001(tom),27(sudo),100(users),129(docker)
You can try:
FROM jupyter/datascience-notebook:latest
ADD ./config.json /home/jovyan/.jupyter/jupyter_notebook_config.json
ADD ./books /home/jovyan/work
RUN chown joyvan /books
if that user already exists, but with RUN you can execute all commands in your docker file.
Related
I have been working with a docker container for a few months now and was unaware of the fact that everything I was creating (folders, files) were created under the root user of my container. Now I want to reclaim ownership over all of these files so that I can have the permissions to move or write into them while I am outside of the container.
To make it a bit more concrete/clear, I have a local user named johndoe, and a local folder under the path of /home/johndoe/pythoncodes which is owned by johndoe. I mount this local folder to my docker container when I run the command
docker run -v /home/johndoe/pythoncodes:/home/johndoe/pythoncodes ...
Then when inside my container, I created a folder at /home/johndoe/pythoncodes/ProjectRepo. ProjectRepo is now owned by root from the container and so when I leave the container and go back to being the johndoe user, I no longer have the permissions to do anything with this folder (e.g. if I try to run git init I get a permission error that doesn't allow the creation of the .git folder.
I have seen answers on how to create a container that logs me in as my local user and have gotten this to work as well by using the adduser flag, but this only seem helpful for creating new files and doesn't help me with all of these files that have been already created as root.
but this only seem helpful for creating new files and doesn't help me with all of these files that have been already created as root
You could directly use chown from within the docker container to change the ownership of these bind mounts. But for this to work you will need to mount two folders which contain the username and password information for your user, /etc/passwd and /etc/group (below, :ro means 'read-only').
$ docker run -idt -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro --name try ubuntu:16.04 /bin/bash
$ docker exec -it try mkdir -p /tmp/abc/newfolder
$ cd abc
$ ls -alh
total 12K
drwxr-xr-x 3 atg atg 4.0K Jul 7 16:43 .
drwxr-xr-x 60 atg atg 4.0K Jul 7 16:42 ..
drwxr-xr-x 2 root root 4.0K Jul 7 16:43 newfolder
$ sudo chown -R atg:atg .
[sudo] password for atg:
$ ls -alh
total 12K
drwxr-xr-x 3 atg atg 4.0K Jul 7 16:43 .
drwxr-xr-x 60 atg atg 4.0K Jul 7 16:42 ..
drwxr-xr-x 2 atg atg 4.0K Jul 7 16:43 newfolder
I have a volume which uses bind to share a local directory. Sometimes this directory doesn't exist and everything goes to shit. How can I tell docker-compose to look for the directory and use it if it exists or to continue without said volume if errors?
Volume example:
- type: bind
read_only: true
source: /srv/share/
target: /srv/share/
How can I tell docker-compose to look for the directory and use it if it exists or to continue without said volume if errors?
As far I am aware you can't do conditional logic to mount a volume, but i am getting around it in a project of mine, like this:
version: "2.1"
services:
elixir:
image: elixir:alpine
volumes:
- ${VOLUME_SOURCE:-/dev/null}:${VOLUME_TARGET:-/.devnull}:ro
Here I am using /dev/null as the fallback, but in my real project I just use an empty file to do the mapping.
This ${VOLUME_SOURCE:-/dev/null} is how bash works with default values for variables not set, and docker compose supports them.
Testing it without setting the env vars
$ sudo docker-compose run --rm elixir sh
/ # ls -al /.devnull
crw-rw-rw- 1 root root 1, 3 May 21 12:27 /.devnull
Testing it with the env vars set
Creating the .env file:
$ printf "VOLUME_SOURCE=./testing \nVOLUME_TARGET=/testing\n" > .env && cat .env
VOLUME_SOURCE=./testing
VOLUME_TARGET=/testing
Creating the volume for test purposes:
$ mkdir testing && touch testing/test.txt && ls -al testing
total 8
drwxr-xr-x 2 exadra37 exadra37 4096 May 22 13:12 .
drwxr-xr-x 3 exadra37 exadra37 4096 May 22 13:12 ..
-rw-r--r-- 1 exadra37 exadra37 0 May 22 13:12 test.txt
Running the container:
$ sudo docker-compose run --rm elixir sh
/ # ls -al /testing/
total 8
drwxr-xr-x 2 1000 1000 4096 May 22 12:01 .
drwxr-xr-x 1 root root 4096 May 22 12:07 ..
-rw-r--r-- 1 1000 1000 0 May 22 12:01 test.txt
/ #
I don't think there is a way to do that with the docker-compose syntax easily yet, here is how I went, but the container would not start at all if there is no volume.
check the launch command with a docker inspect on the unpatched container
change your command with something like this (here using egorive/seafile-mc:8.0.7-rpi on a raspberry pi, where the data is on an external disk that might not always be plugged):
volumes:
- '/data/seafile-data:/shared:Z'
command: sh -c "( [ -f /shared/.docker-volume-check ] || ( echo volume not mounted, not starting; sleep 60; exit 1 )) && exec /sbin/my_init -- /scripts/start.py"
restart: always
touch .docker-volume-check in the root of your volume
That way, you have a restartable container, that would fail and wait if the volume is not mounted. It also supports volume in a generic way: for instance, when you just create a new container that has not initialized its volume yet with a first setup, it will still boot as you're checking a file you created.
I have an NFS share with the following properties:
Mounted on my host on /nfs/external_disk
Owner user is test_user with UID 1234
Group is test_group with GID 2222
Permissions is 750
I have a small Dockerfile with the following content
ARG tag=lts
from jenkins/jenkins:${tag}
user root
# Create a new user and new group that matches what is on the host.
ARG username=test_user
ARG groupname=test_group
ARG uid=1234
ARG gid=2222
RUN groupadd -g ${gid} ${groupname} && \
mkdir -p /users && \
useradd -l -m -u ${uid} -g ${groupname} -s /bin/bash -d /users/${username} ${username}
user ${username}
After building the image (named custom_jenkins), and when I run the following command, the container is started properly and I see the original Jenkins homer stuff now copied to the share.
docker run -td --rm -v /nfs/external_disk:/var/jenkins_home custom_jenkins
However if I want to mount a sub-directory of the NFS share, say ${NFS_SHARE}/jenkins_home, then I get an error:
docker run -td --rm -v /nfs/external_disk/jenkins_home:/var/jenkins_home custom_jenkins
docker: Error response from daemon: error while creating mount source path '/nfs/external_disk/jenkins_home': mkdir /nfs/external_disk/jenkins_home: permission denied.
Now even if I attempt to create the sub-directory myself before starting the container, I still get the same error. Even when I set the permissions of the sub-directory to be 777.
Note that I am running as test_user which has the same UID/GID as in the container and it actually owns the NFS share.
I have a feeling that when docker attempts to create a sub-directory, it attempts to create it as some different user (e.g. the "docker" user) which causes it to fail while creating the folder since it has no access inside the share.
Can anyone help? thanks in advance.
I tried to reproduce. Works just fine for me. Perhaps I am missing some constraint. Hope this helps anyway. Note at step 6 the owner and the group for the file that I created from the container. This might answer one of your questions.
Step 1: I created a NFS share somewhere in my LAN
Step 2: I mounted the share on the machine that's running the docker engine
sudo mount 192.168.0.xxx:/i-data/b4024d5b/nfs/NFS /mnt/nsa320/
neo#neo-desktop:nsa320$ mount | grep NFS
192.168.0.xxx:/i-data/b4024d5b/nfs/NFS on /mnt/nsa320 type nfs (rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0.xxx,mountvers=3,mountport=3775,mountproto=udp,local_lock=none,addr=192.168.0.xxx)
Step 3: I created some sample files and a sub-directory:
neo#neo-desktop:nsa320$ ls -la /mnt/nsa320/
total 12
drwxrwxrwx 3 root root 4096 Jul 21 22:54 .
drwxr-xr-x 3 root root 4096 Jul 21 22:41 ..
-rw-r--r-- 1 neo neo 0 Jul 21 22:45 dummyFile
-rw-r--r-- 1 root root 0 Jul 21 22:53 fileCreatedFromContainer << THIS WAS CREATED FROM A CONTAINER THAT WAS NOT LAUNCHED WITH THE --user OPTION
drwxr-xr-x 2 neo neo 4096 Jul 21 22:54 subfolder
Step 4: Launched a dummy container and mounted the sub-directory (1000 is the UID of the user neo in the my OS):
docker run -d -v /mnt/nsa320/subfolder:/var/externalMount --user 1000 alpine tail -f /dev/null
Step 5: Connected in container to check the mount(I can read and write in the subfolder located on the NFS)
neo#neo-desktop:nsa320$ docker exec -ti ded1dc79773e sh
/ $ ls /var/externalMount/
fileInSubfolder
/ $ touch /var/externalMount/fileInSubfolderCreatedFromContainer
Step 6: Back on the host, to whom does the file that I created from the container belongs to:
neo#neo-desktop:nsa320$ ls -la /mnt/nsa320/subfolder/
total 8
drwxr-xr-x 2 neo neo 4096 Jul 21 23:23 .
drwxrwxrwx 3 root root 4096 Jul 21 22:54 ..
-rw-r--r-- 1 neo neo 0 Jul 21 22:54 fileInSubfolder
-rw-r--r-- 1 neo root 0 Jul 21 23:23 fileInSubfolderCreatedFromContainer
Maybe off-topic: whoami executed in the container returns just the UID:
$ whoami
whoami: unknown uid 1000
In my docker-compose (3.7) file I have something like
- ./a/src/:/var/www/html/
- ./a/config/local.php.ini:/usr/local/etc/php/conf.d/local.ini
as can be found for example in this example.
Whenever I change something on host in the ./a/src directory or in the container in /var/www/html/ it gets changed on the other side as expected. They are the same as they should be.
Not so with the file. It gets copied (I guess) to the container. But then, if I change local.php.ini on the host or /usr/local/etc/php/conf.d/local.ini the other one remains the same.
Is that the expected behavior? If yes, why and is it possible to change it so, both files are the same like with the directory
Note: This is not a duplicate of How to mount a single file in a volume. I get my file as file not as directory or such. I nevertheless tried it with absolute directories with ${PWD} as suggested there but that changed nothing.
Docker version 19.03.1, build 74b1e89
docker-compose version 1.24.1, build 4667896b
Host and container systems are Debian.
Please go through this.
I guess it might have caused because of this reason.
If you edit the file using text editor like vim, when you save the
file it does not save the file directly, rather it creates a new file
and copies it into place. This breaks the bind-mount, which is based
on inode. Since saving the file effectively changes the inode, changes
will not propagate into the container. Restarting the container will
pick up the new inode and changes will got reflected.
Here is an example, explaining what I mean:
# Create a file on host and list it contents and its inode number
-------------------
$ echo 'abc' > /root/file.txt
$ cat /root/file.txt
abc
$ ls -ltrhi /root/
total 4K
1623230 -rw-r--r-- 1 root root 4 Aug 23 17:44 file.txt
$
# Run an alpine container by mounting this file.txt
---------------------
$ docker run -itd -v /root/file.txt:/var/tmp/file.txt alpine sh
d59a2ad308d2de7dfbcf042439b295b27370e4014be94bc339f1c5c880bf205f
$
# Check file contents of file.txt and its inode number inside alpine container
$ docker exec -it d59a2ad308d2 sh
/ # cat /var/tmp/file.txt
abc
/ # ls -ltrhi /var/tmp/
total 4K
1623230 -rw-r--r-- 1 root root 4 Aug 23 17:44 file.txt
/ #
## NOTE: The inode number of file.txt is same here 1623230 on host and inside the container.
# Edit the file.txt inside alpine container using some text editor like vi
--------------------------
/ # vi /var/tmp/file.txt
/ # ls -ltrhi /var/tmp/
total 4K
1623230 -rw-r--r-- 1 root root 5 Aug 23 17:46 file.txt
/ # cat /var/tmp/file.txt
abcd
/ #
# Check content of file.txt on host, it will be the same as the one inside container since the inode number of file.txt inside container and on host is still same 1623230
--------------------------
$ cat /root/file.txt <<=== ran it on host
abcd
# Now edit content of file.txt on host and check its inode number.
$ vi file.txt
$ ls -ltrhi /root/
total 4K
862510 -rw-r--r-- 1 root root 6 Aug 23 17:47 file.txt
$ cat file.txt
abcde
$
## NOTE: the inode number of file.txt on host is changed to 862510 after editing the file using vi editor.
# Check content of file.txt inside alpine container and list it inode number
----------------------------
$ docker exec -it d59a2ad308d2 sh
/ # ls -ltrhi /var/tmp/
total 4K
1623230 -rw-r--r-- 0 root root 5 Aug 23 17:46 file.txt
/ # cat /var/tmp/file.txt
abcd
/ #
## NOTE: inode number here is the old one and doesn't match with the one on the host and hence the content of file.txt also doesn't match.
# Restart alpine container
---------------------------
$ docker restart d59a2ad308d2
d59a2ad308d2
$ docker exec -it d59a2ad308d2 sh
/ # cat /var/tmp/file.txt
abcde
/ # ls -ltrhi /var/tmp/
total 4K
862510 -rw-r--r-- 1 root root 6 Aug 23 17:47 file.txt
/ # [node1] (local) root#192.168.0.38 ~
$
## NOTE: After restarting container, the inode of file.txt is matching with the one on host and so the file contents also match.
I also highly recommend you to go through this link, it has more info.
Hope this helps.
How can I control which user owns the files I copy in and out of a container?
The docker cp command says this about file ownership:
The cp command behaves like the Unix cp -a command in that directories are copied recursively with permissions preserved if possible. Ownership is set to the user and primary group at the destination. For example, files copied to a container are created with UID:GID of the root user. Files copied to the local machine are created with the UID:GID of the user which invoked the docker cp command. However, if you specify the -a option, docker cp sets the ownership to the user and primary group at the source.
It says that files copied to a container are created as the root user, but that's not what I see. I create two files owned by user id 1005 and 1006. Those owners are translated into the container's user namespace. The -a option seems to make no difference when I copy the file into a container.
$ sudo chown 1005:1005 test.txt
$ ls -l test.txt
-rw-r--r-- 1 1005 1005 29 Oct 6 12:43 test.txt
$ docker volume create sandbox1
sandbox1
$ docker run --name run1 -vsandbox1:/data alpine echo OK
OK
$ docker cp test.txt run1:/data/test1005.txt
$ docker cp -a test.txt run1:/data/test1005a.txt
$ sudo chown 1006:1006 test.txt
$ docker cp test.txt run1:/data/test1006.txt
$ docker cp -a test.txt run1:/data/test1006a.txt
$ docker run --rm -vsandbox1:/data alpine ls -l /data
total 16
-rw-r--r-- 1 1005 1005 29 Oct 6 19:43 test1005.txt
-rw-r--r-- 1 1005 1005 29 Oct 6 19:43 test1005a.txt
-rw-r--r-- 1 1006 1006 29 Oct 6 19:43 test1006.txt
-rw-r--r-- 1 1006 1006 29 Oct 6 19:43 test1006a.txt
When I copy files out of the container, they are always owned by me. Again, the -a option seems to do nothing.
$ docker run --rm -vsandbox1:/data alpine cp /data/test1006.txt /data/test1007.txt
$ docker run --rm -vsandbox1:/data alpine chown 1007:1007 /data/test1007.txt
$ docker cp run1:/data/test1006.txt .
$ docker cp run1:/data/test1007.txt .
$ docker cp -a run1:/data/test1006.txt test1006a.txt
$ docker cp -a run1:/data/test1007.txt test1007a.txt
$ ls -l test*.txt
-rw-r--r-- 1 don don 29 Oct 6 12:43 test1006a.txt
-rw-r--r-- 1 don don 29 Oct 6 12:43 test1006.txt
-rw-r--r-- 1 don don 29 Oct 6 12:47 test1007a.txt
-rw-r--r-- 1 don don 29 Oct 6 12:47 test1007.txt
-rw-r--r-- 1 1006 1006 29 Oct 6 12:43 test.txt
$
You can also change the ownership by logging in as root user into the container :
docker exec -it --user root <container-id> /bin/bash
chown -R <username>:<groupname> <folder/file>
In addition to #Don Kirkby's answer, let me provide a similar example in bash/shell script for the case that you want to copy something into a container while applying different ownership and permissions than those of the original file.
Let's create a new container from a small image that will keep running by itself:
docker run -d --name nginx nginx:alpine
Now wel'll create a new file which is owned by the current user and has default permissions:
touch foo.bar
ls -ahl foo.bar
>> -rw-rw-r-- 1 my-user my-group 0 Sep 21 16:45 foo.bar
Copying this file into the container will set ownership and group to the UID of my user and preserve the permissions:
docker cp foo.bar nginx:/foo.bar
docker exec nginx sh -c 'ls -ahl /foo.bar'
>> -rw-rw-r-- 1 4098 4098 0 Sep 21 14:45 /foo.bar
Using a little tar work-around, however, I can change the ownership and permissions that are applied inside of the container.
tar -cf - foo.bar --mode u=+r,g=-rwx,o=-rwx --owner root --group root | docker cp - nginx:/
docker exec nginx sh -c 'ls -ahl /foo.bar'
>> -r-------- 1 root root 0 Sep 21 14:45 /foo.bar
tar options explained:
c creates a new archive instead of unpacking one.
f - will write to stdout instead of a file.
foo.bar is the input file to be packed.
--mode specifies the permissions for the target. Similar to chown, they can be given in symbolic notation or as an octal number.
--owner sets the new owner of the file.
--group sets the new group of the file.
docker cp - reads the file that is to be copied into the container from stdin.
This approach is useful when a file needs to be copied into a created container before it starts, such that docker exec is not an option (which can only operate on running containers).
Just a one-liner (similar to #ramu's answer), using root to make the call:
docker exec -u 0 -it <container-id> chown node:node /home/node/myfile
In order to get complete control of file ownership, I used the tar stream feature of docker cp:
If - is specified for either the SRC_PATH or DEST_PATH, you can also stream a tar archive from STDIN or to STDOUT.
I launch the docker cp process, then stream a tar file to or from the process. As the tar entries go past, I can adjust the ownership and permissions however I like.
Here's a simple example in Python that copies all the files from /outputs in the sandbox1 container to the current directory, excludes the current directory so its permissions don't get changed, and forces all the files to have read/write permissions for the user.
from subprocess import Popen, PIPE, CalledProcessError
import tarfile
def main():
export_args = ['sudo', 'docker', 'cp', 'sandbox1:/outputs/.', '-']
exporter = Popen(export_args, stdout=PIPE)
tar_file = tarfile.open(fileobj=exporter.stdout, mode='r|')
tar_file.extractall('.', members=exclude_root(tar_file))
exporter.wait()
if exporter.returncode:
raise CalledProcessError(exporter.returncode, export_args)
def exclude_root(tarinfos):
print('\nOutputs:')
for tarinfo in tarinfos:
if tarinfo.name != '.':
assert tarinfo.name.startswith('./'), tarinfo.name
print(tarinfo.name[2:])
tarinfo.mode |= 0o600
yield tarinfo
main()