I'm currently running Openshift, but I am running into a problem when I try to build/deploy my custom Docker container. The container works properly on my local machine, but once it gets built in openshift and I try to deploy it, I get the error message. I believe the problem is because I am trying to run commands inside of the container as root.
(13)Permission denied: AH00058: Error retrieving pid file /run/httpd/httpd.pid
My Docker file that I am deploying looks like this -
FROM centos:7
MAINTAINER me<me#me>
RUN yum update -y
RUN yum install -y git https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
RUN yum install -y ansible && yum clean all -y
RUN git clone https://github.com/dockerFileBootstrap.git
RUN ansible-playbook "-e edit_url=andrewgarfield edit_alias=emmastone site_url=testing.com" dockerAnsible/dockerFileBootstrap.yml
RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == systemd-tmpfiles-setup.service ] || rm -f $i; done); \
rm -f /lib/systemd/system/multi-user.target.wants/*;\
rm -f /etc/systemd/system/*.wants/*;\
rm -f /lib/systemd/system/local-fs.target.wants/*; \
rm -f /lib/systemd/system/sockets.target.wants/*udev*; \
rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \
rm -f /lib/systemd/system/basic.target.wants/*;\
rm -f /lib/systemd/system/anaconda.target.wants/*;
COPY supervisord.conf /usr/etc/supervisord.conf
RUN rm -rf supervisord.conf
VOLUME [ "/sys/fs/cgroup" ]
EXPOSE 80 443
#CMD ["/usr/bin/supervisord"]
CMD ["/usr/sbin/httpd", "-D", "FOREGROUND"]
Ive run into a similar problem multiple times where it will say things like Permission Denied on file /supervisord.log or something similar.
How can I set it up so that my container doesnt run all of the commands as root? It seems to be causing all of the problems that I am having.
Openshift has strictly security policy regarding custom Docker builds.
Have a look a this OpenShift Application Platform
In particular at point 4 into the FAQ section, here quoted.
4. Why doesn't my Docker image run on OpenShift?
Security! Origin runs with the following security policy by default:
Containers run as a non-root unique user that is separate from other system users
They cannot access host resources, run privileged, or become root
They are given CPU and memory limits defined by the system administrator
Any persistent storage they access will be under a unique SELinux label, which prevents others from seeing their content
These settings are per project, so containers in different projects cannot see each other by default
Regular users can run Docker, source, and custom builds
By default, Docker builds can (and often do) run as root. You can control who can create Docker builds through the builds/docker and builds/custom policy resource.
Regular users and project admins cannot change their security quotas.
Many Docker containers expect to run as root (and therefore edit all the contents of the filesystem). The Image Author's guide gives recommendations on making your image more secure by default:
Don't run as root
Make directories you want to write to group-writable and owned by group id 0
Set the net-bind capability on your executables if they need to bind to ports <1024
Otherwise, you can see the security documentation for descriptions on how to relax these restrictions.
I hope it helps.
Although you don't have access to root, your OpenShift container, by default, is a member of the root group. You can change some dir/file permissions to avoid the Permission Denied errors.
If you're using a Dockerfile to deploy an image to OpenShift, you can add the following RUN command to your Dockerfile:
RUN chgrp -R 0 /run && chmod -R g=u /run
This will change the group for everything in the /run directory to the root group and then set the group permission on all files to be equivalent to the owner (group equals user) of the file. Essentially, any user in the root group has the same permissions as the owner for every file.
You can run docker as any user , also root (and not Openshift default build-in account UID - 1000030000 when issuing this two commands in sequence on command line oc cli tools
oc login -u system:admin -n default following with oc adm policy add-scc-to-user anyuid -z default -n projectname where projectname is name of your project inside which you assigned under your docker
Related
Any help from any source is appreciated.
Server has a Docker container with alpine, nginx, php. This container is able to write in bind mounted host directory, only when I set "chown -R nobody directory" to the host directory (nobody is a user in container).
I am using VSCode's extension "Remote - SSH" to connect to server as user ubuntu. VSCode is able to edit files in that same host directory (being used for bind mount), only when I set "chown -R ubuntu directory".
Problem: if I set "ubuntu" as owner, container can't write (using php to write), if I set "nobody" as owner, VSCode SSH can't write. I am finding a way to allow both to write without changing directory owner user again and again, or similar ease.
Image used: https://hub.docker.com/r/trafex/php-nginx
What I tried:
In Container, I added user "nobody" to group "ubuntu". On host, directory (used as mount) was set "sudo chown -R ubuntu:ubuntu directory", user "ubuntu" was already added to group "ubuntu".
VSCode did edit, container was unable to edit. (Edit: IT WORKED, I changed the directory permission for the group to allow write)
Edit: the container already created without Dockerfile also ran and maybe edited with important changes, so maybe I can't use Dockerfile or entrypoint.sh way to solve problem. Can It be achieved through running commands inside container or without creating container again? This container can be stopped.
Edit: I am wondering, in Triet Doan's answer, an option is to modify UID and GID of already created user in the container, will doing this for the user and group "nobody" can cause any problems inside container, I am wondering because probably many commands for settings already executed inside container, files are already edited by php on mounted directory & container is running for days
Edit: I found that alpine has no usermod & groupmod.
This article wrote about this problem very nicely. I would just summarize the main ideas here.
The easiest way to tackle with this permission problem is to modify UID and GID in the container to the same UID and GID that are used in the host machine.
In your case, we try to get the UID and GID of user ubuntu and use them in the container.
The author suggests 3 ways:
1. Create a new user with the same UID and GID of the host machine in entrypoint.sh.
Here’s the Dockerfile version for Ubuntu base image.
FROM ubuntu:latest
RUN apt-get update && apt-get -y install gosu
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
The entrypoint.sh was created as follows:
#!/bin/bash
USER_ID=${LOCAL_UID:-9001}
GROUP_ID=${LOCAL_GID:-9001}
echo "Starting with UID: $USER_ID, GID: $GROUP_ID"
useradd -u $USER_ID -o -m user
groupmod -g $GROUP_ID user
export HOME=/home/user
exec /usr/sbin/gosu user "$#"
Simply build the container with the docker build command.
docker build -t ubuntu-test1 .
The LOCAL_UID and LOCAL_GID can be passed to the container in the docker run command.
$ docker run -it --name ubuntu-test -e LOCAL_UID=$(id -u $USER) -e LOCAL_GID=$(id -g $USER) ubuntu-test1 /bin/bash
Starting with UID: 1001, GID: 1001
user#1291224a8029:/$ id
uid=1001(user) gid=1001(user) groups=1001(user)
We can see that the UID and GID in the container are the same as those in the host.
2. Mount the host machine’s /etc/passwd and /etc/group to a container
This is also a fine approach and simpler at a glance. One drawback of this approach is that a new user created in a container can’t access the bind-mounted file and directories because UID and GID are different from the host machine’s ones.
One must be careful to have /etc/passwd and /etc/group with read-only access, otherwise the container might access and overwrite the host machine’s /etc/passwd and /etc/group. Therefore, the author doesn't recommend this way.
$ docker run -it --name ubuntu-test --mount type=bind,source=/etc/passwd,target=/etc/passwd,readonly --mount type=bind,source=/etc/group,target=/etc/g
roup,readonly -u $(id -u $USER):$(id -g $USER) ubuntu /bin/bash
ether#903ad03490f3:/$ id
uid=1001(user) gid=1001(user) groups=1001(user)
3. Modify UID and GID with the same UID and GID of the host machine
This is mostly the same approach as No.1, but just modify the UID and GID in case a new user has been created in the container already. Assume you have a new user created in the Dockerfile, then just call these commands in either Dockerfile or entrypoint.sh.
If your username and group name were "test", then you can use usermod and groupmod commands to modify UID and GID in the container. The taken UID and GID as environment variables from the host machine will be used for this "test" user.
usermod -u $USER_ID -o -m -d <path-to-new-home> test
groupmod -g $GROUP_ID test
Problem: if I set "ubuntu" as owner, container can't write (using php to write), if I set "nobody" as owner, VSCode SSH can't write. I am finding a way to allow both to write without changing directory owner user again and again, or similar ease.
First, I'd recommend the container image should create a new username for the files inside the container, rather than reusing nobody since that user may also be used for other OS tasks that shouldn't have any special access.
Next, as Triet suggests, an entrypoint that adjusts the container's user/group to match the volume is preferred. My own version of these scripts can be found in this base image that includes a fix-perms script that makes the user id and group id of the container user match the id's of a mounted volume. In particular, the following lines of that script where $opt_u is the container username, $opt_g is the container group name, and $1 is the volume mount location:
# update the uid
if [ -n "$opt_u" ]; then
OLD_UID=$(getent passwd "${opt_u}" | cut -f3 -d:)
NEW_UID=$(stat -c "%u" "$1")
if [ "$OLD_UID" != "$NEW_UID" ]; then
echo "Changing UID of $opt_u from $OLD_UID to $NEW_UID"
usermod -u "$NEW_UID" -o "$opt_u"
if [ -n "$opt_r" ]; then
find / -xdev -user "$OLD_UID" -exec chown -h "$opt_u" {} \;
fi
fi
fi
# update the gid
if [ -n "$opt_g" ]; then
OLD_GID=$(getent group "${opt_g}" | cut -f3 -d:)
NEW_GID=$(stat -c "%g" "$1")
if [ "$OLD_GID" != "$NEW_GID" ]; then
echo "Changing GID of $opt_g from $OLD_GID to $NEW_GID"
groupmod -g "$NEW_GID" -o "$opt_g"
if [ -n "$opt_r" ]; then
find / -xdev -group "$OLD_GID" -exec chgrp -h "$opt_g" {} \;
fi
fi
fi
Then I start the container as root, and the container runs the fix-perms script from the entrypoint, followed by a command similar to:
exec gosu ${container_user} ${orig_command}
This replaces the entrypoint that's running as root with the application running as the specified user. I've got more examples of this in:
DockerCon presentation
Similar SO questions
What I tried: In Container, I added user "nobody" to group "ubuntu".
On host, directory (used as mount) was set "sudo chown -R
ubuntu:ubuntu directory", user "ubuntu" was already added to group
"ubuntu". VSCode did edit, container was unable to edit.
I'd avoid this and create a new user. Nobody is designed to be as unprivileged as possible, so there could be unintended consequences with giving it more access.
Edit: the container already created without Dockerfile also ran and
maybe edited with important changes, so maybe I can't use Dockerfile
or entrypoint.sh way to solve problem. Can It be achieved through
running commands inside container or without creating container again?
This container can be stopped.
This is a pretty big code smell in containers. They should be designed to be ephemeral. If you can't easily replace them, you're missing the ability to upgrade to a newer image, and creating a lot of state drift that you'll eventually need to cleanup. Your changes that should be preserved need to be in a volume. If there are other changes that would be lost when the container is deleted, they will be visible in docker diff and I'd recommend fixing this now rather than increasing the size of the technical debt.
Edit: I am wondering, in Triet Doan's answer, an option is to modify
UID and GID of already created user in the container, will doing this
for the user and group "nobody" can cause any problems inside
container, I am wondering because probably many commands for settings
already executed inside container, files are already edited by php on
mounted directory & container is running for days
I would build a newer image that doesn't depend on this username. Within the container, if there's data you need to preserve, it should be in a volume.
Edit: I found that alpine has no usermod & groupmod.
I use the following in the entrypoint script to install it on the fly, but the shadow package should be included in the image you build rather than doing this on the fly for every new container:
if ! type usermod >/dev/null 2>&1 || \
! type groupmod >/dev/null 2>&1; then
if type apk /dev/null 2>&1; then
echo "Warning: installing shadow, this should be included in your image"
apk add --no-cache shadow
else
echo "Commands usermod and groupmod are required."
exit 1
fi
fi
My Dockerfile extends from php:8.1-apache. The following happens while developing:
The application creates log files (as www-data, 33:33)
I create files (as the image's default user root, 0:0) within the container
These files are mounted on my host where I'm acting as user (1000:1000). Of course I'm running into file permission issues now. I'd like to update/delete files created in the container on my host and vice versa.
My current solution is to set the image's user to www-data. In that way, all created files belong to it. Then, I change its user and group id from 33 to 1000. That solves my file permission issues.
However, this leads to another problem:
I'm prepending sudo -E to the entrypoint and command. I'm doing that because they're normally running as root and my custom entrypoint requires root permissions. But in that way the stop signal stops working and the container has to be killed when I want it to stop:
~$ time docker-compose down
Stopping test_app ... done
Removing test_app ... done
Removing network test_default
real 0m10,645s
user 0m0,167s
sys 0m0,004s
Here's my Dockerfile:
FROM php:8.1-apache AS base
FROM base AS dev
COPY entrypoint.dev.sh /usr/local/bin/custom-entrypoint.sh
ARG user_id=1000
ARG group_id=1000
RUN set -xe \
# Create a home directory for www-data
&& mkdir --parents /home/www-data \
&& chown --recursive www-data:www-data /home/www-data \
# Make www-data's user and group id match my host user's ones (1000 and 1000)
&& usermod --home /home/www-data --uid $user_id www-data \
&& groupmod --gid $group_id www-data \
# Add sudo and let www-data execute it without asking for a password
&& apt-get update \
&& apt-get install --yes --no-install-recommends sudo \
&& rm --recursive --force /var/lib/apt/lists/* \
&& echo "www-data ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/www-data
USER www-data
# Run entrypoint and command as sudo, as my entrypoint does some config substitution and both normally run as root
ENTRYPOINT [ "sudo", "-E", "custom-entrypoint.sh" ]
CMD [ "sudo", "-E", "apache2-foreground" ]
Here's my custom-entrypoint.sh
#!/bin/sh
set -e
sed --in-place 's#^RemoteIPTrustedProxy.*#RemoteIPTrustedProxy '"$REMOTEIP_TRUSTED_PROXY"'#' $APACHE_CONFDIR/conf-available/remoteip.conf
exec docker-php-entrypoint "$#"
What do I need to do to make the container catch the stop signal (it is SIGWINCH for the Apache server) again? Or is there a better way to handle the file permission issues, so I don't need to run the entrypoint and command with sudo -E?
What do I need to do to make the container catch the stop signal (it is SIGWINCH for the Apache server) again?
First, get rid of sudo, if you need to be root in your container, run it as root with USER root in your Dockerfile. There's little value add to sudo in the container since it should be an environment to run one app and not a multi-user general purpose Linux host.
Or is there a better way to handle the file permission issues, so I don't need to run the entrypoint and command with sudo -E?
The pattern I go with is to have developers launch the container as root, and have the entrypoint detect the uid/gid of the mounted volume, and adjust the uid/gid of the user in the container to match that id before running gosu to drop permissions and run as that user. I've included a lot of this logic in my base image example (note the fix-perms script that tweaks the uid/gid). Another example of that pattern is in my jenkins-docker image.
You'll still need to either configure root's login shell to automatically run gosu inside the container, or remember to always pass -u www-data when you exec into your image, but now that uid/gid will match your host.
This is primarily for development. In production, you probably don't want host volumes, use named volumes instead, or at least hardcode the uid/gid of the user in the image to match the desired id on the production hosts. That means the Dockerfile would still have USER www-data but the docker-compose.yml for developers would have user: root that doesn't exist in the compose file in production. You can find a bit more on this in my DockerCon 2019 talk (video here).
You can use user namespace to map different user/group in your docker to you on the host.
For example, the group www-data/33 in the container could be the group docker-www-data/100033 on the host, you just have be in the group to access log files.
Im trying to installing jira-server via docker-image on openshift.
I pulled the image from docker-desktop for windows. Added simple dockerfile includes USER ROOT etc.
When trying to deploy the pod. I get error and pod enters to loop.
The errror is: Permission Error in diffrent locations.
Tried many times to relocate the jira-home directory but without success.
(Trying to install on closed network)
Thanks for helping!
Short Answer
The official Atlassian Images are incompatible with Kubernetes Derivatives /e.g. Openshift as they violate some key concepts.
In Openshift for example, containers are running with arbitrary user ids, which means a nameless user is executing the processes in the container.
This is a safety mechanism, prevents containers running as root and limits the risk of escaping the container gaining privileges on the cluster host.
Solution
You do need to rebuild the image from scratch.
Furthermore, the behaviour of the startup python script trying to modify file system permissions need to be removed.
Clone official Repo
https://bitbucket.org/atlassian-docker/docker-atlassian-jira/src/master/
Modify the Dockerfile and add to the UserGroup creation Step:
RUN groupadd --gid ${RUN_GID} ${RUN_GROUP} \
...
&& chown -R ${RUN_USER}:${RUN_GROUP} ${JIRA_HOME} \
# make the image compatible to run as an arbitrary uid
&& chgrp -R 0 /etc/container_id \
&& chmod -R g=u /etc/container_id \
&& chmod -R 460 /etc/container_id \
&& chgrp -R 0 ${JIRA_INSTALL_DIR} \
&& chmod -R g=u ${JIRA_INSTALL_DIR} \
&& chgrp -R 0 ${JIRA_HOME} \
&& chmod -R g=u ${JIRA_HOME}
Modify the gen_cfg function from entrypoint_helpers.py and remove the else clause at the end.
The necessary permissions have already been set in step2.
def gen_cfg(tmpl, target, user='root', group='root', mode=0o644, overwrite=True):
if not overwrite and os.path.exists(target):
logging.info(f"{target} exists; skipping.")
return
logging.info(f"Generating {target} from template {tmpl}")
cfg = jenv.get_template(tmpl).render(env)
try:
with open(target, 'w') as fd:
fd.write(cfg)
except (OSError, PermissionError):
logging.warning(f"Container not started as root. Bootstrapping skipped for '{target}'")
# else:
# set_perms(target, user, group, mode)
Rebuild the image using the --build-arg JIRA_VERSION= --build-arg ARTEFACT_NAME
Run and Enjoy
Detail inspection
Firing up the atlassian images, user root is the first to enter, doing modifications (chown...) and later dropping down to user "jira".
All these operations are not possible in openshift.
In most cases, building a new Dockerfile starting from the official image and modifing permissions of the needed files and folders before deploying to a cluster is the solution,
but to make things worse, atlassian choose to "mount" the necessary directories as VOLUME.
They have even referenced the issue in their comments.
VOLUME ["${JIRA_HOME}"] # Must be declared after setting perms
After the volume mount, the permissions can only be modified persistently at runtime.
This forces them to rebuild and set permissions after container startup, using a python function in the entrypoint_helpers.py.
This is also the place, where the container fails with several "permission denied"s.
Would be glad to issue a pull request on this, but unfortunatelly they are hosted on bitbucket.
I am trying to run a Docker container to analyze data in a Google Cloud Bucket.
I have been able to successfully mount the Bucket using gcsfuse, and I tested that I could do things like create and delete files within the Bucket.
In order to be able to install other programs (and mount the bucket), I installed Docker (and didn't use the Docker-optimized instance option). If I run Docker in interactive mode (without mounting a drive), it looks like it is working OK.
However, if I try to run Docker in interactive mode with the mounted drive (which is the gcsfuse-mounted Bucket), I get an error message:
user#instance:~/bucket-name/subfolder$ docker run -it -v /home/user/bucket-name:/mnt/bucket-name gcr.io/deepvariant-docker/deepvariant
docker: Error response from daemon: error while creating mount source path '/home/user/bucket-name': mkdir /home/user/bucket-name: file exists.
I hope that I am close to having this working: does anybody have any ideas about a relatively simple fix for this error message?
BTW, I realize that there are other ways to run DeepVariant on Google Cloud, but I am trying to makes things as similar as possible to what I am doing on AWS (plus, I may need to do some extra troubleshooting for analysis of one of my files).
Thank you very much for your help!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
FYI, this is how I mounted the Bucket:
#mount directory: https://github.com/GoogleCloudPlatform/gcsfuse/blob/master/docs/installing.md
export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`
echo "deb http://packages.cloud.google.com/apt $GCSFUSE_REPO main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo apt-get update
sudo apt-get -y install gcsfuse
#restart and mount directory: https://cloud.google.com/storage/docs/gcs-fuse
#NOTE: please make sure you are in your home directory (I encounter issues if I try to mount from /mnt)
mkdir [bucket-name]
gcsfuse -o allow_other --file-mode 777 --dir-mode 777 [bucket-name] ./[bucket-name]
and this is how I installed Docker:
#install Docker for Debian: https://docs.docker.com/install/linux/docker-ce/debian/
sudo apt-get update
sudo apt-get -y install \
apt-transport-https \
ca-certificates \
curl \
gnupg2 \
software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/debian \
$(lsb_release -cs) \
stable"
sudo apt-get update
sudo apt-get -y --allow-unauthenticated install docker-ce docker-ce-cli containerd.io
#fix Docker sock issue: https://stackoverflow.com/questions/47854463/got-permission-denied-while-trying-to-connect-to-the-docker-daemon-socket-at-uni
sudo usermod -a -G docker [user]
#have to restart after this
For anyone experiencing a similar error / issue - here is what worked for me. Steps I took:
First unmount the disk if it's already mounted: sudo umount /mounted_folder
Remount the disk using the below command, listing the credentials file to be used explicitly
sudo GOOGLE_APPLICATION_CREDENTIALS=/home/user/credentials/example-asdf21b0af7.json gcsfuse -o allow_other bucket_name /mounted_folder
Should now be connected successfully without further errors :)
NOTE: This command needs to be run everytime after restarting the computer / VM. Formatting this into fstab could probably be done so one does not need to manually execute these steps upon each restart.
EXPLANATION: What I did here was explicitly specifying the credentials via a credentials JSON for the user / service account with appropriate access (Not explained here on how to get this but should be googl-able) and referring to that json in the GOOGLE_APPLICATION_CREDENTIALS environment variable option, as suggested by this answer: https://stackoverflow.com/a/39047673/10002593. The need for this environment variable option is likely due to gcsfuse not registering the same level of access as the activated acount in gcloud config for some reason.
I think I figured out at least a partial solution to my problem:
As mentioned in this tutorial, you also need to run gcloud auth configure-docker.
I found you also needed to exit and restart your instance, but this strictly solved the original error message for this post.
I think got a strange message, but perhaps that is more about the specific container. So, I ran another test:
docker run -it -v /home/user/bucket-name:/mnt/bucket-name cwarden45/dnaseq-dependencies
This time, I got an error message about storage space on the instance (to be able to download and run the Docker container). So, I went back and created a new instance with a larger local hard drive:
1) From the Google Cloud Console, I selected "Compute Instance" and "VM instances"
2) I clicked "create instance" (similar to before)
3) I select "change" under "boot disk"
4) I set size to 300 GB instead of 10 GB (currently, towards bottom-right, under "Size (GB)")
Similar to before, I choose 8 vCPUs for the "Machine type", I selected "Allow full access to all Cloud APIs" under "Identity and API access", and I checked boxes for both "Allow HTTP traffic" and "Allow HTTPS traffic" (under "Firewall").
I am not selecting "Deploy a container image to this VM instance," which I believe is how I got Docker installed with "sudo" to be able to install gcsfuse.
I also have to call this a "parital" solution because this allows me to run the Docker container successfully in interactive mode, but the mounted bucket appears empty within Docker.
For another project, I noticed that executables could work if I installed them on the local hard drive under /opt, but not if I tried to install them on my bucket (in order to save the installation time for those programs each time). On AWS, I believe I needed to use EFS storage instead of S3 storage to do something similar, but I will keep learning more about using the Google Cloud Bucket for mounted storage / analysis.
Also, it is a different issue, but I noticed that I could fix an issue with running exectuable files from the bucket from changing the command from gcsfuse [bucket-name] ./[bucket-name] to gcsfuse --file-mode 777 --dir-mode 777 [bucket-name] ./[bucket-name] (and I changed the example code accordingly)
I noticed more recently that the set of commands above is no longer sufficient to be able to have a functional directory (I can't add or edit files, for example).
Based upon this discussion, I thought that I needed to add the -o allow_other parameter.
However, if that is all I do, I get the following error message
fusermount: option allow_other only allowed if 'user_allow_other' is set in /etc/fuse.conf
I can resolve that error message if I uncomment the corresponding line in that file. However, that still doesn't resolve having the right file permissions in the mounted directory.
So, I then tried editing my /etc/fstab file, by adding the following entry
[bucket-name] /home/[username]/[bucket-name] gcsfuse rw,allow_other,file_mode=777,dir_mode=777
I am also accordingly editing the content at the top (for whatever seems like it might help).
Also, please note that this was not a Docker-specific issue. This was necessary to essentially do anything within the bucket. Plus, I haven't actually solved this new problem.
For example, I still can't create files as root, after changing to the superuser via sudo su - (as described here)
Recently we were asked to remove unnecessary users from our images to comply with the PCI DSS requirement 2.1:
2.1 Always change vendor-supplied defaults and remove or disable unnecessary default accounts before installing a system on the network.
The rationale behind this requirement is to not multuply entities beyond necessity and reduce the number of possible attack vectors. On the one hand what the accessor asked us to do makes perfect sense. On the other hand containers already run in a constrained namespaced environment.
Let's describe it with a few examples.
The first one is the dumbest - run as root, no security enforced, etc.
$ docker run --rm -ti alpine
/ # cat /etc/passwd
root:x:0:0:root:/root:/bin/ash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
news:x:9:13:news:/usr/lib/news:/sbin/nologin
uucp:x:10:14:uucp:/var/spool/uucppublic:/sbin/nologin
operator:x:11:0:operator:/root:/bin/sh
man:x:13:15:man:/usr/man:/sbin/nologin
postmaster:x:14:12:postmaster:/var/spool/mail:/sbin/nologin
cron:x:16:16:cron:/var/spool/cron:/sbin/nologin
ftp:x:21:21::/var/lib/ftp:/sbin/nologin
sshd:x:22:22:sshd:/dev/null:/sbin/nologin
at:x:25:25:at:/var/spool/cron/atjobs:/sbin/nologin
squid:x:31:31:Squid:/var/cache/squid:/sbin/nologin
xfs:x:33:33:X Font Server:/etc/X11/fs:/sbin/nologin
games:x:35:35:games:/usr/games:/sbin/nologin
postgres:x:70:70::/var/lib/postgresql:/bin/sh
cyrus:x:85:12::/usr/cyrus:/sbin/nologin
vpopmail:x:89:89::/var/vpopmail:/sbin/nologin
ntp:x:123:123:NTP:/var/empty:/sbin/nologin
smmsp:x:209:209:smmsp:/var/spool/mqueue:/sbin/nologin
guest:x:405:100:guest:/dev/null:/sbin/nologin
nobody:x:65534:65534:nobody:/:/sbin/nologin
/ # su -s /bin/sh nobody
~ $ whoami
nobody
~ $ su
su: must be suid to work properly
This example shows that we can easily switch from root to a regular user inside the container meaning that anyone who could break into the host could jump into container as root and switch to the user the main container's process is running on behalf of. Apparently you can't su running as a regular user. And of course you are free to setcap in order to give more freedom to non-priviliged users.
$ docker run --rm -ti alpine
/ # apk add libcap; setcap cap_net_raw=+ep /bin/busybox
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
(1/1) Installing libcap (2.25-r1)
Executing busybox-1.28.4-r1.trigger
OK: 4 MiB in 14 packages
/ # getcap /bin/busybox
/bin/busybox = cap_net_raw+ep
But let's add some more flags (which are sane defaults for all of our containers in fact):
$ docker run --rm -ti --cap-drop all --security-opt no-new-privileges alpine
/ # su -s /bin/sh nobody
su: can't set groups: Operation not permitted
/ # apk add libcap
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
(1/1) Installing libcap (2.25-r1)
Executing busybox-1.28.4-r1.trigger
ERROR: busybox-1.28.4-r1.trigger: script exited with error 127
OK: 4 MiB in 14 packages
/ # setcap cap_net_raw=+ep /bin/busybox
unable to set CAP_SETFCAP effective capability: Operation not permitted
Let's now do this kind of thing. Can we run a container from the arbitrary user, as someone who is not present in /etc/passwd:
$ docker run --rm -ti --cap-drop all --security-opt no-new-privileges --user 60000:60000 alpine
/ $ whoami
whoami: unknown uid 60000
/ $ id
uid=60000 gid=60000
Ok, this means that effectively we can run a container with any uid:gid pair no matter if it is present inside a container actually or not.
Now what will happen if we completely remove the information about users from the container:
$ docker build -f- .<<EOF
FROM alpine
RUN rm -f /etc/shadow /etc/group /etc/passwd
EOF
Sending build context to Docker daemon 2.607kB
Step 1/2 : FROM alpine
---> 196d12cf6ab1
Step 2/2 : RUN rm -f /etc/shadow /etc/group /etc/passwd
---> Using cache
---> 7c22df16e0dd
Successfully built 7c22df16e0dd
$ docker run --rm -ti --user 60000:60000 7c22df16e0dd
/ $ touch /etc/passwd /etc/group
touch: /etc/passwd: Permission denied
touch: /etc/group: Permission denied
/ $ adduser lol
adduser: permission denied (are you root?)
/ $ whoami
whoami: unknown uid 60000
/ $ id
uid=60000 gid=60000
And yet if --user 60000:60000 is removed we will identify ourselves as an effective root:
$ docker run --rm -ti 7c22df16e0dd
/ # touch /etc/passwd /etc/group
/ # adduser lol
passwd: unknown uid 0
/ # cat /etc/passwd
lol:x:1000:1000:Linux User,,,:/home/lol:/bin/sh
/ # su lol
/ $ whoami
lol
Now let's check if there is anything that can be relevant to users present in default passwd and group files. Apparently there is no processes. But as for files:
$ docker run --rm -ti alpine:3.8
/ # for user in `cat /etc/passwd | grep -v -E ^root | cut -d ':' -f 1`; do find / -xdev -user $user;done
/ #
$ docker run --rm -ti debian:9
root#4fac719bb234:/# for user in `cat /etc/passwd | grep -v -E ^root | cut -d ':' -f 1`; do find / -xdev -user $user;done
root#4fac719bb234:/#
$ docker run --rm -ti centos:7
[root#fa1242222f1e /]# for user in `cat /etc/passwd | grep -v -E ^root | cut -d ':' -f 1`; do echo "=== $user ==="; find / -xdev -user $user;done
=== bin ===
=== daemon ===
=== adm ===
=== lp ===
=== sync ===
=== shutdown ===
=== halt ===
=== mail ===
=== operator ===
=== games ===
=== ftp ===
=== nobody ===
=== systemd-network ===
/run/systemd/netif
/run/systemd/netif/leases
/run/systemd/netif/links
=== dbus ===
CentOS container is the only one who has got files owned by a non-root user.
Some thoughts and observations:
Running a container with --read-only --cap-drop all --security-opt no-new-privileges and --user with uid:gid higher than 0 seems enough to protect it from gaining additional privileges.
The uid:gid pair can be any pair of numbers.
Linux Standard Base lists only root as a required user with bin and daemon marked as legacy and a bunch of optional users none of which make any sense in case of base container images.
The questions are:
Why all those users (especially thoise beyond listed in LSB) exist in base images and what risk do they add?
Can we remove all but root and leave creating additional users as an excercise to application developers/package creators/et al.?
Does it make sense to remove the /etc/passwd and /etc/group entirely bearing in mind that containers run as root implicitly by default no matter if these files are present in the system or not and we are still able to use any uid:gid pair where uid > 0 and gid > 0 which will effectively mean running as non-root?
This requirement makes little sense to me in the context of containers. The users listed in the /etc/passwd file are a mapping of uid to username. They do not have any login credentials, that's controlled by /etc/shadow. And there is no login or similar daemon running inside the container unless you explicitly start one yourself. Having the uid to name mapping gives the advantage that directory listing commands will show a user name instead of a uid. It also assists with application install scripts that run commands as different users by username for security.
As an aside, if you want security I would recommend having a user defined inside the container to run your application, assuming your application can be run by a non-root user. While root inside the container has fewer capabilities than a normal root user (to prevent the ability to escape the container), a non-root user is even further constrained. And whatever user you run commands as inside the container should be defined in your /etc/passwd file. That said, even if your application is running as root inside the container, that user is so limited with reduced capabilities and isolated to a namespace that has little ability to do damage. The risks are, to the best of my knowledge, theoretical, or require additional security holes to exploit. Running applications as a non-root user is therefore an added layer of security should a future exploit be discovered.
Overall, this feels like a requirement created by someone skilled in locking down Linux servers and VM's, but failing to understand why a policy that makes sense on a multi-user host does not make sense in a container environment. I would challenge them to show what exploits are made possible by /etc/passwd entries that doesn't also involve treating a container like a VM with things like adding /etc/shadow entries and an sshd daemon.
To answer the specific questions:
Why all those users (especially thoise beyond listed in LSB) exist in base images and what risk do they add?
These come from a tar file of a minimal OS filesystem for the base image distribution. I suspect they are assumed to exist and are there for package installers that depend on these users. There's no real value add to removing them, but a real risk to breaking installers.
Can we remove all but root and leave creating additional users as an excercise to application developers/package creators/et al.?
Could you? Yes. But to me this is an example of security introducing potential failures into an existing system without adding value. I have seen some applications break when the uid does not exist in the /etc/passwd file, so I don't recommend running commands as an unlisted uid without lots of testing.
Does it make sense to remove the /etc/passwd and /etc/group entirely bearing in mind that containers run as root implicitly by default no matter if these files are present in the system or not and we are still able to use any uid:gid pair where uid > 0 and gid > 0 which will effectively mean running as non-root?
I'm fairly certain this will cause things to break. You can try this in your environment, but would be in an unsupported configuration. Note that there may be some binaries installed in the filesystem with suid/sgid functionality configured and giving users access to these uid/gid's could introduce security vulnerabilities that would be solved by maintaining a mapping of the currently used uid/gid's and running your application as a new uid/gid. The built in functionality to do this is, you guessed it, /etc/passwd.