Getting permission denied even as root inside the docker container - docker

Question

Connecting to running docker container as a root still gets Operation not permitted error when trying to apt-get update, yet I can still see sensitive file like /etc/passwd. Below are my configurations and also the error message from apt-get update. My host operating system is Ubuntu 18.04.3. My docker version is Docker version 19.03.5, build 633a0ea838
I create a container with the following Dockerfile
FROM python:3.8-slim-buster
RUN useradd -ms /bin/bash andrej
WORKDIR /home/andrej
COPY . /home/andrej/
RUN apt-get update && \
apt-get install -y gcc && \
pip install -r requirements.txt && \
apt-get remove -y gcc && apt-get -y autoremove
RUN chown andrej:andrej pycurl && \
chmod 0744 pycurl
USER andrej
ENTRYPOINT ["uwsgi"]
CMD ["--ini", "uwsgi.ini"]
starting it with docker compose looking like this:
version: "3.3"
services:
andrej-cv:
build: ./andrej_cv
container_name: andrej-cv
restart: always
security_opt:
- no-new-privileges
expose:
- 5000
healthcheck:
test: ./pycurl --host=127.0.0.1 --port=5050 --uri=/health_check
interval: 1m30s
timeout: 10s
retries: 3
My docker daemon config:
{
"icc": false,
"userns-remap": "default",
"log-driver": "syslog",
"live-restore": true,
"userland-proxy": false,
"no-new-privileges": true
}
I connect to the container with following command (as root):
docker exec -it -u root <container_hash> /bin/bash but when I try to update I got the following:
root#ed984abff684:/home/andrej# apt-get update
E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (1: Operation not permitted)
E: seteuid 100 failed - seteuid (1: Operation not permitted)
E: setgroups 0 failed - setgroups (1: Operation not permitted)
Hit:1 http://deb.debian.org/debian buster InRelease
Ign:2 http://deb.debian.org/debian buster-updates InRelease
Err:4 http://deb.debian.org/debian buster-updates Release
Could not open file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster-updates_Release - open (13: Permission denied) [IP: 151.101.36.204 80]
Hit:3 http://security-cdn.debian.org/debian-security buster/updates InRelease
rm: cannot remove '/var/cache/apt/archives/partial/*.deb': Permission denied
Reading package lists... Done
W: chown to _apt:root of directory /var/lib/apt/lists/partial failed - SetupAPTPartialDirectory (1: Operation not permitted)
W: chmod 0700 of directory /var/lib/apt/lists/partial failed - SetupAPTPartialDirectory (1: Operation not permitted)
W: chown to _apt:root of directory /var/lib/apt/lists/auxfiles failed - SetupAPTPartialDirectory (1: Operation not permitted)
W: chmod 0700 of directory /var/lib/apt/lists/auxfiles failed - SetupAPTPartialDirectory (1: Operation not permitted)
E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (1: Operation not permitted)
E: seteuid 100 failed - seteuid (1: Operation not permitted)
W: Download is performed unsandboxed as root as file '/var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster_InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
E: setgroups 0 failed - setgroups (1: Operation not permitted)
W: Problem unlinking the file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster_InRelease - PrepareFiles (13: Permission denied)
W: Problem unlinking the file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster-updates_InRelease - PrepareFiles (13: Permission denied)
W: Problem unlinking the file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster-updates_Release - PrepareFiles (13: Permission denied)
E: The repository 'http://deb.debian.org/debian buster-updates Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: Problem unlinking the file /var/lib/apt/lists/partial/security.debian.org_debian-security_dists_buster_updates_InRelease - PrepareFiles (13: Permission denied)
In the container /etc/subuid and /etc/subgid look like this (both):
andrej:100000:65536
On the host /etc/subuid and /etc/subgid look like this (both):
andrej:100000:65536
dockremap:165536:65536
Apparmor is running on Ubuntu host with following status (only docker-default profile):
andrej#machine:/etc/apparmor.d$ sudo aa-status
apparmor module is loaded.
38 profiles are loaded.
36 profiles are in enforce mode.
/sbin/dhclient
/snap/core/8268/usr/lib/snapd/snap-confine
/snap/core/8268/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
/usr/bin/evince
/usr/bin/evince-previewer
/usr/bin/evince-previewer//sanitized_helper
/usr/bin/evince-thumbnailer
/usr/bin/evince//sanitized_helper
/usr/bin/man
/usr/lib/NetworkManager/nm-dhcp-client.action
/usr/lib/NetworkManager/nm-dhcp-helper
/usr/lib/connman/scripts/dhclient-script
/usr/lib/cups/backend/cups-pdf
/usr/lib/snapd/snap-confine
/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
/usr/sbin/cups-browsed
/usr/sbin/cupsd
/usr/sbin/cupsd//third_party
/usr/sbin/ippusbxd
/usr/sbin/tcpdump
docker-default
libreoffice-senddoc
libreoffice-soffice//gpg
libreoffice-xpdfimport
man_filter
man_groff
snap-update-ns.core
snap-update-ns.gnome-calculator
snap-update-ns.gnome-characters
snap-update-ns.gnome-logs
snap-update-ns.gnome-system-monitor
snap.core.hook.configure
snap.gnome-calculator.gnome-calculator
snap.gnome-characters.gnome-characters
snap.gnome-logs.gnome-logs
snap.gnome-system-monitor.gnome-system-monitor
2 profiles are in complain mode.
libreoffice-oopslash
libreoffice-soffice
17 processes have profiles defined.
14 processes are in enforce mode.
docker-default (1101)
docker-default (1102)
docker-default (1111)
docker-default (1600)
docker-default (1728)
docker-default (1729)
docker-default (1730)
docker-default (1731)
docker-default (1732)
docker-default (1798)
docker-default (1799)
docker-default (1800)
docker-default (1801)
docker-default (1802)
0 processes are in complain mode.
3 processes are unconfined but have a profile defined.
/sbin/dhclient (491)
/usr/sbin/cups-browsed (431)
/usr/sbin/cupsd (402)
Selinux seems to be disabled as there is not /etc/selinux/config file and getenfoce and sestatus command are not available.
Also su andrej command run as root (where andrej is unprivileged user in the container) errors out with su: cannot set groups: Operation not permitted

Answer 1

Not sure about Docker, but in kubernetes in runc container for me helps:
Get root access to container
List all containers
minikube ssh docker container ls
Connect to your container (use your container id from previous command instead of 44a7ad70d45b):
minikube ssh "docker container exec -it -u 0 44a7ad70d45b /bin/bash"
As root inside container:
root#mycontainer:/# apt-config dump | grep Sandbox::User
APT::Sandbox::User "_apt";
root#mycontainer:/# cat <<EOF > /etc/apt/apt.conf.d/sandbox-disable
APT::Sandbox::User "root";
EOF
Be sure, that result is valid:
root#mycontainer:/# apt-config dump | grep Sandbox::User
APT::Sandbox::User "root";
apt update and see
Get:1 http://archive.ubuntu.com/ubuntu hirsute InRelease [269 kB]
Get:2 http://apt.postgresql.org/pub/repos/apt hirsute-pgdg InRelease [16.7 kB]
...

Answer 2

I had exatly the same issues when running
Ubuntu-16.04-based container in rootless Podman with Manjaro as the host system.
TL;DR: try to rebuild the image. That helped in my case.
The issue is likely that Docker cannot map the /var/lib/apt/lists/package directory's owner (_apt) UID to host's UID namespace. This might happen if the /etc/sub{u,g}id is modified after the image is pulled/built.
This is only a guess but the reason might be that Docker performs the UID map first for the image and then modifies /etc/sub{u,g}id resulting in different UID map rules -> Docker cannot map the user inside the container.
You can verify this by running docker inspect <image name> and checking the directories in "LowerDir" part. In one of those there should exist a directory var/lib/apt/lists/package with UID outside of the range specified for dockremap in /etc/sub{u,g}id. The exact command for podman was podman image inspect <image name> --format '{{.GraphDriver.Data.LowerDir}}' but the CLI APIs of Podman and Docker should be identical so the same command should work with Docker also.
E.g. I had an entry tlammi:100000:65536 in /etc/sub{u,g}id but /var/lib/apt/lists/package was owned by UID 165538 in host side which is outside of range [100000, 100000 + 65536).

Answer 3

We run into the same issue with rootless podman.
We changed the subuid/subgid range of the user. This means one would need to fix the files stored with the old ranges or just delete the temporary files from the container storage directory:
$ podman info|grep GraphRoot
GraphRoot: /opt/tmp/container_graphroot/.local/share/containers/storage
$ rm -rf /opt/tmp/container_graphroot/.local/share/containers/storage/*

Answer 4

The docker container you describe is working as designed. With the default capabilities.
Fast test if you have a missing capability.
Run your container with Full/All container capabilities (--privileged)
More explanations about capabilities from Docker documentation
The Linux kernel is able to break down the privileges of the root user
into distinct units referred to as capabilities. For example, the
CAP_CHOWN capability is what allows the root user to make arbitrary
changes to file UIDs and GIDs. The CAP_DAC_OVERRIDE capability allows
the root user to bypass kernel permission checks on file read, write
and execute operations. Almost all of the special powers associated
with the Linux root user are broken down into individual capabilities.
This breaking down of root privileges into granular capabilities
allows you to:
Remove individual capabilities from the root user account, making it less powerful/dangerous.
Add privileges to non-root users at a very granular level.
Capabilities apply to both files and threads. File capabilities allow
users to execute programs with higher privileges. This is similar to
the way the setuid bit works. Thread capabilities keep track of the
current state of capabilities in running programs.
The Linux kernel lets you set capability bounding sets that impose
limits on the capabilities that a file/thread can gain.
Docker imposes certain limitations that make working with capabilities
much simpler. For example, file capabilities are stored within a
file’s extended attributes, and extended attributes are stripped out
when Docker images are built. This means you will not normally have to
concern yourself too much with file capabilities in containers.
It is of course possible to get file capabilities into containers at runtime, however this is not recommended.
In an environment without file based capabilities, it’s not possible
for applications to escalate their privileges beyond the bounding set
(a set beyond which capabilities cannot grow). Docker sets the
bounding set before starting a container. You can use Docker commands
to add or remove capabilities to or from the bounding set.
By default, Docker drops all capabilities except those needed, using a
whitelist approach.

Answer 5

This is not an answer yet (as of 2021-05-10) but my current research on this issue so far. Hopefully, it would give others hints about where to look further. Maybe I'll come back to edit this post to make it a real answer.
As far as I see, the "issue" is caused by the use of the security option no-new-privileges. Note that it is specified in OP's docker-compose file and the Docker daemon's configuration file.
Here is its description in the Docker's doc:
--security-opt="no-new-privileges:true" Disable container processes from gaining new privileges
...
If you want to prevent your container processes from gaining additional privileges, you can execute the following command:
$ docker run --security-opt no-new-privileges -it centos bash
no-new-privileges is a flag that was added to Linux kernel since 3.5. Here is its doc:
The no_new_privs bit (since Linux 3.5) is a new, generic mechanism to make it safe for a process to modify its execution environment in a manner that persists across execve. Any task can set no_new_privs. Once the bit is set, it is inherited across fork, clone, and execve and cannot be unset. With no_new_privs set, execve() promises not to grant the privilege to do anything that could not have been done without the execve call. For example, the setuid and setgid bits will no longer change the uid or gid; file capabilities will not add to the permitted set, and LSMs will not relax constraints after execve.
Notice the last sentence of "the setuid and setgid bits will no longer change the uid or gid". This may be why you would see the following error messages:
E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (1: Operation not permitted)
E: seteuid 100 failed - seteuid (1: Operation not permitted)
E: setgroups 0 failed - setgroups (1: Operation not permitted)
I found an article that talks about it with examples clearly: Running Docker application containers more securely.
My current thoughts:
I don't call the failed "apt-get update" an "issue" or a "problem" because that should be an intentional behavior for security consideration. In other words, it's a good thing.
Because the quoted kernel doc says "Once the bit is set, it ... cannot be unset", I believe you won't be able to "fix" it in the existing containers.
I don't think removing no-new-privileges is the right solution. At least it's not right before you fully discuss it with your team.
Alternatively, create a container without no-new-privileges for testing purpose only.

Answer 6

NOTE: Only if you have docker and docker-compose installed
If initially you had not been running as root and rebuilt the image, run a prune
docker system prune -f
docker-compose up
This then makes sure you're running on a fresh build