Docker run throws error when user namespaces are used with SELinux on - docker

I am trying to get Docker user namespaces to work with SELinux enabled on Centos 7.5. However, I get this error everytime:
docker run -itd --name temp -p 80:80 httpd
1a83588651b407e547881e15190b6d39692a7a2cf2df73dcaf4f37730ebdca65
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:301: running exec setns process for init caused \"exit status 40\"": unknown.
This does not happen if I turn of SELinux.
Here is my /etc/docker/daemon.json:
{
"userns-remap": "dockerspace",
"hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:2376"],
"tls": true,
"tlscacert": "/etc/pki/tls/certs/docker-ca.pem",
"tlscert": "/etc/pki/tls/certs/docker-cert.pem",
"tlskey": "/etc/pki/tls/private/docker-key.pem",
"tlsverify": true,
"selinux-enabled": true
}
uname -a output:
Linux atlantis.newtarget.net 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 16:29:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
cat /proc/cmdline output:
BOOT_IMAGE=/vmlinuz-3.10.0-862.9.1.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=en_US.UTF-8 namespace.unpriv_enable=1 user_namespace.enable=1
Any help is greatly appreciated. Thanks.

You've got a bit more going on than I do when I was getting that error, but here goes!
Based on your cat /proc/cmdline output it looks like you have already done:
sudo grubby --args="namespace.unpriv_enable=1" --update-kernel=/boot/vmlinuz-$(uname -r)
You might need to restart for this to take effect (if you haven't already).
You also need to make sure the value in /proc/sys/user/max_user_namespaces is at least greater than 0:
echo 12345 > /proc/sys/user/max_user_namespaces
These settings along with configuring [/etc/subuid, /etc/subgid, /etc/docker/daemon.json] correctly it worked for me with selinux enabled. docker documentation on user namespacing, how to configure the above files (must be done manually for each on centos/rhel)

Related

process_linux.go:458: setting cgroup config for procHooks process caused: can't load program: operation not permitted: unknown (Docker))

I'm trying to deploy a docker container to perform some testings with gitlab runners but when I'm doing the docker-compose up command I get the following output:
admin#runners-test:~/runner-test$ sudo docker-compose up -d
Starting gitlab-runner ... error
ERROR: for gitlab-runner Cannot start service gitlab-runner: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: can't load program: operation not permitted: unknown
ERROR: for gitlab-runner Cannot start service gitlab-runner: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: can't load program: operation not permitted: unknown
ERROR: Encountered errors while bringing up the project.
Abd this is the output of journalctl:
Sep 23 07:23:24 runners-test dockerd[231]: time="2022-09-23T07:23:24.560275437Z" level=error msg="stream copy error: reading from a closed fifo"
Sep 23 07:23:24 runners-test dockerd[231]: time="2022-09-23T07:23:24.577270402Z" level=error msg="stream copy error: reading from a closed fifo"
Sep 23 07:23:24 runners-test dockerd[231]: time="2022-09-23T07:23:24.675282811Z" level=error msg="0cd3bbb779a947012c9059921f092b569eb088bb2fe0bf99a8ae3266ec43abbd cleanup: failed to delete container from containerd: no such container"
Sep 23 07:23:24 runners-test dockerd[231]: time="2022-09-23T07:23:24.675625496Z" level=error msg="Handler for POST /v1.25/containers/0cd3bbb779a947012c9059921f092b569eb088bb2fe0bf99a8ae3266ec43abbd/start returned error: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: can't load program: operation not permitted: unknown"
And the docker-compose file:
version: '3'
services:
gitlab-runner:
container_name: gitlab-runner
image: 'gitlab/gitlab-runner:latest'
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./config:/etc/gitlab-runner
restart:
unless-stopped
This error occurs with any container I try to lift, even doing a simple docker run hello-world...
I also take a look to this post on proxmox forum but I don't know how to add kernel command line parameters for a lxc container, I'v been trying editing the /etc/default/grub file but update-grub command doesn't work as I want inside the lxc, I get the following output:
admin#runners-test:/$ sudo update-grub
[sudo] password for admin:
/usr/sbin/grub-probe: error: failed to get canonical path of `/dev/mapper/pve-vm--1010118--disk--0'.
I'm really stuck with this, so any kind of help would be welcome :) Thanks!
I referenced many other links when finally discovering doing a (on CentOS) update did the trick. So:
yum update -y
reboot
After the reboot, the system came back online and I proceeded to run my docker container, in this case a docker registry:
[root#server ~]# docker run -d --name registry registry:2
c40941bf42c853709bcca05bad4e8914df1f4932a355607f37b55f7e0ed01e60
[root#server ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
c40941bf42c8 registry:2 "/entrypoint.sh /e..." 3 seconds ago Up 2 seconds 5000/tcp
The docker container booted up fine.
So, also here are reference pages I checked out, and I did not have to change anything in /boot/boot.txt as this one references, but is good info:
https://my-take-on.tech/2021/05/07/fix-docker-cgroup-errors-after-systemd-248-update/
and this one, which after I read it, it had me finally trying to do a yum update for fix:
docker-compose throws errors by starting mariadb for an private nextcloud installation

docker - start failed because /etc/fstab not found

I'm using Window Linux Subsystem (Debian stretch). Followed the instruction on Docker website, I installed docker-ce, but it cannot start. Here is the info:
$ sudo service docker start
grep: /etc/fstab: No such file or directory
[ ok ] Starting Docker: docker.
$ sudo service docker status
[FAIL] Docker is not running ... failed!
What should I do with /etc/fstab not found?
to fix fstab
touch /etc/fstab
if you run dockerd, it will give you the failed message:
INFO[2022-01-27T17:55:14.100489400+07:00] Loading containers: start.
WARN[2022-01-27T17:55:14.191666800+07:00] Running iptables --wait -t nat -L -n failed with message: `iptables v1.8.2 (nf_tables): CHAIN_ADD failed (No such file or directory): chain PREROUTING`, error: exit status 4
INFO[2022-01-27T17:55:14.493716300+07:00] stopping event stream following graceful shutdown error="<nil>" module=libcontainerd namespace=moby
INFO[2022-01-27T17:55:14.494906600+07:00] stopping event stream following graceful shutdown error="context canceled" module=libcontainerd namespace=plugins.moby
INFO[2022-01-27T17:55:14.495048400+07:00] stopping healthcheck following graceful shutdown module=libcontainerd
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables --wait -t nat -N DOCKER: iptables v1.8.2 (nf_tables): CHAIN_ADD failed (No such file or directory): chain PREROUTING
(exit status 4)
that is Debian nat issue, fix it with:
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
now you can start the service again
you can follow this to make it start on startup https://askubuntu.com/a/1356147/138352
Edited:
if the issue with IP table still persisted try to set WSL version to 2, run the command from Windows shell:
wsl --set-version <distribution name> 2
the distribution list can be found with command wsl -l
I was getting the same error. Apparently on my install of WSL with Debian, I didn't have an etc/fstab file. Surprisingly, just creating the file via 'touch' worked:
sudo touch /etc/fstab
Perhaps a good signal https://learn.microsoft.com/en-us/windows/wsl/release-notes#build-17093
WSL now processes the /etc/fstab file during instance start [GH 2636].
For anybody stumbling across this years later like me, Docker doesn't work inside WSL.
But you can use Docker for Windows and WSL2 to run native containers inside your Linux Distro and the install and config is quite painless https://learn.microsoft.com/en-us/windows/wsl/tutorials/wsl-containers

docker - driver "devicemapper" failed to remove root filesystem after process in container killed

I am using Docker version 17.06.0-ce on Redhat with devicemapper storage. I am launching a container running a long-running service. The master process inside the container sometimes dies for whatever reason. I get the following error message.
/bin/bash: line 1: 40 Killed python -u scripts/server.py start go
I would like the container to exit and to be restarted by docker. However docker never exits. If I do it manually I get the following error:
Error response from daemon: driver "devicemapper" failed to remove root filesystem.
After googling, I tried a bunch of things:
docker rm -f <container>
rm -f <pth to mount>
umount <pth to mount>
All result in device is busy. The only remedy right now is to reboot the host system which is obviously not a long-term solution.
Any ideas?
I had the same problem and the solution was a real surprise.
So here is the error om docker rm:
$ docker rm 08d51aad0e74
Error response from daemon: driver "devicemapper" failed to remove root filesystem for 08d51aad0e74060f54bba36268386fe991eff74570e7ee29b7c4d74047d809aa: remove /var/lib/docker/devicemapper/mnt/670cdbd30a3627ae4801044d32a423284b540c5057002dd010186c69b6cc7eea: device or resource busy
Then I did the following (basically go through all processes and look for docker in mountinfo):
$ grep docker /proc/*/mountinfo | grep 958722d105f8586978361409c9d70aff17c0af3a1970cb3c2fb7908fe5a310ac
/proc/20416/mountinfo:629 574 253:15 / /var/lib/docker/devicemapper/mnt/958722d105f8586978361409c9d70aff17c0af3a1970cb3c2fb7908fe5a310ac rw,relatime shared:288 - xfs /dev/mapper/docker-253:5-786536-958722d105f8586978361409c9d70aff17c0af3a1970cb3c2fb7908fe5a310ac rw,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota
This got be the PID of the offending process keeping it busy - 20416 (the item after /proc/)
So I did a ps -p and to my surprise find:
[devops#dp01app5030 SeGrid]$ ps -p 20416
PID TTY TIME CMD
20416 ? 00:00:19 ntpd
A true WTF moment. So I pair problem solved with Google and found this:
Then found this https://github.com/docker/for-linux/issues/124
Turns out I had to restart ntp daemon and that fixed the issue!!!

Packer docker build exits code 137 when running runit cookbook

I'm trying to use Packer to build a docker image of the webapp I'm working on. Whenever I run packer build, when it gets to the step that it runs the runit recipe, I would get Build 'docker' errored: Error executing Chef: Non-zero exit status: 137
I looked into 137, and found out this is the exit code commonly associated with a kill -9. In most cases this would imply that the system is running critically low on memory, and the system is attempting to compensate.
I tried to find the smallest possible reproduction, and I came up with this packer configuration:
{
"builders":[{
"type": "docker",
"pull": false,
"image": "silkstart/basic_server",
"export_path": "image.tar",
"run_command":[
"-d",
"-i",
"-t",
"--memory-reservation",
"1G",
"{{.Image}}",
"/bin/bash"
]
}],
"provisioners":[
{
"type": "chef-solo",
"cookbook_paths": ["cookbooks", "vendor/cookbooks"],
"data_bags_path": "data_bags",
"roles_path": "roles",
"environments_path": "environments",
"run_list": [
"recipe[runit]"
]
}
],
"post-processors": [
{
"type": "docker-import",
"repository": "silkstart/docker_test",
"tag": "0.1"
}
]
}
When I run packer build on this configuration, this is my output:
TMPDIR=/opt/shared packer build packer_files/docker_test.json
docker output will be in this color.
==> docker: Creating a temporary directory for sharing data...
==> docker: Starting docker container...
docker: Run command: docker run -v /opt/shared/packer-docker484290992:/packer-files -d -i -t --memory-reservation 1G silkstart/basic_server /bin/bash
docker: Container ID: 1f87b0cf1fe71f07b580ae6b18415a79c23a1a32a40f5f0366be90f160977a50
==> docker: Provisioning with chef-solo
docker: Installing Chef...
docker: % Total % Received % Xferd Average Speed Time Time Time Current
docker: Dload Upload Total Spent Left Speed
docker: 100 20022 100 20022 0 0 45092 0 --:--:-- --:--:-- --:--:-- 45196
docker: Getting information for chef stable for ubuntu...
docker: downloading https://omnitruck-direct.chef.io/stable/chef/metadata?v=&p=ubuntu&pv=14.04&m=x86_64
docker: to file /tmp/install.sh.23/metadata.txt
docker: trying curl...
docker: url https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/14.04/x86_64/chef_12.6.0-1_amd64.deb
docker: md5 5cfc19d5a036b3f7860716bc9795a85e
docker: sha256 e0b42748daf55b5dab815a8ace1de06385db98e29a27ca916cb44f375ef65453
docker: version 12.6.0downloaded metadata file looks valid...
docker: downloading https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/14.04/x86_64/chef_12.6.0-1_amd64.deb
docker: to file /tmp/install.sh.23/chef_12.6.0-1_amd64.deb
docker: trying curl...
docker: Comparing checksum with sha256sum...
docker:
docker: WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING
docker:
docker: You are installing an omnibus package without a version pin. If you are installing
docker: on production servers via an automated process this is DANGEROUS and you will
docker: be upgraded without warning on new releases, even to new major releases.
docker: Letting the version float is only appropriate in desktop, test, development or
docker: CI/CD environments.
docker:
docker: WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING
docker:
docker: Installing chef
docker: installing with dpkg...
docker: Selecting previously unselected package chef.
docker: (Reading database ... 17195 files and directories currently installed.)
docker: Preparing to unpack .../chef_12.6.0-1_amd64.deb ...
docker: Unpacking chef (12.6.0-1) ...
docker: Setting up chef (12.6.0-1) ...
docker: Thank you for installing Chef!
docker: Creating directory: /tmp/packer-chef-solo
docker: Creating directory: /tmp/packer-chef-solo/cookbooks-0
docker: Creating directory: /tmp/packer-chef-solo/cookbooks-1
docker: Creating directory: /tmp/packer-chef-solo/roles
docker: Creating directory: /tmp/packer-chef-solo/data_bags
docker: Creating directory: /tmp/packer-chef-solo/environments
docker: Creating configuration file 'solo.rb'
docker: Creating JSON attribute file
docker: Executing Chef: sudo chef-solo --no-color -c /tmp/packer-chef-solo/solo.rb -j /tmp/packer-chef-solo/node.json
docker: [2016-01-29T06:42:48+00:00] INFO: Forking chef instance to converge...
docker: [2016-01-29T06:42:48+00:00] INFO: *** Chef 12.6.0 ***
docker: [2016-01-29T06:42:48+00:00] INFO: Chef-client pid: 207
docker: [2016-01-29T06:42:50+00:00] INFO: Setting the run_list to ["recipe[runit]"] from CLI options
docker: [2016-01-29T06:42:50+00:00] INFO: Run List is [recipe[runit]]
docker: [2016-01-29T06:42:50+00:00] INFO: Run List expands to [runit]
docker: [2016-01-29T06:42:50+00:00] INFO: Starting Chef Run for 1f87b0cf1fe7
docker: [2016-01-29T06:42:50+00:00] INFO: Running start handlers
docker: [2016-01-29T06:42:50+00:00] INFO: Start handlers complete.
docker: [2016-01-29T06:42:52+00:00] INFO: Processing service[runit] action nothing (runit::default line 20)
docker: [2016-01-29T06:42:52+00:00] INFO: Processing execute[start-runsvdir] action nothing (runit::default line 24)
docker: [2016-01-29T06:42:52+00:00] INFO: Processing execute[runit-hup-init] action nothing (runit::default line 33)
docker: [2016-01-29T06:42:52+00:00] INFO: Processing apt_package[runit] action install (runit::default line 64)
docker: [2016-01-29T06:42:55+00:00] INFO: Processing cookbook_file[/var/chef/cache/preseed/runit/runit-2.1.1-6.2ubuntu3.seed] action create (dynamically defined)
docker: [2016-01-29T06:42:55+00:00] INFO: cookbook_file[/var/chef/cache/preseed/runit/runit-2.1.1-6.2ubuntu3.seed] created file /var/chef/cache/preseed/runit/runit-2.1.1-6.2ubuntu3.seed
docker: [2016-01-29T06:42:55+00:00] INFO: cookbook_file[/var/chef/cache/preseed/runit/runit-2.1.1-6.2ubuntu3.seed] updated file contents /var/chef/cache/preseed/runit/runit-2.1.1-6.2ubuntu3.seed
docker: [2016-01-29T06:42:55+00:00] INFO: apt_package[runit] pre-seeding package installation instructions
==> docker: Killing the container: 1f87b0cf1fe71f07b580ae6b18415a79c23a1a32a40f5f0366be90f160977a50
Build 'docker' errored: Error executing Chef: Non-zero exit status: 137
I'm not entirely sure what is causing the code 137, and any help would be appreciated.
Update 1
I'm including a gist of the full debug output from Chef. It's much more verbose, mainly due it would seem to all of the attempts Ohai makes to get information.
https://gist.github.com/jrstarke/4c5f3b432aaee70c7f77
No references in here seem to suggest an out of memory error, at least on the docker host.
After much much digging, I found the problem. The underlying problem, and the solution were both found on an issue in cloudfoundry-incubator/garden-linux.
Apparently as part of the setup process one of the post init scripts for runit executes a kill -s HUP 1. Why I'm not entirely sure, but as they noted there, doing a trap '' HUP right before my apt-get install runit and a trap HUP afterwards totally solved my problem.
Check the OOM log on the host machine. Also you can use the execute_command configuration value to turn the log level to debug.
This answer seemed to work for me: https://stackoverflow.com/a/42398166/2878244
I had to increase the memory resources assigned to docker by going to the Docker Tab > Preferences > Advanced

Docker on RHEL 6 Cgroup mounting failing

I'm trying to get my head around something that's been working on a Centos+Vagrant, but not on our providers RHEL (Red Hat Enterprise Linux Server release 6.5 (Santiago)). A sudo service docker restart hands this:
Stopping docker: [ OK ]
Starting cgconfig service: Error: cannot mount cpuset to /cgroup/cpuset: Device or resource busy
/sbin/cgconfigparser; error loading /etc/cgconfig.conf: Cgroup mounting failed
Failed to parse /etc/cgconfig.conf [FAILED]
Starting docker: [ OK ]
The service starts okey enough, but images cannot run. A mounting failed error is shown when I try. And the startup-log also gives a warning or two. Regarding the kernelwarning, centos gives the same and has no problems as Epel should resolve this:
WARNING: You are running linux kernel version 2.6.32-431.17.1.el6.x86_64, which might be unstable running docker. Please upgrade your kernel to 3.8.0.
2014/08/07 08:58:29 docker daemon: 1.1.2 d84a070; execdriver: native; graphdriver:
[1233d0af] +job serveapi(unix:///var/run/docker.sock)
[1233d0af] +job initserver()
[1233d0af.initserver()] Creating server
2014/08/07 08:58:29 Listening for HTTP on unix (/var/run/docker.sock)
[1233d0af] +job init_networkdriver()
[1233d0af] -job init_networkdriver() = OK (0)
2014/08/07 08:58:29 WARNING: mountpoint not found
Anyone had any success overcoming this problem or should I throw in the towel and wait for the provider to update to RHEL 7?
I have the same issue.
(1) check cgconfig status
# /etc/init.d/cgconfig status
if it stopped, restart it
# /etc/init.d/cgconfig restart
check cgconfig is running
(2) check cgconfig is on
# chkconfig --list cgconfig
cgconfig 0:off 1:off 2:off 3:off 4:off 5:off 6:off
if cgconfig is off, turn it on
(3) if still does not work, may be some cgroups modules is missing. In the kernel .config file, make menuconfig, add those modules into kernel and recompile and reboot
after that, it should be OK
I ended up asking the same question at Google Groups and in the end finding a solution with some help. What worked for me was this:
umount cgroup
sudo service cgconfig start
The project of making Docker work was put on halt all the same. Later a problem of network connection for the containers. This took to much time to solve and had to give up.
So I spent the whole day trying to rig docker to work on my vps. I was running into this same error. Basically what it came down to was the fact that OpenVZ didn't support docker containers up until a couple months ago. Specifically this RHEL update:
https://openvz.org/Download/kernel/rhel6/042stab105.14
Assuming this is your problem, or some variation of it, the burden of solving it is on your host. They will need to follow these steps:
https://openvz.org/Docker_inside_CT
In my case
/etc/rc.d/rc.cgconfig start
was generating
Starting cgconfig service: Error: cannot mount cpu,cpuacct,memory to
/cgroup/cpu_and_mem: Device or resource busy /usr/sbin/cgconfigparser;
error loading /etc/cgconfig.conf: Cgroup mounting failed Failed to
parse /etc/cgconfig.conf
i had to use:
/etc/rc.d/rc.cgconfig restart
and it automagicly umouted and mounted groups
Stopping cgconfig service: Starting cgconfig service:
it seems like the cgconfig service not running,so check it!
# /etc/init.d/cgconfig status
# mkdir -p /cgroup/cpuacct /cgroup/memory /cgroup/devices /cgroup/freezer net_cls /cgroup/blkio
# cat /etc/cgconfig.conf |tail|grep "="|awk '{print "mount -t cgroup -o",$1,$1,$NF}'>cgroup_mount.sh
# sh ./cgroup_mount.sh
# /etc/init.d/cgconfig restart
# /etc/init.d/docker restart
This situation occurs when the kernel is booted with cgroup_disable=memory and /etc/cgconfig.conf contains memory = /cgroup/memory;
This causes only /cgroup/cpuset to be mounted instead of the full set.
Solution: either remove cgroup_disable=memory from your kernel boot options or comment out memory = /cgroup/memory; from cgconfig.conf.
The cgconfig service startup uses mount and umount which requires an extra privilege bump from docker.
See the --privileged=true flag here for more info.
I was able to overcome this issue by starting my container with:
docker run -it --privileged=true my-image.
Tested in Centos6, Centos6.5.

Resources