mounting permission denied in docker - docker

Question

I was facing issues installing docker on cloud server according to the official guide(Install Docker Engine on Ubuntu). I finished old version's uninstallation, the repository setting up and docker engine installation (sudo apt-get install docker-ce docker-ce-cli containerd.io). However, I got an error when running hello-world.
wyf#VM1103-Timi:~$ sudo docker run hello-world
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"proc\\\" to rootfs \\\"/var/lib/docker/overlay2/e9fedf64e8983aa01e513cee591cdfd7fc60962466a476b51fc1ead682ec8022/merged\\\" at \\\"/proc\\\" caused \\\"permission denied\\\"\"": unknown.
ERRO[0000] error waiting for container: context canceled
I tried restart docker and server, but the problem still exists.
So, it would be great if someone can guide me in fixing this error.
Please let me know if you have any idea about this issue.
Thank you very much!
Ps:
My system is Ubuntu 18.04. Thus, I did not have selinux. Instead of selinux, I checked AppArmor log.
May 19 21:14:55 VM1103-Timi networkd-dispatcher[155]: WARNING:Unknown index 37 seen, reloading interface list
May 19 21:14:55 VM1103-Timi systemd-networkd[126]: veth71cf495: Link UP
May 19 21:14:55 VM1103-Timi containerd[170]: time="2020-05-19T21:14:55.679793295+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/4c207ce1273d2c863ee419c5ebb271163a031394bd4c17ee75d44267d631954d/shim.sock" debug=false pid=106265
May 19 21:14:55 VM1103-Timi containerd[170]: time="2020-05-19T21:14:55.767796543+08:00" level=info msg="shim reaped" id=4c207ce1273d2c863ee419c5ebb271163a031394bd4c17ee75d44267d631954d
May 19 21:14:55 VM1103-Timi dockerd[15100]: time="2020-05-19T21:14:55.776863367+08:00" level=error msg="stream copy error: reading from a closed fifo"
May 19 21:14:55 VM1103-Timi dockerd[15100]: time="2020-05-19T21:14:55.776953910+08:00" level=error msg="stream copy error: reading from a closed fifo"
May 19 21:14:55 VM1103-Timi systemd-networkd[126]: veth71cf495: Link DOWN
May 19 21:14:55 VM1103-Timi dockerd[15100]: time="2020-05-19T21:14:55.927805156+08:00" level=error msg="4c207ce1273d2c863ee419c5ebb271163a031394bd4c17ee75d44267d631954d cleanup: failed to delete container from containerd: no such container"
The strange thing is that there is no record of permission-denied error.
Here are my ubuntu version, kernal version and docker info:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic
5.3.18-3-pve
Client:
Debug Mode: false
Server:
Containers: 8
Running: 0
Paused: 0
Stopped: 8
Images: 1
Server Version: 19.03.8
Storage Driver: overlay2
Backing Filesystem: <unknown>
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.3.18-3-pve
Operating System: Ubuntu 18.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 4GiB
Name: VM1103-Timi
ID: 3G3F:LTVZ:NO25:C7LA:XKQV:ETMB:B6QU:3ZFJ:KBA5:R3KK:QZEA:ZONC
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
It seemed that the AppArmor Profile "docker-default" was lost. "docker-default" was not correctly generated. Check as follows:
root#VM1103-Timi:/etc/apparmor.d# aa-status
apparmor module is loaded.
12 profiles are loaded.
12 profiles are in enforce mode.
/sbin/dhclient
/usr/bin/man
/usr/lib/NetworkManager/nm-dhcp-client.action
/usr/lib/NetworkManager/nm-dhcp-helper
/usr/lib/connman/scripts/dhclient-script
/usr/lib/lightdm/lightdm-guest-session
/usr/lib/lightdm/lightdm-guest-session//chromium
/usr/sbin/mysqld
/usr/sbin/tcpdump
docker-default
man_filter
man_groff
0 profiles are in complain mode.
1 processes have profiles defined.
1 processes are in enforce mode.
/usr/sbin/mysqld (258)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

Answer 1

Solution is probably to open ports needed. Your system might be running selinux and (ufw or firewalld or iptables) ?and/or others?. Read up a bit on linux firewall tools, in particular the ones running on your system.
For the selinux case, you need to check selinux logs, is it blocking access? Add exceptions using selinux commands.
https://wiki.centos.org/HowTos/SELinux These tools are well worth learning but can be complicated. A quick test disabling selinux and firewalld can confirm that this is the source of problem and you can enable selinux and firewalld later and allow/open ports in a secure way.
Simple test: disable selinux and firewalld, e.g. on CentOS
systemctl stop firewalld;
setenforcing 0;
If you can create containers with selinux disabled then you have confirmed selinux is your problem. You can enable firewall and selinux and then add exceptions and open ports as needed later.
This looks good (specific to ubuntu but general enough IMHO), It details ufw commands, firewalld commands and iptables commands needed for opening ports to allow docker swarm to work) https://www.digitalocean.com/community/tutorials/how-to-configure-the-linux-firewall-for-docker-swarm-on-ubuntu-16-04
I originally got useful info on ufw commands to open ports needed from here:
Error response from daemon: attaching to network failed, make sure your network options are correct and check manager logs: context deadline exceeded
ufw allow 2376/tcp
ufw allow 2377/tcp
ufw allow 7946/tcp
ufw allow 7946/udp
ufw allow 4789/udp
ufw enable #maybe
ufw reload
systemctl restart docker
This is a common enough problem where something usually selinux is not allowing access to ports needed.
e.g.
https://github.com/google/cadvisor/issues/333