Rootless dind running in a kubernetes slave, 'docker run' fails - docker

Has anyone seen / resolved the below:
I have a jenkins slave with rootless dind configured, all docker commands work except docker run, details and error below:
Error:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:319: getting the final child's pid from pipe caused \"EOF\"": unknown.
ERRO[0004] error waiting for container: context canceled
config and versions:
uname -a
Linux jnlp-5n7x4 4.4.0-1092-aws #103-Ubuntu SMP Tue Aug 27 10:21:48 UTC 2019 x86_64 Linux
docker info:
Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 1
Server Version: 19.03.8
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: none
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
seccomp
Profile: default
rootless
Kernel Version: 4.4.0-1092-aws
Operating System: Alpine Linux v3.11 (containerized)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.812GiB
Name: jnlp-5n7x4
ID: X54B:QFRO:NKMQ:YJMW:NEVU:QU2A:VDHC:RJBI:M3YQ:KUU6:C4N7:IXNN
Docker Root Dir: /home/jenkins/.local/share/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
Thanks in advance

I was able to resolve the issue by using a different default runtime for docker.
#update default runtime
RUN wget -O crun https://github.com/containers/crun/releases/download/0.13/crun-0.13-static-x86_64 \
&& cp crun /usr/local/bin \
&& chmod a+x /usr/local/bin/crun \
&& chown -R rootless:rootless /usr/local/bin/crun
the service is then started with supervisor, config file shown:
[program:docker]
command=/home/rootless/bin/dockerd-rootless.sh --experimental --default-
runtime crun --add-runtime crun=/usr/local/bin/crun --storage-driver vfs
autorestart=true
user=rootless
detailed discussion here:
https://github.com/moby/moby/issues/40068

Related

How to solve failed to create shim error?

Using docker I am trying to run few rails commands. While running docker-compose run web rake db:create db:migrate db:seed getting Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "./docker-entrypoint_dev.sh": stat ./docker-entrypoint_dev.sh: no such file or directory: unknown
Docker Info is,
Context: default
Debug Mode: false
Server:
Containers: 13
Running: 1
Paused: 0
Stopped: 12
Images: 79
Server Version: 20.10.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version:
runc version:
init version:
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.14.0-1054-oem
Operating System: Ubuntu 20.04.2 LTS (fossa-kakuna X63)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.38GiB
Name: nichesolv-Latitude-3420
ID: PJ2E:4C3L:FJ3F:C3T7:7ZE4:4YCN:GDY6:WIMV:QIIN:N6HI:P25R:HNJ2
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
In docker-entrypoint_dev.sh,
export RAILS_ENV=development
bundle install
cd frontend
npm install
cd ..
npm install
npm audit fix
foreman start -f Procfile.dev

Docker Insecure Registry Problem when `docker build' pulls base image

When using docker build, pulling the base image from a private registry fails with
http: server gave HTTP response to HTTPS client
It should not a problem of setting insecure-registries however, because
insecure-registries is set
docker pull and push are working without problems
I can workaround it by pulling the base image first and then calling docker build, but I would prefer to get to the root of the problem.
The problem occurred on Mac as well as Windows.
Any idea what might be the problem?
Edit: The solution of Dockerfile FROM Insecure Registry doesn't work for me.
For my insecure-registries entry I use ip:port. When I try to enter http://ip:port I get an error message:
Error invoking remote method 'desktop-backend': Error: "invalid daemon settings: \"insecure-registries\":\n Must be host[:port] or CIDR"
Output of docker info on the Mac:
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Build with BuildKit (Docker Inc., v0.6.1-docker)
compose: Docker Compose (Docker Inc., v2.0.0-rc.2)
scan: Docker Scan (Docker Inc., v0.8.0)
Server:
Containers: 4
Running: 0
Paused: 0
Stopped: 4
Images: 23
Server Version: 20.10.8
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: e25210fe30a0a703442421b0f60afac609f950a3
runc version: v1.0.1-0-g4144b63
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 5.10.47-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.94GiB
Name: docker-desktop
ID: K4NV:OOUS:LNWC:BE2D:W75R:I3AH:CD5B:OFIJ:QFGT:57TT:SXEY:JP3C
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 43
Goroutines: 45
System Time: 2021-09-16T13:31:24.174487976Z
EventsListeners: 4
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
images:5000
127.0.0.0/8
Live Restore Enabled: false
daemon.json
{
"builder" : {
"gc" : {
"defaultKeepStorage" : "20GB",
"enabled" : true
}
},
"features" : {
"buildkit" : true
},
"insecure-registries" : [
"<dns-name>:<port>"
],
"registry-mirrors" : [
],
"experimental" : true,
"debug" : true
}
Dockerfile
FROM <dns-name>:<port>/java-base:latest
ADD sedexclient /opt/sedexclient
ENV SCS_USER scs
RUN useradd -M -g nogroup $SCS_USER
RUN chown -R $SCS_USER: /opt/sedexclient
RUN cp /opt/sedexclient/jce/oracle_java8/* $JAVA_HOME/jre/lib/security/
EXPOSE 8000
CMD ["/opt/sedexclient/bin/sedex-client-start.sh"]

How to add insecure registry to Docker on CentOS using shell script

I have a docker 18.03.1-ce running on CentOS 7. I am trying to add another insecure registry to it.
docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 18.03.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-514.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.639GiB
Name: ecs-cicd-0634.novalocal
ID: RGQ5:HMFO:NV2S:VXZG:U3RO:NKQ2:2BMT:XGMX:O6L7:SSWU:TCXV:5BVT
Docker Root Dir: /data/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
registry.titan.languoguang.com:5000
127.0.0.0/8
Live Restore Enabled: false
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
I just want to add another insecure registry to Docker, something like
registry-cbu.languoguang.com
Because my docker environment is a random docker environment, so i can't ssh The docker environment to modify the /etc/docker/daemon.json file when i use it in CI pipeline.
Add Insecure Registry to Docker
How to add my own insecure registry to Docker using shell script, thanks.
use jp merge two json files, it works for me.
# install jp
yum install epel-release -y
yum install jq -y
jq --version
# add registry-cbu.languoguang.com
cat /etc/docker/daemon.json
jq -s '.[0]."insecure-registries"=([.[]."insecure-registries"]|flatten)|.[0]' /etc/docker/daemon.json $PWD/ci/daemon.json > /etc/docker/daemon.extension.json
mv -f /etc/docker/daemon.extension.json /etc/docker/daemon.json
cat /etc/docker/daemon.json
echo "restart docker..."
sudo systemctl daemon-reload
sudo systemctl restart docker

Docker start container: error creating overlay mount ... /merged: no such file or directory

I had an issue with docker running on a raspberry pi 4 and I purged all of it apart from the folder with the containers/images.
Reinstalled docker, run helloworld, runs ok. Installed portainer to manage the containers, several of them start but a particular one, homeassistant, doesn't
It reports
pi#raspberrypi:~ $ docker start homeassistant
Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/7c4732337e167e0f8178b8667bddc0393c2c1e4f7ffba5d68fe57379cc1de7c0/merged: no such file or directory
Error: failed to start containers: homeassistant
Here's the result of docker info
pi#raspberrypi:~ $ docker info
Client:
Debug Mode: false
Server:
Containers: 7
Running: 4
Paused: 0
Stopped: 3
Images: 32
Server Version: 19.03.8
Storage Driver: overlay2
Backing Filesystem: <unknown>
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.19.97-v7l+
Operating System: Raspbian GNU/Linux 10 (buster)
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 3.814GiB
Name: raspberrypi
ID: CS34:4SMJ:FTCR:T3O3:JVWH:LNEC:RWIE:2D7X:435Z:4WHM:UZZZ:CMS3
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
Here are the contents of the overlay folder
pi#raspberrypi:~ $ sudo ls /var/lib/docker/overlay2/7c4732337e167e0f8178b8667bddc0393c2c1e4f7ffba5d68fe57379cc1de7c0/
diff link lower work
I've checked the overlay2 folder and indeed there's no merged folder. I've tried creating an empty one, but it gets deleted as soon as I attempt starting the container. What does this folder indicate when it comes to the container and how do I work around it? Ideally I would like to keep the container as otherwise I would have to set up the whole thing again.

"service -s" not working inside Docker container

I found an issue when using the same image on a different PC (same Linux version and same Docker version), causing different results when executing service -s in Docker.
I am using docker attach xx to execute this command.
Works fine on PC 1:
service -s
Checking for udevd: unused
Checking for Cron: unused
Checking for service D-Bus daemon unused
/etc/init.d/network is disabled,
can't find any another service enabled as the network.service.
Checking for mounted nfs shares (from /etc/fstab):gssd not running
idmapd not running
Warning: portmap/rpcbind not running - nfs may not work well
unused
Checking for UPS monitoring service unused
unused
unknown
Checking for service sshd unused
Fails on PC 2:
service -s
Failed to get D-Bus connection: No connection to service manager.
docker system info output in PC 1:
Containers: 6
Running: 3
Paused: 0
Stopped: 3
Images: 55
Server Version: 17.10.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 0351df1c5a66838d0c392b4ac4cf9450de844e2d
init version: 949e6fa
Security Options:
apparmor
Kernel Version: 4.2.0-27-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 47.16GiB
Name: hadoop3
ID: 4WGL:SPWW:7IFX:ENVE:7742:VB37:A4VY:HUI4:S74E:RYIS:Q4DS:6BQO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
docker system info output on PC 2:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 2
Server Version: 17.10.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 0351df1c5a66838d0c392b4ac4cf9450de844e2d
init version: 949e6fa
Security Options:
apparmor
Kernel Version: 4.2.0-27-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 55.03GiB
Name: hadoop1
ID: ZAYC:DRNC:HJGM:MFKP:JBKS:SAMB:U4WZ:5E7L:RR2B:PJFN:EH4D:GGOA
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
File Descriptors: 22
Goroutines: 30
System Time: 2017-11-12T23:00:09.145808427+08:00
EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
The following is my Dockerfile:
FROM opensuse:13.1
RUN zypper -n install net-tools tar python-xml python rsyslog sudo nfs-client
RUN zypper -n install wget vim less tcpdump aaa_base openssh
RUN zypper -n install syslinux
WORKDIR /root
ENTRYPOINT ["/bin/bash"]
I already try to reinstall Docker on PC 2 several times, but it isn't working out at all.
Please help me, thanks in advance!
The "service" command will actually forward all calls to "systemctl" which in turn will forward it to the SystemD daemon. It will reach the SystemD daemon by using the messagebus, aka kernel-dbus. The error "Failed to get D-Bus connection: No connection to service manager." simply says that the SystemD daemon is not running which is normal in a docker container.
I don't know why the first call had worked altogether. You may try if it helps in both cases to follow the steps of using https://github.com/gdraheim/docker-systemctl-replacement to avoid service/systemctl wanting to access the messagebus.

Resources