Building docker image, inside an alpine linux container - docker

We are using docker containers running alpine linux as build agents in bamboo. As part of the build plan a docker image needs to be created.
Our build agent has docker installed, however we are getting errors as the docker daemon isn't running. Starting the daemon with
/usr/local/bin/dockerd
gives the following:
INFO[0000] libcontainerd: new containerd process, pid: 640
ERRO[0001] 'overlay' is not supported over overlayfs
INFO[0001] Graph migration to content-addressability took 0.00 seconds
INFO[0001] Loading containers: start.
WARN[0001] Running modprobe bridge br_netfilter failed with message: modprobe: can't change directory to '/lib/modules': No such file or directory
, error: exit status 1
WARN[0001] Running modprobe nf_nat failed with message: `modprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1
WARN[0001] Running modprobe xt_conntrack failed with message: `modprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1
Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain: Iptables not found
To try and get round this I've installed iptables. Now when I try to run the docker daemon I get:
INFO[0000] libcontainerd: new containerd process, pid: 705
ERRO[0001] 'overlay' is not supported over overlayfs
INFO[0001] Graph migration to content-addressability took 0.00 seconds
INFO[0001] Loading containers: start.
WARN[0001] Running modprobe bridge br_netfilter failed with message: modprobe: can't change directory to '/lib/modules': No such file or directory
, error: exit status 1
WARN[0001] Running modprobe nf_nat failed with message: `modprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1
WARN[0001] Running modprobe xt_conntrack failed with message: `modprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1
Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain: iptables failed: iptables -t nat -N DOCKER: iptables v1.6.0: can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.
(exit status 3)
I'm a little stuck on where to go now as sudo isn't a thing on alpine.

Running the Docker daemon inside the Docker container is generally a dodgy prospect. We have this exact same requirement, and we solve it by mounting /var/run/docker.sock from the Docker host into the Docker container:
docker run -v /var/run/docker.sock:/var/run/docker.sock --privileged
That way the docker command inside the build agent container is actually talking to the Docker daemon on the host, not inside the container. Works really well for us.

Related

process_linux.go:458: setting cgroup config for procHooks process caused: can't load program: operation not permitted: unknown (Docker))

I'm trying to deploy a docker container to perform some testings with gitlab runners but when I'm doing the docker-compose up command I get the following output:
admin#runners-test:~/runner-test$ sudo docker-compose up -d
Starting gitlab-runner ... error
ERROR: for gitlab-runner Cannot start service gitlab-runner: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: can't load program: operation not permitted: unknown
ERROR: for gitlab-runner Cannot start service gitlab-runner: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: can't load program: operation not permitted: unknown
ERROR: Encountered errors while bringing up the project.
Abd this is the output of journalctl:
Sep 23 07:23:24 runners-test dockerd[231]: time="2022-09-23T07:23:24.560275437Z" level=error msg="stream copy error: reading from a closed fifo"
Sep 23 07:23:24 runners-test dockerd[231]: time="2022-09-23T07:23:24.577270402Z" level=error msg="stream copy error: reading from a closed fifo"
Sep 23 07:23:24 runners-test dockerd[231]: time="2022-09-23T07:23:24.675282811Z" level=error msg="0cd3bbb779a947012c9059921f092b569eb088bb2fe0bf99a8ae3266ec43abbd cleanup: failed to delete container from containerd: no such container"
Sep 23 07:23:24 runners-test dockerd[231]: time="2022-09-23T07:23:24.675625496Z" level=error msg="Handler for POST /v1.25/containers/0cd3bbb779a947012c9059921f092b569eb088bb2fe0bf99a8ae3266ec43abbd/start returned error: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: can't load program: operation not permitted: unknown"
And the docker-compose file:
version: '3'
services:
gitlab-runner:
container_name: gitlab-runner
image: 'gitlab/gitlab-runner:latest'
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./config:/etc/gitlab-runner
restart:
unless-stopped
This error occurs with any container I try to lift, even doing a simple docker run hello-world...
I also take a look to this post on proxmox forum but I don't know how to add kernel command line parameters for a lxc container, I'v been trying editing the /etc/default/grub file but update-grub command doesn't work as I want inside the lxc, I get the following output:
admin#runners-test:/$ sudo update-grub
[sudo] password for admin:
/usr/sbin/grub-probe: error: failed to get canonical path of `/dev/mapper/pve-vm--1010118--disk--0'.
I'm really stuck with this, so any kind of help would be welcome :) Thanks!
I referenced many other links when finally discovering doing a (on CentOS) update did the trick. So:
yum update -y
reboot
After the reboot, the system came back online and I proceeded to run my docker container, in this case a docker registry:
[root#server ~]# docker run -d --name registry registry:2
c40941bf42c853709bcca05bad4e8914df1f4932a355607f37b55f7e0ed01e60
[root#server ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
c40941bf42c8 registry:2 "/entrypoint.sh /e..." 3 seconds ago Up 2 seconds 5000/tcp
The docker container booted up fine.
So, also here are reference pages I checked out, and I did not have to change anything in /boot/boot.txt as this one references, but is good info:
https://my-take-on.tech/2021/05/07/fix-docker-cgroup-errors-after-systemd-248-update/
and this one, which after I read it, it had me finally trying to do a yum update for fix:
docker-compose throws errors by starting mariadb for an private nextcloud installation

Base Docker in Docker image cannot start Docker daemon

I have reduced my dockerfile to the following
FROM docker:latest
EXPOSE 3000
But when running the image, docker daemon cannot start.
Running dockerd in the container results in a large chain of info, errors and warnings ending with the following:
WARN[2021-12-09T01:07:36.691842800Z] grpc: addrConn.createTransport failed to connect to {unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial unix:///var/run/docker/containerd/containerd.sock: timeout". Reconnecting... module=grpc
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: Iptables not found
Am I missing something? I can manually install iptables but then it fails again with
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.7 (legacy): can't initialize iptables table `nat': Permission denied (you must be root)
So I am assuming I have some setting wrong as it seems to be working out of the box here https://hub.docker.com/_/docker
I am running docker on Windows with the WSL 2 backend.

docker set iptables false, minikube start fails

I'm having an error trying to have docker set iptables false when minikube start fails.
Below are my logs:
minikube v1.20.0 on Centos 7.6.1810 (amd64)
* Using the none driver based on existing profile
* Starting control plane node minikube in cluster minikube
* Restarting existing none bare metal machine for "minikube" ...
* OS release is CentOS Linux 7 (Core)
* Preparing Kubernetes v1.20.2 on Docker 20.10.6 ...
! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.20.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem": exit status 1
stdout:
[init] Using Kubernetes version: v1.20.2
[preflight] Running pre-flight checks
stderr:
[WARNING Firewalld]: firewalld is active, please ensure ports [8443 10250] are open or your cluster may not function correctly
[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING FileExisting-socat]: socat not found in system path
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.6. Latest validated version: 19.03
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
Error you included states that you are misising bridge-nf-call-iptables.
bridge-nf-call-iptables is exported by br_netfilter.
What you need to do is issue the command
sudo modprobe br_netfilter
and then ensure net.bridge.bridge-nf-call-iptables is set to 1 in your sysctl
cat <<EOF > /etc/sysctl.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
This should fix your problem

docker - start failed because /etc/fstab not found

I'm using Window Linux Subsystem (Debian stretch). Followed the instruction on Docker website, I installed docker-ce, but it cannot start. Here is the info:
$ sudo service docker start
grep: /etc/fstab: No such file or directory
[ ok ] Starting Docker: docker.
$ sudo service docker status
[FAIL] Docker is not running ... failed!
What should I do with /etc/fstab not found?
to fix fstab
touch /etc/fstab
if you run dockerd, it will give you the failed message:
INFO[2022-01-27T17:55:14.100489400+07:00] Loading containers: start.
WARN[2022-01-27T17:55:14.191666800+07:00] Running iptables --wait -t nat -L -n failed with message: `iptables v1.8.2 (nf_tables): CHAIN_ADD failed (No such file or directory): chain PREROUTING`, error: exit status 4
INFO[2022-01-27T17:55:14.493716300+07:00] stopping event stream following graceful shutdown error="<nil>" module=libcontainerd namespace=moby
INFO[2022-01-27T17:55:14.494906600+07:00] stopping event stream following graceful shutdown error="context canceled" module=libcontainerd namespace=plugins.moby
INFO[2022-01-27T17:55:14.495048400+07:00] stopping healthcheck following graceful shutdown module=libcontainerd
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables --wait -t nat -N DOCKER: iptables v1.8.2 (nf_tables): CHAIN_ADD failed (No such file or directory): chain PREROUTING
(exit status 4)
that is Debian nat issue, fix it with:
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
now you can start the service again
you can follow this to make it start on startup https://askubuntu.com/a/1356147/138352
Edited:
if the issue with IP table still persisted try to set WSL version to 2, run the command from Windows shell:
wsl --set-version <distribution name> 2
the distribution list can be found with command wsl -l
I was getting the same error. Apparently on my install of WSL with Debian, I didn't have an etc/fstab file. Surprisingly, just creating the file via 'touch' worked:
sudo touch /etc/fstab
Perhaps a good signal https://learn.microsoft.com/en-us/windows/wsl/release-notes#build-17093
WSL now processes the /etc/fstab file during instance start [GH 2636].
For anybody stumbling across this years later like me, Docker doesn't work inside WSL.
But you can use Docker for Windows and WSL2 to run native containers inside your Linux Distro and the install and config is quite painless https://learn.microsoft.com/en-us/windows/wsl/tutorials/wsl-containers

Checking container connectivity with ping in a Calico network of Docker containers

While checking the connectivity of Docker container in Calico VPN an error occurs. Here's the command and error snapshot:
root#ubuntu12# docker exec Container1 ping -c 4 www.google.com
rpc error: code = 2 desc = "oci runtime error: exec failed: exec: \"ping\": executable file not found in $PATH"

Resources