docker AWS fail to launch - custom kernel - docker

I cannot launch docker on my AWS instance:
root#system:~# docker -H tcp://127.0.0.1:2375 -H
unix:///var/run/docker.sock -d
root#system:~# INFO[0000] +job serveapi(tcp://127.0.0.1:2375, unix:///var/run/docker.sock)
INFO[0000] +job init_networkdriver()
INFO[0000] Listening for HTTP on tcp (127.0.0.1:2375)
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
Unable to allow incoming packets: iptables failed: iptables --wait -I
FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT:
iptables: No chain/target/match by that name.
(exit status 1)
INFO[0000] -job init_networkdriver() = ERR (1)
FATA[0000] (exit status 1)
How would you troubleshoot that error sequence message?

Double-check the docker installation procedure on AWS:
[ec2-user ~]$ sudo yum update -y
[ec2-user ~]$ sudo yum install -y docker
[ec2-user ~]$ sudo service docker start
Starting cgconfig service: [ OK ]
Starting docker: [ OK ]
[ec2-user ~]$ sudo usermod -a -G docker ec2-user
# Log out and log back in again to pick up the new docker group permissions.
# Verify that the ec2-user can run Docker commands without sudo.
[ec2-user ~]$ docker info

Related

Using docker in Jenkins pipeline fails

In Jenkinsfile I am trying to download docker image and use in this docker image for all the git pull and frontend build stuff.
My jenkinsfile is so far:
pipeline {
agent any
stages {
stage('Install Docker-CE') {
steps {
sh '''curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install -y docker-ce'''
}
}
stage('Start Docker') {
steps {
sh 'sudo service docker start'
sh 'sudo service docker status'
}
}
stage('Verify Docker') {
steps {
sh 'sudo docker run hello-world'
}
}
stage('Build Back End') {
steps {
git ([url : 'https://github....git', branch : 'develop', credentialsId : 'xxx' ])
}
}
}
The docker is installed and running, because in the 'Start Docker' step when I run sudo service docker status, it says: Docker is running
but when trying to do 'hello world' in the next step, it says docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
I also tried to run docker daemon with: sudo dockerd but it didn't help, this is the part of the output:
time="2021-07-28T08:45:04.782471391Z" level=error msg="failed to mount overlay: permission denied" storage-driver=overlay2
time="2021-07-28T08:45:04.782647045Z" level=error msg="exec: \"fuse-overlayfs\": executable file not found in $PATH" storage-driver=fuse-overlayfs
time="2021-07-28T08:45:04.783389746Z" level=warning msg="[graphdriver] WARNING: the aufs storage-driver is deprecated, and will be removed in a future release"
time="2021-07-28T08:45:04.810999832Z" level=warning msg="Your kernel does not support CPU realtime scheduler"
time="2021-07-28T08:45:04.811283522Z" level=info msg="Loading containers: start."
time="2021-07-28T08:45:04.816297382Z" level=warning msg="Running iptables --wait -t nat -L -n failed with message: `iptables v1.6.0: can't initialize iptables table `nat': Permission denied (you must be root)\nPerhaps iptables or your kernel needs to be upgraded.`, error: exit status 3"
time="2021-07-28T08:45:04.887136525Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
time="2021-07-28T08:45:04.887859247Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
time="2021-07-28T08:45:04.887889167Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.6.0: can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.
(exit status 3)
Ok, so the problem was I used sudo in sudo docker run hello-world. Without sudo it works...

How do you publish ports from a Docker container on a system using nftables?

I have a fairly plain Debian Buster install. Debian Buster uses nftables rather than iptables. If I try and run a container with a published port:
sudo docker run -it --rm --name=port-test -p 1234:1234/tcp debian:jessie-slim
then I get this error:
docker: Error response from daemon: driver failed programming external connectivity on endpoint port-test (941052b9f420df39ac3d191dcbe12e97276703903911e7b5172663e7736d59e0): (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 1234 -j DNAT --to-destination 172.17.0.2:1234 ! -i docker0: iptables v1.8.2 (nf_tables): Chain 'DOCKER' does not exist
How do I get port publishing working?
Please see this issue and comment:
https://github.com/moby/moby/issues/26824#issuecomment-517046804
You can run sudo update-alternatives --config iptables and sudo update-alternatives --config ip6tables (if you use IPv6), and set it to iptables-legacy which is a compatibility mode that Docker can work with.

install docker on debian - 'overlay' is not supported over overlayfs

I created a docker image for my build pipeline and I am trying to run docker build -t someTag .
However, I am getting this error.
DEBU[0000] Listener created for HTTP on unix (/var/run/docker.sock)
INFO[0000] libcontainerd: new containerd process, pid: 1026
DEBU[0000] Permission denied writing "-500" to /proc/1026/oom_score_adj
DEBU[0000] containerd: grpc api on /var/run/docker/libcontainerd/docker-containerd.sock
DEBU[0000] containerd: read past events count=0
DEBU[0000] containerd: supervisor running cpus=2 memory=976 runtime=docker-runc runtimeArgs=[] stateDir="/var/run/docker/libcontainerd/containerd"
DEBU[0000] libcontainerd: containerd health check returned error: rpc error: code = 14 desc = grpc: the connection is unavailable
DEBU[0001] Permission denied writing "-500" to /proc/self/oom_score_adj
DEBU[0001] Using default logging driver json-file
DEBU[0001] Golang's threads limit set to 6840
DEBU[0001] [zfs] zfs command is not available: exec: "zfs": executable file not found in $PATH
ERRO[0001] 'overlay' is not supported over overlayfs
DEBU[0001] Using graph driver vfs
DEBU[0001] Max Concurrent Downloads: 3
DEBU[0001] Max Concurrent Uploads: 5
INFO[0001] Graph migration to content-addressability took 0.00 seconds
INFO[0001] Loading containers: start.
DEBU[0001] Option Experimental: false
DEBU[0001] Option DefaultDriver: bridge
DEBU[0001] Option DefaultNetwork: bridge
WARN[0001] Running modprobe bridge br_netfilter failed with message: , error: exec: "modprobe": executable file not found in $PATH
WARN[0001] Running modprobe nf_nat failed with message: ``, error: exec: "modprobe": executable file not found in $PATH
WARN[0001] Running modprobe xt_conntrack failed with message: ``, error: exec: "modprobe": executable file not found in $PATH
DEBU[0001] Fail to initialize firewalld: Failed to connect to D-Bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory, using raw iptables instead
DEBU[0001] /usr/sbin/iptables, [-t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[0001] /usr/sbin/iptables, [-t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER]
DEBU[0001] /usr/sbin/iptables, [-t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[0001] /usr/sbin/iptables, [-t nat -D PREROUTING]
DEBU[0001] /usr/sbin/iptables, [-t nat -D OUTPUT]
DEBU[0001] /usr/sbin/iptables, [-t nat -F DOCKER]
DEBU[0001] /usr/sbin/iptables, [-t nat -X DOCKER]
DEBU[0001] /usr/sbin/iptables, [-t filter -F DOCKER]
DEBU[0001] /usr/sbin/iptables, [-t filter -X DOCKER]
DEBU[0001] /usr/sbin/iptables, [-t filter -F DOCKER-ISOLATION]
DEBU[0001] /usr/sbin/iptables, [-t filter -X DOCKER-ISOLATION]
DEBU[0001] /usr/sbin/iptables, [-t nat -n -L DOCKER]
DEBU[0001] /usr/sbin/iptables, [-t nat -N DOCKER]
DEBU[0001] start clean shutdown of all containers with a 15 seconds timeout...
DEBU[0001] Cleaning up old mountid : start.
DEBU[0001] Cleaning up old mountid : done.
Here is the info of my Dockerfile that is installing docker. I found these steps here.
FROM debian:latest
ENV DEBIAN_FRONTEND noninteractive
ENV JENKINS_HOME /var/jenkins_home
USER root
WORKDIR $JENKINS_HOME
RUN apt-get update
RUN apt-get install \
apt-transport-https \
ca-certificates \
curl \
gnupg2 \
software-properties-common -y
RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
RUN apt-get install -y iptables
RUN wget https://download.docker.com/linux/debian/dists/jessie/pool/stable/amd64/docker-ce_17.06.0~ce-0~debian_amd64.deb
RUN dpkg -i docker-ce_17.06.0~ce-0~debian_amd64.deb
What I have tried:
I have tried starting dockerd.
From my best guess, it looks like I need dockerd to start.
root#90302deb48a0:/var/jenkins_home# ps -ef | grep docker
root 6 1 0 14:47 ? 00:00:00 grep docker
I have looked into updating
I have tried updating my iptables
root#e898473347a2:/var/jenkins_home# apt-get install -y iptables
Reading package lists... Done
Building dependency tree
Reading state information... Done
iptables is already the newest version (1.6.0+snapshot20161117-6).
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
I have verified that I am root and the directories are owned by root.
root#e898473347a2:/var/jenkins_home# whoami
root
I have tried to research where to post this. Please tell me if this needs to be moved somewhere else. I will kindly move this if this doesn't classify as "programming".
It sounds like you are trying to do Docker in Docker (DinD). It requires some hacks and changing the storage driver (you can't do overlay on overlay). See the Dockerfile and entrypoint script at https://github.com/docker-library/docker/tree/a73d96e731e2dd5d6822c99a9af4dcbfbbedb2be/19.03/dind for more information. It has information on where it gets the dind script and the necessary entrypoint changes to run dockerd.
However, there are official dind tagged images on Docker Hub that you may be better off building from. See https://hub.docker.com/_/docker/. You could just use image docker:17.05.0-dind or base your image off of it if you need customizations.

Docker Swarm consul not starting

When I try to start docker consul by this command
docker run --restart=unless-stopped -d -p 8500:8500 -h consul progrium/consul -server -bootstrap
it gives the following error.
docker: Error response from daemon: driver failed programming external
connectivity on endpoint tiny_bhaskara
(b53c9aa988d96750bfff7c19c6717b18756c7b92287f0f7a4d9e9fa81f42c43d):
iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0
--dport 8500 -j DNAT --to-destination 172.17.0.2:8500 ! -i docker0: iptables: No chain/target/match by that name.
No idea what's going on!!
From this answer:
Something on your system has removed the docker iptables entries that it needs to work. Two fixes have been suggested here:
For CentOS:
sudo service docker restart
sudo service iptables save
And for Ubuntu:
sudo apt-get install iptables-persistent
sudo service docker restart
iptables-save > /etc/iptables/rules.v4 # you may need to "sudo -s" to get a root shell first
After the restart of docker, you should see the docker chain under the nat table:
iptables -t nat -vL

restcomm docker image installation on windows 8.1

I am trying to install Docker image of Restcomm on my windows 8.1 laptop by following http://www.telestax.com/rapid-webrtc-application-development-with-restcomm-and-docker/.
I am able to install DOCKER and run the container Hello-world properly.
But when i run the command to create container... "docker run –name=restcomm -d -e STATIC_ADDRESS=”YOUR_HOST_IP_ADDRESS_HERE” -p 8080:8080 -p 5080:5080 -p 5082:5082 -p 5080:5080/udp -p 65000-65535:65000-65535/udp gvagenas/restcomm"
i am getting the following error...
Error response from daemon: Cannot start container c88fcab56034096e98ddcd71d1d2db17e5858b88c64b1859efcb86d740e3972c: failed to create endpoint restcomm on network bridge: iptables failed: iptables --wait -t nat -A DOCKER -p udp -d 0/0 --dport 65116 -j DNAT --to-destination 172.17.0.2:65116 ! -i docker0: (fork/exec /usr/local/sbin/iptables: cannot allocate memory)
Request for your help and suggestion and thanks in advance
Rgds
Ias
This doesn't seem directly related to RestComm Docker image but to docker itself https://github.com/docker/docker/issues/8539 and https://github.com/docker/docker/issues/9139

Resources