We are running a docker build agent inside a docker container.
It's based off debian jessie, and gets docker directly from docker as documented here.
The docker daemon runs fine the first time you start the container, but not the second time. (if you don't delete the container)
Dockerfile:
FROM debian:jessie
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update \
&& apt-get -y install -q \
apt-transport-https \
ca-certificates \
software-properties-common \
curl \
&& curl -fsSL https://yum.dockerproject.org/gpg | apt-key add - \
&& add-apt-repository \
"deb https://apt.dockerproject.org/repo/ \
debian-$(lsb_release -cs) \
main" \
&& apt-get update \
&& apt-get install -y \
docker-engine
CMD []
docker-compose.yml:
services:
dockerTest:
container_name: dockerTest
privileged: true
image: tomeinc/intel-docker-node:latest
command: bash -c "service docker start && sleep 2 && docker ps"
To reproduce: build the Dockerfile with docker build -t test . and then use docker-compose up twice. The second time, docker-ps will fail with
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Weirdly, if the container keeps running, you can manually start docker by running docker exec -it test /bin/bash and then executing service docker start and docker ps.
I'm not really sure how to approach debugging this, any suggestions are welcomed.
Turns out to be that docker thought that it and or containterd was still running(which it wasn't, but the PID files didn't get cleaned up)
Recommended starting approach to debugging issues: Look at the log files. I am shocked by this revelation.
Anyway adding rm /var/run/docker/libcontainerd/docker-containerd.pid /var/run/docker.pid to the start command before service docker start fixes it.
Related
I have a Dockerfile that extends the Apache Airflow 2.5.1 base image. What I want to do is be able to use docker inside my airflow containers (i.e. docker-in-docker) for testing and evaluation purposes.
My docker-compose.yaml has the following mount:
volumes:
- /var/run/docker.sock:/var/run/docker.sock
My Dockerfile looks as follows:
FROM apache/airflow:2.5.1
USER root
RUN apt-get update && apt-get install -y ca-certificates curl gnupg lsb-release nano
RUN mkdir -p /etc/apt/keyrings
RUN curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
RUN echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \
$(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
RUN apt-get update
RUN apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
RUN groupadd -f docker
RUN usermod -a -G docker airflow
RUN service docker start
USER airflow
Basically:
Install docker.
Add the airflow user to the docker group.
Start the docker service.
Continue as airflow.
Unfortunately, this does not work. During RUN service docker start, I encounter the following error:
Step 11/12 : RUN service docker start
---> Running in 77e9b044bcea
mkdir: cannot create directory ‘cpuset’: Read-only file system
I have another Dockerfile for building a local jenkins image, which looks as follows:
FROM jenkins/jenkins:lts-jdk11
USER root
RUN apt-get update && apt-get install -y ca-certificates curl gnupg lsb-release nano
RUN mkdir -p /etc/apt/keyrings
RUN curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
RUN echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \
$(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
RUN apt-get update
RUN apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
RUN groupadd -f docker
RUN usermod -a -G docker jenkins
RUN service docker start
USER jenkins
I.e. it is exactly the same, except that I am using the jenkins user. Building this image works.
I have not set any extraneous permission on my /var/run/docker.sock:
$ ls -la /var/run/docker.sock
srw-rw---- 1 root docker 0 Jan 18 17:14 /var/run/docker.sock
My questions are:
Why does RUN service start docker not work when building my airflow image?
Why does the exact same command in my jenkins Dockerfile work?
I've tried most of the answers to similar questions, e.g. here and here, but they have unfortunately not helped.
I'd rather try to avoid the chmod 777 /var/run/docker.sock solution if at all possible, and it should be since my jenkins image can build correctly...
Just delete the RUN service start docker line.
The docker CLI tool needs to connect to a Docker daemon, which it normally does through the /var/run/docker.sock Unix socket file. Bind-mounting the socket into the container is enough to make the host's Docker daemon accessible; you do not need to separately start Docker in the container.
There are several issues with the RUN service ... line specifically. Docker has a kind of complex setup internally, and some of the things it does aren't normally allowed in a container; that's probably related to the "cannot create directory" error. In any case, a Docker image doesn't persist running processes, so if you were able to start Docker inside the build, it wouldn't still be running when the container eventually ran.
More conceptually, a container doesn't "run services", it is a wrapper around only a single process (and its children). Commands like service or systemctl often won't work the way you expect, and I'd generally avoid them in a Docker context.
I have set up Airflow with docker-compose as described here.
https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html
And there is a airflow task that have to execute docker command like
BashOperator(
task_id='my',
bash_command="""
docker run ..............
""",
dag=dag,
)
This means Docker package required in the airflow docker image, but there is not.
So, I tried to build my own airflow image based docker installed image. like,
FROM apache/airflow:2.0.1 as airflow-image
SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
USER root
RUN apt-get update && apt-get --assume-yes install \
apt-transport-https \
ca-certificates \
curl \
gnupg \
lsb-release
RUN curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
RUN echo $(lsb_release -cs)
RUN echo \
"deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
RUN apt-get update
RUN apt-get --assume-yes install docker-ce docker-ce-cli containerd.io
With an image of above Dockerfile, docker is installed like,
$ whereis docker
docker: /usr/bin/docker /usr/libexec/docker
$ whereis dockerd
dockerd: /usr/bin/dockerd
But I can not make docker daemon started so that task of airflow can execute docker run command.
After I see this ENTRYPOINT and tested several things,
https://github.com/apache/airflow/blob/master/Dockerfile#L535
ENTRYPOINT ["/usr/bin/dumb-init", "--", "/entrypoint"]
I wonder whether I can start docker daemon or not.
Any advice?
Can I make an docker image which is based airflow:2.0.1 and support docker commans?
If I understand you correctly you are trying to run a custom Docker image as a task in Airflow.
There is a DockerOperator available, see an example here.
I think what you tried to do is to host Docker inside of a Docker container. This is not necessary, since the Airflow Docker container should have access to Docker running on your machine.
So if you run Airflow 2.0 make sure to install this Python package apache-airflow-backport-providers-docker in your Airflow Docker container. And include this in your Python DAG file: from airflow.providers.docker.operators.docker import DockerOperator.
Then use the operator, something like this:
dockertask = DockerOperator(
task_id='docker_command',
image='centos:latest',
api_version='auto',
auto_remove=True,
command="/bin/sleep 30",
docker_url="unix://var/run/docker.sock",
network_mode="bridge"
)
Is there a proper way to run Docker commands through a Jenkins containerized service?
I see there are many plugins to support Docker commands in the Jenkins ecosystem, although all of them raise errors because Docker isn't installed in the Jenkins container.
I have a Dockerfile that provides a Jenkins image with a working Docker installation, but to work I have to mount the host's Docker socket:
FROM jenkins/jenkins:lts
USER root
RUN apt-get -y update && \
apt-get -y install sudo \
apt-transport-https \
ca-certificates \
curl \
gnupg-agent \
software-properties-common
RUN add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/debian \
$(lsb_release -cs) \
stable"
RUN apt-get -y update && \
apt-get -y install --allow-unauthenticated \
docker-ce \
docker-ce-cli \
containerd.io
RUN echo "jenkins:jenkins" | chpasswd && adduser jenkins sudo
RUN echo jenkins ALL= NOPASSWD: ALL >> /etc/sudoers
USER jenkins
It can be run like this:
docker run -d -p 8080:8080 -v /var/run/docker.sock:/var/run/docker.sock
This way it's possible to run Docker commands inside the Jenkins container. Although, I am concerned about security: namely this way the Jenkins container can access all the containers running in the host machine, moreover Jenkins is a root user, which I wouldn't like for production.
I seek to run a Jenkins instance within a Kubernetes cluster to support CI and CD pipelines within that cluster, therefore I'm guessing Jenkins must be containerized.
Am I missing something?
I installed docker inside a container running on ubuntu:18.04 to run my nodejs app, I need docker installed inside this container because i need to dockerize an other small app
Her is my Dockerfile
FROM ubuntu:18.04
WORKDIR /app
COPY package*.json ./
# Install Nodejs
RUN apt-get update
RUN apt-get -y install curl wget dirmngr apt-transport-https lsb-release ca-certificates software-properties-common gnupg-agent
RUN curl -sL https://deb.nodesource.com/setup_12.x | bash -
RUN apt-get -y install nodejs
# Install Chromium
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'
RUN apt-get update
RUN apt-get install -y google-chrome-unstable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst \
--no-install-recommends
RUN rm -rf /var/lib/apt/lists/*
# Install Docker
RUN curl -fsSL https:/download.docker.com/linux/ubuntu/gpg | apt-key add -
RUN apt-key fingerprint 0EBFCD88
RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
RUN apt-get update -y
RUN apt-get install -y docker-ce docker-ce-cli containerd.io
RUN npm install
COPY . .
CMD [ "npm", "start" ]
EXPOSE 3000
When the container is up, i docker exec -it app bash.
If i do a service docker start then ps ax, got this
PID TTY STAT TIME COMMAND
115 ? Z 0:00 [dockerd] <defunct>
What can i do to be able to use docker inside the container or is there a docker image not using apk but apt-get ? Because when i need to use it, i got this error :
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
First thing better to use one of the base images, either for node-image and install docker and for docker-image and installed node, instead of creating image from scratch. All you need
FROM node:buster
RUN apt-get update
RUN apt install docker.io -y
RUN docker --version
ENTRYPOINT nohup dockerd >/dev/null 2>&1 & sleep 10 && node /app/app.js
second thing, The error Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?, The reason is you are not starting the docker process in the Dockefile, and also running multiple processes in the container is not recommended, as if Docker process dies you will not know the status, you have to put one process in the background.
CMD nohup dockerd >/dev/null 2>&1 & sleep 10 && node /app/app.js
and run
docker run --privileged -it -p 8000:8000 -v /var/run/docker.sock:/var/run/docker.sock your_image
I created a Dockerfile to run Docker inside Docker:
FROM ubuntu:16.04
RUN apt-get update && \
apt-get install -y \
apt-transport-https \
ca-certificates \
curl \
software-properties-common && \
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - &&\
apt-key fingerprint 0EBFCD88
RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" && \
apt-get update && \
apt-get install -y docker-ce && \
systemctl enable docker
After i launched my container and run docker ps i got:
"Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
i executed the command dockerd inside my container resulted:
Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.6.0: can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.
(exit status 3)
Please advise
The recommendation I received for this was to use the -v parameter in docker run to map the docker socket between containers like this:
-v /var/run/docker.sock:/var/run/docker.sock
If you really want to run a Docker container inside an other Docker container, you should use already existing images provided by Docker (https://hub.docker.com/_/docker) instead of creating your own base image : choose images tagged as dind (docker in docker) or <docker_version>-dind (like 18.09.0-dind). If you want to run your own image (not recommended though), don't forget to run it with --privileged option (that's why you get the error).
Example with docker official images :
# run Docker container running Docker daemon
docker run --privileged --name some-docker -d docker:18.09.0-dind
# run hello-world Docker image inside the Docker container previously started
docker exec -i -t some-docker docker run hello-world
Nevertheless, I agree with #DavidMaze comment and the reference blog post he referred to (Do not use Docker-in-Docker for CI) : Docker-in-Docker should be avoided as much as possible.