Dockerfile build inconsistency in a Jenkinsfile (Connection failed error) - docker

In my Jenkins CICD pipeline, I have stage where I build my backend image.
stage('Build Backend Image') {
steps {
sh '''
docker login -u <id> -p <pw> <dockerhub>
docker -v
docker build \
--label com.build.stage=3 \
-f ./backend/Dockerfile \
-t <dockerhub>/back_stage_1:build .
'''
sh "docker tag <dockerhub>/back_stage_1:build <dockerhub>/ack_stage_1:${GIT_COMMIT}"
}
}
The Dockerfile tied to this stage looks like:
#Using ubuntu as our base
FROM ubuntu:latest
#Install commands in ubuntu, including pymongo for DB handling
RUN apt-get update -y
RUN apt-get install -y python3 python3-pip python3-dev build-essential
#Unsure of COPY command's purpose, but WORKDIR points to /backend
COPY /backend/ /backend
WORKDIR /backend/
RUN pip3 install requirements.txt
#Run order for starting up the backend
# ENTRYPOINT ["python3"]
CMD ["python3", "-u", "app.py"]
Just a simple Dockerfile that installs the dependencies.
This step sometimes works fine but sometimes gets stuck.
The error that I am getting looks like:
Step 5/11 : RUN apt-get install -y python3 python3-pip python3-dev build-essential
---> Running in ba358993dc8b
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
...
...
0 upgraded, 135 newly installed, 0 to remove and 2 not upgraded.
Need to get 117 MB of archives.
After this operation, 390 MB of additional disk space will be used.
Err:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 liblocale-gettext-perl amd64 1.07-3build2
Connection failed [IP: ...]
It just keeps giving me Connection failed error until I abort it.
When it works, it just successfully installs all packages and goes through the entire steps successfully.
I don't understand because it's too inconsistent and fails even though I did not change anything in the Dockerfile or Jenkins. I feel like it has something to do with some Jenkins' trait that I don't know.
Any help?

Related

dockerfile does not build due to failed size validation

My dockerfile used to build successfully.
I tried to build today (5 days after successful build) with docker build -t fv ., and kept getting the following error:
failed commit on ref "layer-sha256:7a3de07a56633b9096304d02c47f097f3e28ae6c6dd442d1e7c4d26452ecd90a": "layer-sha256:7a3de07a56633b9096304d02c47f097f3e28ae6c6dd442d1e7c4d26452ecd90a" failed size validation: 581433721 != 600361569: failed precondition
any suggestions what this means and how to correct?
my dockerfile is:
FROM rocker/verse
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends build-essential libpq-dev python3.8 python3-pip python3-setuptools python3-dev
RUN pip3 install --upgrade pip
ADD . ./home/rstudio
ADD requirements.txt .
ADD install_packages.r .
# Miniconda and dependencies
RUN cd /tmp/ && \
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh && \
bash Miniconda3-latest-Linux-x86_64.sh -b -p $HOME/miniconda3 && \
/root/miniconda3/condabin/conda install -y python=3.7
ENV PATH=$PATH:/root/miniconda3/bin
#RUN npm install phantomjs-prebuilt --phantomjs_cdnurl=http://cnpmjs.org/downloads
# installing python libraries
RUN pip3 install -r requirements.txt
# installing r libraries
RUN Rscript install_packages.r
another reference I got was:
=> => sha256:7a3de07a56633b9096304d02c47f097f3e28ae6c6dd442d1e7c4d26452ecd90a 580.97MB / 600.36MB 1150.8s
------
> [ 1/10] FROM docker.io/rocker/verse#sha256:3b417b991a32cc8bf9c1fa173ec976a5cc65522a76918df61b5c6cf6261e63a5:
Would this be because of an issue with the base image pulled?
I found that docker build would fail with this error, but it would work if I first pulled the failing image with docker pull <image> and then ran docker build.
this was due to security encryption from my local ip.
when tethering, was able to generate the docker image with non problems
On my side, I got something like below
------
> [1/3] FROM docker.io/library/python#sha256:10fc14aa6ae69f69e4c953cffd9b0964843d8c163950491d2138af891377bc1d:
------
failed commit on ref "layer-sha256:049db2c7eb8a5bd3833cac2f58c6c72b481f1a0288a8b20527529c4970b52762": "layer-sha256:049db2c7eb8a5bd3833cac2f58c6c72b481f1a0288a8b20527529c4970b52762" failed size validation: 311296 != 3056504: failed precondition
On my side, I managed to solve this by disconnecting from a VPN I was connected to.

Docker build error while building a kafka-connect 5.5.0 docker image

Getting this error while building a custom docker image for kafka-connect:5.5.0.
I do have "apt-get update" in necessary locations. Anyone has a clue on how to resolve this?
W: Duplicate sources.list entry
https://s3-us-west-2.amazonaws.com/staging-confluent-packages-5.5.0/deb/5.5/
stable/main amd64 Packages
(/var/lib/apt/lists/s3-us-west-2.amazonaws.com_staging-confluent-packages-5.5.0_deb_5.5_dists_stable_main_binary-amd64_Packages.gz)
W: You may want to run apt-get update to correct these problems
Basically, I should have approached installing any new packages on top of the pre-built image rather than adding them inside the original image.
My docker file to install influxDB connectors in Kafka-Connect ended up looking like this.
ARG DOCKER_TAG=5.5.0
FROM confluentinc/cp-kafka-connect:${DOCKER_TAG}
ENV COMPONENT=kafka-connect
USER root
RUN echo "===> Installing InfluxDB Connectors"
RUN confluent-hub install confluentinc/kafka-connect-influxdb:latest --no-prompt \
&& echo "===> Cleaning up ..." \
&& apt-get clean && rm -rf /tmp/* /var/lib/apt/lists/*
And Influx connectors are now working in my production.

Unable to locate package when installing Java in docker

I'm trying to install Java in a docker image in order to make working with firebase a bit easier.
When running the installation command I get the following error:
Unable to locate package openjdk-11-jre-headless
Here is the docker image definition:
FROM node:14.5
ADD . /src
WORKDIR /src
# Install OpenJDK-11
RUN apt-get update && \
apt-get install -y openjdk-11-jre-headless && \
apt-get clean;
RUN npm i -g firebase-tools
RUN firebase --version
RUN firebase emulators:start
EXPOSE 4400 4500 5000 5001 8001 8080 8085 9000
I guess the node image is based on debian which does not have the repository by default.
Adding the following got it working for me:
RUN echo 'deb http://ftp.debian.org/debian stretch-backports main' | tee /etc/apt/sources.list.d/stretch-backports.list

Why does docker rebuild all layers every time I change build args

I have a docker file which has a lot of layers. At the top of the file I have some args like
FROM ubuntu:18.04
ARGS USER=test-user
ARGS UID=1000
#ARGS PW=test-user
# Then several Layers which does not use any ARGS. Example
LABEL version="1.0"
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
RUN mkdir ~/mapped-volume
RUN apt-get update && apt-get install -y wget bzip2 ca-certificates build-essential curl git-core htop pkg-config unzip unrar tree freetds-dev vim \
sudo nodejs npm net-tools flex perl automake bison libtool byacc
# And so on
# And finally towards the end
# Setup User
RUN useradd -m -d /home/${USER} --uid ${UID} -G sudo -s /bin/bash ${USER}
# && echo "${USER}:${PW}" | chpasswd
# Couple of more commands to change dir, entry point etc. Example
When I build this docker file with any arg value different from the last build and/or after small changes in the last two layers, the build builds everything again. It does not use cached layer. The command I use to build is something like this
docker build --build-arg USER=new-user --build-arg UID=$UID -t my-image:1.0 .
And every time I change the values the build goes all through again. With a truncated top like below
UID -t my-image:1.0 .
Sending build context to Docker daemon 44.54kB
Step 1/23 : FROM ubuntu:18.04
---> ccc6e87d482b
Step 2/23 : ARG USER=ml-user
---> Using cache
---> 6c0c5d5c5056
Step 3/23 : ARG UID=1000
---> Using cache
---> b25867c282c7
Step 4/23 : LABEL version="1.0"
---> Running in 1ffff70d56c1
Removing intermediate container 1ffff70d56c1
---> 0f1277def3ca
Step 5/23 : ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
---> Running in 49d08c41b233
Removing intermediate container 49d08c41b233
---> f5b345573c1f
Step 6/23 : RUN mkdir ~/mapped-volume
---> Running in e4f8a5956450
Removing intermediate container e4f8a5956450
---> 1b22731d9051
Step 7/23 : RUN apt-get update && apt-get install -y wget bzip2 ca-certificates build-essential curl git-core htop pkg-config unzip unrar tree freetds-dev vim sudo nodejs npm net-tools flex perl automake bison libtool byacc
---> Running in ffc297de6234
Get:1 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
So from step 7 it keeps doing all steps without using the cache of that layer which should have a bunch of packages
Why? How can I stop this? Previously when I did not have args, this layer and other layers used to be picked up from cache.
Move your args to just before you need them. Docker does not replace args in the RUN commands before running them. Instead, the args are passed as environment variables and expanded by the shell within the temporary container. Because of that, a change to an arg is a change to the environment, and a miss of the build cache for that step. Once one step misses the cache, all following steps must be rebuilt.
FROM ubuntu:18.04
# Then several Layers which does not use any ARGS. Example
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
RUN mkdir ~/mapped-volume
RUN apt-get update && apt-get install -y wget bzip2 ca-certificates build-essential curl git-core htop pkg-config unzip unrar tree freetds-dev vim \
sudo nodejs npm net-tools flex perl automake bison libtool byacc
# And so on
# And finally towards the end
# Setup User
ARGS USER=test-user
ARGS UID=1000
RUN useradd -m -d /home/${USER} --uid ${UID} -G sudo -s /bin/bash ${USER}
# && echo "${USER}:${PW}" | chpasswd
# Couple of more commands to change dir, entry point etc. Example
LABEL version="1.0"
Also, labels, environment variables that aren't needed at build time, exposed ports, and any other meta data is often best left to the end of the Dockerfile since they have minimal impact on build time and there's no need to miss the cache when they change.

docker error Unable to locate package cowsay

When I am running the build command:
$ docker build -t cowsay .
Sending build context to Docker daemon 3.072 kB
Step 1 : FROM
Unknown instruction: FROM
dev#ub:~/cowsay$ docker build -t cowsay .
Sending build context to Docker daemon 3.072 kB
Step 1 : FROM ubuntu:14.04
---> 90d5884b1ee0
Step 2 : RUN apt-get -y install cowsay
---> Running in 587aaba2824b
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package cowsay
The command '/bin/sh -c apt-get -y install cowsay' returned a non-zero code: 100
Content of Dockerfile is as follow:
FROM ubuntu:14.04
RUN apt-get -y install cowsay
RUN apt-get -y install fortune
ENTRYPOINT ["/usr/games/cowsay"]
CMD ["Docker is so awesomoooooooo!"]
ONBUILD RUN /usr/games/fortune | /usr/games/cowsay
How can I avoid this error message?
Try and add RUN apt-get update first.
Once the packages are updated, you can install, for instance, cowsay.
See for instance this Dockerfile as an example of RUN apt-get commands:
FROM debian:jessie
RUN apt-get update && apt-get install -y cowsay
COPY docker.cow /usr/share/cowsay/cows/docker.cow
ENTRYPOINT ["/usr/games/cowsay","-f","docker"]
CMD ["moby","dock"]

Resources