Can you install command-line packages in jib docker image? - docker

I need to install command line tools like jq, curl etc in the docker image created by maven jib plugin. How can I achieve this? Any help would be greatly appreciated. Thanks.

As explained in the other answer, using a base image customized with pre-installed tools that rarely change is a good solution.
Alternatively, you may put curl using Jib's <extraDirectories> feature, which enables adding arbitrary files to the target image. Check the Maven and Gradle docs for more details. As explained in the docs, you will also need to configure <permissions> to set executable bits to curl.
If you prefer, you could even set up your Maven or Gradle builds to download curl and unpack it. Here's an example Jib setup (showing both Maven and Gradle) from the Jib repository.

Adding a reference Dockerfile and you can build your own base image by creating your Dockerfile and then build it.
FROM openjdk:8-jdk-alpine
RUN apk add --no-cache curl tar bash procps
# Downloading and installing Maven
ARG MAVEN_VERSION=3.6.1
ARG USER_HOME_DIR="/root"
ARG SHA=b4880fb7a3d81edd190a029440cdf17f308621af68475a4fe976296e71ff4a4b546dd6d8a58aaafba334d309cc11e638c52808a4b0e818fc0fd544226d952544
ARG BASE_URL=https://apache.osuosl.org/maven/maven-3/${MAVEN_VERSION}/binaries
RUN mkdir -p /usr/share/maven /usr/share/maven/ref \
&& echo "Downlaoding maven" \
&& curl -fsSL -o /tmp/apache-maven.tar.gz ${BASE_URL}/apache-maven-${MAVEN_VERSION}-bin.tar.gz \
\
&& echo "Checking download hash" \
&& echo "${SHA} /tmp/apache-maven.tar.gz" | sha512sum -c - \
\
&& echo "Unziping maven" \
&& tar -xzf /tmp/apache-maven.tar.gz -C /usr/share/maven --strip-components=1 \
\
&& echo "Cleaning and setting links" \
&& rm -f /tmp/apache-maven.tar.gz \
&& ln -s /usr/share/maven/bin/mvn /usr/bin/mvn
ENV MAVEN_HOME /usr/share/maven
ENV MAVEN_CONFIG "$USER_HOME_DIR/.m2"
# Downloading and installing Gradle
# 1- Define a constant with the version of gradle you want to install
ARG GRADLE_VERSION=4.0.1
# 2- Define the URL where gradle can be downloaded from
ARG GRADLE_BASE_URL=https://services.gradle.org/distributions
# 3- Define the SHA key to validate the gradle download
# obtained from here https://gradle.org/release-checksums/
ARG GRADLE_SHA=d717e46200d1359893f891dab047fdab98784143ac76861b53c50dbd03b44fd4
# 4- Create the directories, download gradle, validate the download, install it, remove downloaded file and set links
RUN mkdir -p /usr/share/gradle /usr/share/gradle/ref \
&& echo "Downlaoding gradle hash" \
&& curl -fsSL -o /tmp/gradle.zip ${GRADLE_BASE_URL}/gradle-${GRADLE_VERSION}-bin.zip \
\
&& echo "Checking download hash" \
&& echo "${GRADLE_SHA} /tmp/gradle.zip" | sha256sum -c - \
\
&& echo "Unziping gradle" \
&& unzip -d /usr/share/gradle /tmp/gradle.zip \
\
&& echo "Cleaning and setting links" \
&& rm -f /tmp/gradle.zip \
&& ln -s /usr/share/gradle/gradle-${GRADLE_VERSION} /usr/bin/gradle
# 5- Define environmental variables required by gradle
ENV GRADLE_VERSION 4.0.1
ENV GRADLE_HOME /usr/bin/gradle
ENV GRADLE_USER_HOME /cache
ENV PATH $PATH:$GRADLE_HOME/bin
VOLUME $GRADLE_USER_HOME
CMD [""]
Ref:- https://docs.docker.com/engine/reference/builder/
Once your custom image is ready, push it to Registry and then reference it in jib in following manner.
mvn compile jib:build \
-Djib.from.image=customImage

Related

Clone git repo from github with dockerfile

I'm trying to clone GitHub repo with below docker file after installing java,maven
FROM openjdk:8-jdk-alpine
LABEL WebAutomation Test <waaanjula#gmail.com>
RUN apk add --no-cache curl tar bash procps
# Downloading and installing Maven
# 1- Define a constant with the version of maven you want to install
ARG MAVEN_VERSION=3.8.1
# 3- Define the SHA key to validate the maven download
ARG SHA=0ec48eb515d93f8515d4abe465570dfded6fa13a3ceb9aab8031428442d9912ec20f066b2afbf56964ffe1ceb56f80321b50db73cf77a0e2445ad0211fb8e38d
# 4- Define the URL where maven can be downloaded from
ARG BASE_URL=https://apache.osuosl.org/maven/maven-3/${MAVEN_VERSION}/binaries
#5- Create the directories, download maven, validate the download, install it, remove downloaded file and set links
RUN mkdir -p /usr/share/maven /usr/share/maven/ref \
&& echo "Downlaoding maven" \
&& curl -fsSL -o /tmp/apache-maven.tar.gz ${BASE_URL}/apache-maven-${MAVEN_VERSION}-bin.tar.gz \
\
&& echo "Checking download hash" \
&& echo "${SHA} /tmp/apache-maven.tar.gz" | sha512sum -c - \
\
&& echo "Unziping maven" \
&& tar -xzf /tmp/apache-maven.tar.gz -C /usr/share/maven --strip-components=1 \
\
&& echo "Cleaning and setting links" \
&& rm -f /tmp/apache-maven.tar.gz \
&& ln -s /usr/share/maven/bin/mvn /usr/bin/mvn
# 6- Define environmental variables required by Maven, like Maven_Home directory and where the maven repo is located
ENV MAVEN_HOME /usr/share/maven
ENV MAVEN_CONFIG "$USER_HOME_DIR/.m2"
# # Install dependencies required to git clone.
RUN apk update && \
apk add --update git && \
apk add --update openssh
# 1. Create the SSH directory.
# 2. Populate the private key file.
# 3. Set the required permissions.
# 4. Add github to our list of known hosts for ssh.
RUN mkdir -p /root/.ssh/
ADD id_rsa /root/.ssh/id_rsa
ADD id_rsa.pub /root/.ssh/id_rsa.pub
RUN chmod -R 700 /root/.ssh/ && \
touch ~/.ssh/known_hosts && \
ssh-keyscan -t rsa github.com >> /root/.ssh/known_hosts
RUN mkdir /usr/AutomationProject/ && \
chmod -R 600 /usr/AutomationProject/
# # Clone a repository (MatificWebAutomation project)
RUN git clone git#github.com:anjulaw/Selenium_Keyword_driven_Example.git /usr/AutomationProject
CMD [""]
and I'm getting below error
"git#github.com: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists."
This is my first time using dockerfiles, but from what I have read (and taken from working configs) I cannot see why this doesn't work.
My id_rsa is in the same folder as my dockerfile and is a copy of my local key which can clone this repo no problem.
You need to create a folder with keys in the local project folder, transfer there the generated keys that are already installed in the remote repository.
# SSH Keys
ADD .ssh/id_rsa /root/.ssh/id_rsa
ADD .ssh/id_rsa.pub /root/.ssh/id_rsa.pub
RUN chmod 600 /root/.ssh/id_rsa &&\
chmod 600 /root/.ssh/id_rsa.pub
It turns out that after each build, the key will be generated, which means that after each build of this Dockerfile, you will have to reconfigure the keys on the remote repository 💁

Edit / hide Nginx Server header under Alpine Linux

When I use curl --head to test my website, it returns the server information.
I followed this tutorial to hide the nginx server header.
But when I run the command yum install nginx-module-security-headers
, it returns yum: not found.
I also tried apk add nginx-module-security-headers, and it shows that the package is missing.
I have used nginx:1.17.6-alpine as my base docker image. Does anyone know how to hide the server from header under this Alpine?
I think I have an easier solution here: https://gist.github.com/hermanbanken/96f0ff298c162a522ddbba44cad31081. Big thanks to hermanbanken on Github for sharing this gist.
The idea is to create a multi stage build with the nginx alpine image to be a base for compiling the module. This turns into the following Dockerfile:
ARG VERSION=alpine
FROM nginx:${VERSION} as builder
ENV MORE_HEADERS_VERSION=0.33
ENV MORE_HEADERS_GITREPO=openresty/headers-more-nginx-module
# Download sources
RUN wget "http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz" -O nginx.tar.gz && \
wget "https://github.com/${MORE_HEADERS_GITREPO}/archive/v${MORE_HEADERS_VERSION}.tar.gz" -O extra_module.tar.gz
# For latest build deps, see https://github.com/nginxinc/docker-nginx/blob/master/mainline/alpine/Dockerfile
RUN apk add --no-cache --virtual .build-deps \
gcc \
libc-dev \
make \
openssl-dev \
pcre-dev \
zlib-dev \
linux-headers \
libxslt-dev \
gd-dev \
geoip-dev \
perl-dev \
libedit-dev \
mercurial \
bash \
alpine-sdk \
findutils
SHELL ["/bin/ash", "-eo", "pipefail", "-c"]
RUN rm -rf /usr/src/nginx /usr/src/extra_module && mkdir -p /usr/src/nginx /usr/src/extra_module && \
tar -zxC /usr/src/nginx -f nginx.tar.gz && \
tar -xzC /usr/src/extra_module -f extra_module.tar.gz
WORKDIR /usr/src/nginx/nginx-${NGINX_VERSION}
# Reuse same cli arguments as the nginx:alpine image used to build
RUN CONFARGS=$(nginx -V 2>&1 | sed -n -e 's/^.*arguments: //p') && \
sh -c "./configure --with-compat $CONFARGS --add-dynamic-module=/usr/src/extra_module/*" && make modules
# Production container starts here
FROM nginx:${VERSION}
COPY --from=builder /usr/src/nginx/nginx-${NGINX_VERSION}/objs/*_module.so /etc/nginx/modules/
.... skipped inserting config files and stuff ...
# Validate the config
RUN nginx -t
Alpine repo probably doesn't have the ngx_security_headers module but, the mentioned tutorial also provides an option of using Headers More module. You should be able to install this module in your alpine distro using the command:
apk add nginx-mod-http-headers-more
Hope it helps.
Source
I found the alternate solution. The reason that it shows binary not compatible is because I have one nginx pre-installed under the target route, and it is not compatible with the header-more module I am using. That means I cannot simply install the third party library from Alpine package.
So I prepare a clean Alpine OS, and follow the GitHub repository to build Nginx from the source with additional feature. The path of build result is the prefix path you specified.

CSV data set config- Jmeter Docker

I am creating a Jmeter docker container. Test inputs are driven from CSV(data set config). What should be filename path that i need set in the script
Given you're creating a JMeter docker container you should be aware where to drop the CSV file. Normally it is recommended to use relative paths to the CSV files in scripts for better maintainability or for distributed testing
So I would suggest using Docker COPY instruction in order to transfer your CSV file to JMeter's "bin" folder and use just filename in the CSV Data Set Config
Given the example Dockerfile from the Make Use of Docker with JMeter - Learn How article:
# 1
FROM alpine:3.6
# 2
LABEL maintainer=”vincenzo.marrazzo#domain.personal>
# 3
ARG JMETER_VERSION="5.0"
# 4
ENV JMETER_HOME /opt/apache-jmeter-${JMETER_VERSION}
ENV JMETER_BIN ${JMETER_HOME}/bin
ENV MIRROR_HOST http://mirrors.ocf.berkeley.edu/apache/jmeter
ENV JMETER_DOWNLOAD_URL ${MIRROR_HOST}/binaries/apache-jmeter-${JMETER_VERSION}.tgz
ENV JMETER_PLUGINS_DOWNLOAD_URL http://repo1.maven.org/maven2/kg/apc
ENV JMETER_PLUGINS_FOLDER ${JMETER_HOME}/lib/ext/
# 5
RUN apk update \
&& apk upgrade \
&& apk add ca-certificates \
&& update-ca-certificates \
&& apk add --update openjdk8-jre tzdata curl unzip bash \
&& cp /usr/share/zoneinfo/Europe/Rome /etc/localtime \
&& echo "Europe/Rome" > /etc/timezone \
&& rm -rf /var/cache/apk/* \
&& mkdir -p /tmp/dependencies \
&& curl -L --silent ${JMETER_DOWNLOAD_URL} > /tmp/dependencies/apache-jmeter-${JMETER_VERSION}.tgz \
&& mkdir -p /opt \
&& tar -xzf /tmp/dependencies/apache-jmeter-${JMETER_VERSION}.tgz -C /opt \
&& rm -rf /tmp/dependencies
# 6
RUN curl -L --silent ${JMETER_PLUGINS_DOWNLOAD_URL}/jmeter-plugins-dummy/0.2/jmeter-plugins-dummy-0.2.jar -o ${JMETER_PLUGINS_FOLDER}/jmeter-plugins-dummy-0.2.jar
RUN curl -L --silent ${JMETER_PLUGINS_DOWNLOAD_URL}/jmeter-plugins-cmn-jmeter/0.5/jmeter-plugins-cmn-jmeter-0.5.jar -o ${JMETER_PLUGINS_FOLDER}/jmeter-plugins-cmn-jmeter-0.5.jar
# 7
ENV PATH $PATH:$JMETER_BIN
# 8
COPY launch.sh /
COPY somefile.csv $JMETER_BIN
#9
WORKDIR ${JMETER_HOME}
#10
ENTRYPOINT ["/launch.sh"]
So this line:
COPY somefile.csv $JMETER_BIN
will transfer your CSV file into "bin" folder of your JMeter installation therefore you will be able to refer it just as somefile.csv
you should set the file path to the path seen from docker that is related to the volume.:
https://docs.docker.com/storage/volumes/#choose-the--v-or---mount-flag
For example:
docker run -v "DIR of machine":"DIR inside docker container"

Gradle docker container ignores cache when started from Jenkinsfile

I am running a build in a gradle container with a volume for the cache, but gradle does not make use of the downloaded dependencies in the cache for the subsequent builds.
Here's the dockerfile for the gradle image:
FROM **custom image base**
# Install the Java Development Kit
RUN apk --no-cache add openjdk8=8.131.11-r2
CMD ["gradle"]
ENV GRADLE_HOME /opt/gradle
ENV GRADLE_VERSION 4.6
ARG GRADLE_DOWNLOAD_SHA256=98bd5fd2b30e070517e03c51cbb32beee3e2ee1a84003a5a5d748996d4b1b915
RUN set -o errexit -o nounset \
&& echo "Installing build dependencies" \
&& apk add --no-cache --virtual .build-deps \
ca-certificates \
openssl \
unzip \
\
&& echo "Downloading Gradle" \
&& wget -O gradle.zip "https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip" \
\
&& echo "Checking download hash" \
&& echo "${GRADLE_DOWNLOAD_SHA256} *gradle.zip" | sha256sum -c - \
\
&& echo "Installing Gradle" \
&& unzip gradle.zip \
&& rm gradle.zip \
&& mkdir /opt \
&& mv "gradle-${GRADLE_VERSION}" "${GRADLE_HOME}/" \
&& ln -s "${GRADLE_HOME}/bin/gradle" /usr/bin/gradle \
\
&& apk del .build-deps \
\
&& echo "Adding gradle user and group" \
&& addgroup -S -g 1000 gradle \
&& adduser -D -S -G gradle -u 1000 -s /bin/ash gradle \
&& mkdir /home/gradle/.gradle \
&& chown -R gradle:gradle /home/gradle \
\
&& echo "Symlinking root Gradle cache to gradle Gradle cache" \
&& ln -s /home/gradle/.gradle /root/.gradle
# Create Gradle volume
USER gradle
VOLUME "/home/gradle/.gradle"
WORKDIR /home/gradle
RUN set -o errexit -o nounset \
&& echo "Testing Gradle installation" \
&& gradle --version
In the Jenkinsfile I have the build stage declared like this:
stage('Build') {
docker.image('custom-gradle').withRun('-v gradle-cache:/home/gradle/.gradle') { c ->
docker.image('custom-gradle').inside {
sh './scripts/build.py -br ' + branchName
sh 'cp build/libs/JARFILE*.jar build/libs/JARFILE.jar'
}
}
}
The 'gradle-cache' volume is a volume that was created with the docker volume create command.
The python script just runs a gradle command:
gradle clean assemble javaDoc
When I inspect the gradle-cache volume data on the host machine it contains the following files/folders:
4.6 buildOutputCleanup caches daemon native
So the build successfully writes to the cache volume, but appears not to read from it, re-downloading every dependency for every build.
How can I get gradle to use these downloaded dependencies?
UPDATE
stage('Build Bag End') {
docker.image('custom-gradle').inside('-v gradle-cache:/home/gradle/.gradle') {
sh './scripts/build.py -br ' + branchName
sh 'cp build/libs/JARFILE*.jar build/libs/JARFILE.jar'
}
}
So I found the .inside() command also supports parameters but it still won't read from the cache; only write to it.
I believe that Jenkins Docker Plugin always runs containers as the jenkins user. It looks like your gradle image is assuming that it is running as the gradle user. You might try adding the following option:
-e GRADLE_USER_HOME=/home/gradle/.gradle
Though you might run into permission issues with that.

change gradle Dockerfile to be executed as root user

I am working in gitlab and want to use gradle to build my java project, but I ran into this bug with gitlab runner: https://gitlab.com/gitlab-org/gitlab-runner/issues/2570
One comment is: I can confirm that it works in v9.1.3 but v9.2.0 is broken. Only when I use root user inside container I can proceed. That really should be fixed, because that this regression is seriously impacting security.
So my question is on which places I have to change the Dockerfile to execute as root user? https://github.com/keeganwitt/docker-gradle/blob/b0419babd3271f6c8e554fbc8bbd8dc909936763/jdk8-alpine/Dockerfile
So my idea is to change the dockerfile that it is executed as root push it to my registry and use it inside gitlab. But I am not so much into linux/docker that I know where the user is defined in the file. Maybe I am totally wrong?
build_java:
image: gradle:4.4.1-jdk8-alpine-root
stage: build_java
script:
- gradle build
artifacts:
expire_in: 1 hour # Workaround to delete artifacts after build, we only artifacts it to keep it between stages (but not after the build)
paths:
- build/
- .gradle/
Dockerfile
FROM openjdk:8-jdk-alpine
CMD ["gradle"]
ENV GRADLE_HOME /opt/gradle
ENV GRADLE_VERSION 4.4.1
ARG GRADLE_DOWNLOAD_SHA256=e7cf7d1853dfc30c1c44f571d3919eeeedef002823b66b6a988d27e919686389
RUN set -o errexit -o nounset \
&& echo "Installing build dependencies" \
&& apk add --no-cache --virtual .build-deps \
ca-certificates \
openssl \
unzip \
\
&& echo "Downloading Gradle" \
&& wget -O gradle.zip "https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip" \
\
&& echo "Checking download hash" \
&& echo "${GRADLE_DOWNLOAD_SHA256} *gradle.zip" | sha256sum -c - \
\
&& echo "Installing Gradle" \
&& unzip gradle.zip \
&& rm gradle.zip \
&& mkdir /opt \
&& mv "gradle-${GRADLE_VERSION}" "${GRADLE_HOME}/" \
&& ln -s "${GRADLE_HOME}/bin/gradle" /usr/bin/gradle \
\
&& apk del .build-deps \
\
&& echo "Adding gradle user and group" \
&& addgroup -S -g 1000 gradle \
&& adduser -D -S -G gradle -u 1000 -s /bin/ash gradle \
&& mkdir /home/gradle/.gradle \
&& chown -R gradle:gradle /home/gradle \
\
&& echo "Symlinking root Gradle cache to gradle Gradle cache" \
&& ln -s /home/gradle/.gradle /root/.gradle
# Create Gradle volume
USER gradle
VOLUME "/home/gradle/.gradle"
WORKDIR /home/gradle
RUN set -o errexit -o nounset \
&& echo "Testing Gradle installation" \
&& gradle --version
EDIT:
Okay how to use gradle in docker after it is downloaded as image and available in gitlab.
build_java:
image: docker:dind
stage: build_java
script:
- docker images
- docker login -u _json_key -p "$(echo $GCR_SERVICE_ACCOUNT | base64 -d)" https://eu.gcr.io
- docker pull eu.gcr.io/test/gradle:4.4.1-jdk8-alpine-root
- docker images
- ??WHAT COMMAND TO CALL GRADLE BUILD??

Resources