I have a simple Dockerfile where I install Jenkins and some plugins:
FROM jenkins/jenkins:2.169-alpine
USER root
RUN apk update \
&& apk add --no-cache curl docker jq tzdata \
&& rm -rf /var/cache/apk/*
COPY plugins.txt /usr/share/jenkins/plugins.txt
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/plugins.txt
There is now a jenkins plugin with a custom patch I would need to include. There's already a PR open for it but it's been months it was not merged and I can't wait anymore, so I'd like to add a step to install a plugin from a branch of my github repo.
I found out that after the jenkins-cli.jar is available (so, not at build time), one can install a plugin in hpi format doing:
java -jar /var/jenkins_home/war/WEB-INF/jenkins-cli.jar \
-auth user:password \
-s http://localhost:8080 install-plugin file://<HPI_PATH>
but it cannot work at build time.
If not possible in the dockerfile, is there an alternative?
First build the plugin.hpi locally and then Use copy or Add in Dockerfile to add plugin into jenkins docker images in build step.
example
Add https://updates.jenkins-ci.org/download/plugins/sonar/2.8.1/sonar.hpi /var/jenkins_home/plugins/
Or
Install sonar plugin using local build hpi file.
Copy sonar.hpi /var/jenkins_home/plugins/
After digging PR, I found the solution here https://github.com/jenkinsci/docker/pull/799
This is not an install from github but it will work out
So you just need to add at the end of your dockerfile (being root and not jenkins user):
RUN /usr/local/bin/install-plugins.sh plugin-name:plugin-version:hpi-url
Related
I would like to say that this is my first container and actually my first JAVA app so maybe I will have basic questions so be lenient, please.
I wrote spring boot app and my colleague has written the frontend part for it in angular. What I would like to achieve is to have "one button/one command" in IntelliJ to create a container containing whole app backend and front end.
What I need to do is:
Clone FE from company repository (I am using ssh key now)
Clone BE from GitHub
Build FE
Copy built FE to static folder in java app
Build BE
Create a container running this app
My current solution is to create "builder" container and there build FE and BE and then copy it to "production" container like this:
#BUILDER
FROM alpine AS builder
WORKDIR /src
# add credentials on build
ARG SSH_PRIVATE_KEY
RUN mkdir /root/.ssh/ \
&& echo "${SSH_PRIVATE_KEY}" > /root/.ssh/id_rsa \
&& echo "github.com,140.82.121.3 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==" >> /root/.ssh/known_hosts \
&& chmod 600 /root/.ssh/id_rsa
# installing dependencies
RUN apk update && apk upgrade && apk add --update nodejs nodejs-npm \
&& npm install -g #angular/cli \
&& apk add openjdk11 \
&& apk add maven \
&& apk add --no-cache openssh \
&& apk add --no-cache git
#cloning repositories
RUN git clone git#code.siemens.com:apcprague/edge/metal-forming-fe.git
RUN git clone git#github.com:bzumik1/metalForming.git
# builds front end
WORKDIR /src/metal-forming-fe
RUN npm install && ng build
# builds whole java app with front end
WORKDIR /src/metalForming
RUN cp -a /src/metal-forming-fe/dist/metal-forming/. /src/metalForming/src/main/resources/static \
&& mvn install -DskipTests=true
#PRODUCTION CONTAINER
FROM adoptopenjdk/openjdk11:debian-slim
LABEL maintainer jakub.znamenacek#siemens.com
RUN mkdir app
RUN ["chmod", "+rwx", "/app"]
WORKDIR /app
COPY --from=builder /src/metalForming/target/metal_forming-0.0.1-SNAPSHOT.jar .
EXPOSE 4200
RUN java -version
CMD java -jar metal_forming-0.0.1-SNAPSHOT.jar
This works but I takes very long time so I guess this is not correct way how to do it. Could anyone point me in correct direction? I was thinking if there is a way how to make maven to all these steps for me but maybe this is totally off.
Also if you will find any problem in my Dockerfile please let me know as I said this is my first Dockerfile so I could overlook something.
EDITED:
BTW does anyone know how can I get rid of this part:
echo "github.com,140.82.121.3 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==" >> /root/.ssh/known_hosts \
it adds GitHub to known_hosts (I also need to add a company repository there). It is because when I run git clone it will ask me if I trust this ... and I have to write yes but I don't know how to do it if it is automatically running in docker and I have no option to write there yes. I have tried yes | git clone ... but this is also not working
a few things:
1, if this runs "slow" on the machine than it will run slow inside a container too.
2, remove --no-cache,* you want to cache everything that is static, because next time when you build those commands will not run where there is no change. Once there is change in one command than that command will rerun instead using the builder cache and also all subsequent commands will have to rerun too.
*UPDATE: I have mistaken "apk update --no-cache" with "docker build --no-cache". I stated wrong that "apk add --no-cache" would mean that command is not cached, because this command is cached on docker builder level. However with this parameter you wouldn't need to delete in a later step the /var/cache/apk/ directory to make you image smaller, but that you wouldn't need to do here, because you are already using multi stage build, so this would not affect your final image size.
One more thing to clarify, all statements in Dockerfile are checked if they changed, if they did not than docker builder uses the cached layer for it and won't run that statement. Exception is ADD and COPY commands, here builder also checks the copied, added files if they changed with checksum. Also if a statement is changed or ADD-ed COPY-ed file(s) changed than that statement is re-run and all subsequent statements re-run too, so you want to put your source code copy statemant as much at the end as it is possible
If you want to disable this cache, do "docker build --no-cache ..." this way all the steps will be re-run that is in the Dockerfile.
3, specify WORKDIR at the top once, if you need to switch directory later use this:
RUN cd /someotherdir && mycommand
Also specifying a Subsequent WORKDIR will be relativ to the previous WORKDIR so it will mess up readibilty what is the (probably) sole purpose of WORKDIR statement.
4, Enable BuildKit:
Either declare environment variable
DOCKER_BUILDKIT=1
or add this to /etc/docker/daemon.json
{ "features": { "buildkit": true } }
BuildKit might not help in this case, but if you do more complex Dockerfiles with more stages Buildkit can run those parallel so overall build will be faster.
5, Do not skip tests with DskipTests=true :)
6, as stated in a comment, do not clone the repo inside the image build, you do not need to do that at all. Just put the Dockerfile in the / of the repo, and COPY the repo files with a Dockerfile command:
COPY . .
First dot is the source that is your current directory on your machine, second dot is the target, the working dir inside the image, /src with your Dockerfile. You build the image and publish it, push it to a docker registry so others can pull it and start using it. If you want more complex stuff building and publishing with a help of a server, look up CI/CD techniques.
I'm trying to set up a local GoCD CI server using docker for both the base server and agents. I can get everything running fine, but issues spring up when I try make sure the agent containers have everything installed in them that I need to build my projects.
I want to preface this with I'm aware that I might not be using these technologies correctly, but I don't know much better atm. If there are better ways of doing things, I'd love to learn.
To start, I'm using the official GoCD docker image and that works just fine.
Creating a blank agent also works just fine.
However, one of my projects requires node, yarn and webpack to be build (good ol' react site).
Of course a standard agent container has nothing but the agent installed on it so I've had a shot using a Dockerfile to install all the tech I need to build my projects.
FROM gocd/gocd-agent-ubuntu-18.04:v19.11.0
SHELL ["/bin/bash", "-c"]
USER root
RUN apt-get update
RUN apt-get install -y git curl wget build-essential ca-certificates libssl-dev htop openjdk-8-jre python python-pip
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - && \
echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
RUN apt-get update && apt-get install -y yarn
# This user is created in the base agent image
USER go
ENV NVM_DIR /home/go/.nvm
ENV NODE_VERSION 10.17.0
RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.1/install.sh | bash \
&& . $NVM_DIR/nvm.sh \
&& nvm install $NODE_VERSION \
&& nvm alias default $NODE_VERSION \
&& nvm use default \
&& npm install -g webpack webpack-cli
ENV NODE_PATH $NVM_DIR/v$NODE_VERSION/lib/node_modules
ENV PATH $NVM_DIR/v$NODE_VERSION/bin:$PATH
This is the current version of this file, but I've been through many many iterations of frustrations where an globally installed npm package is never on the path and thus not conveniently available.
The docker build works fine, its just that in this iteration of the Dockerfile, webpack is not found when the agent tries running a build.
My question is:
Is a Dockerfile the right place to do things like install yarn, node, webpack etc... ?
If so, how can I ensure everything I install through npm is actually available?
If not, what are the current best practices about this?
Any help, thoughts and anecdotes are fully welcomed and appreciated!
Cheers~!
You should separate gocd-server and gocd-agent to various containers.
Pull images:
docker pull gocd/gocd-server:v18.10.0 docker pull
gocd/gocd-agent-alpine-3.8:v18.10.0
Build and run them, check if it's ok. Then connect into bash in agent container
docker exec -it gocd-agent bash
Install the binaries using the alpine package manager.
apk add --no-cache nodejs yarn
Then logout and update the container image. Now you have an image with needed packeges. Also read this article.
You have two options with gocd agents.
The first one is the agent use docker, and create other containers, for any purpose that the pipeline needs. So you can have a lot of agents with this option, and the rules or definitions occurs in the pipeline. The agent only execute.
The second one, is an agent with al kind of program installed you needed. I use this one. For this case, you use a Dockerfile with all, and generate the image for all the agents.
For example i have an agent with gcloud, kubectl, sonar scanner and jmeter, who test with sonar before the deploy, then deploy in gcp, and for last step, it test with jmeter after the deploy.
I am pretty new to Docker. I need to do the following tasks:
Run Jenkins instance in Docker
Configure it to auto-install JobDSL plugin on startup
I wrote DockerFile
FROM java:8
EXPOSE 8080
ADD jenkins.war jenkins.war
ENTRYPOINT ["java","-jar","jenkins.war"]
and then I run docker run ...
But there is one problem I can't use the console but I have to use the console to install the plugin. I tried to solve this problem using & at the end. It did not help. P.S I can't use the jenkins image
Jenkins use a JENKINS_HOME directory where it stores config, jobs and plugins.
One way to achieve what you want is perhaps to set the plugins in this directory before running jenkins.
If you use the official jenkins image, then perhaps you can use a data volume to store that and run docker to use this data volume: docker run -V /your/data/volume:/var/jenkins_home jenkins/jenkins
If you don't want a data volume, and want an image with the plugins, then you can add to you dockerfile something like:
RUN mkdir -p ~/.jenkins/plugins && \
cd ~/.jenkins/plugins && \
wget http://your/plugins/plugins.jpi
finally you can mix a little bit both by creating a shell script that check if the plugins directory exist, and if not get the plugin file, then start jenkins. This shell script would be your image entry point.
NOTE: The file you need to download as plugins is the .jpi file! not the .hpi.
As reference, here an example:
FROM java:8
RUN wget https://updates.jenkins-ci.org/download/war/2.121.2/jenkins.war && \
mkdir -p ~/.jenkins/plugins && \
cd ~/.jenkins/plugins && \
wget https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/job-dsl/1.33/job-dsl-1.33.jpi
ENTRYPOINT ["java","-jar","jenkins.war"]
I am using an alpine linux container and specifically python:3.4-alpine and openjdk:8-jdk-alpine. When I try to execute any script or executable that I have placed in the executable I get Not Found error.
For example. When in the python:3.4-alpine container I want to install jfrog I follow the command here (after I install curl via apk). This command downloads a shell script and pipes it to sh which downloads and creates a jfrog executable with the correct permissions. When I am trying to run this executable I am getting
bin/sh: ./jfrog: not found
update
I discovered that the root user is using bin/ash by default, which I have no idea what it is. So I invoked bin/sh jfrog manually and I get
/ # bin/sh jfrog
jfrog: line 1: ELF: not found
jfrog: line 1: syntax error: unterminated quoted string
Any idea what I am doing wrong? I suspect that it has to do with only root user existing in the container.
I'm not sure but the jfrog executable is dynamically linked, and with ldd jfrog you get :
ldd jfrog
/lib64/ld-linux-x86-64.so.2 (0x55ffb4c8d000)
libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x55ffb4c8d000)
libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x55ffb4c8d000)
As you can see you have libc dependencies, and alpine come with musl.
You can try to add apk add libc6-compat but I'm not sure it will work
the problem is, that jfrog cli was compiled against glibc and alpine linux only provides uclibc. To make it run under alpine its not trivial, you have to install a sandbox that is bigger than then alpine env. https://wiki.alpinelinux.org/wiki/Running_glibc_programs
Another possibility is to compile the jfrog binary yourself in alpine. This Dockerfile worked for me.
FROM golang:alpine
WORKDIR /app/
RUN apk update && apk add git
# checkout the latest tag of jfrog cli
RUN mkdir -p /go/src/github.com/jfrogdev/jfrog-cli-go \
&& git clone https://github.com/JFrogDev/jfrog-cli-go /go/src/github.com/jfrogdev/jfrog-cli-go\
&& cd /go/src/github.com/jfrogdev/jfrog-cli-go \
&& git checkout $(git describe --tags `git rev-list --tags --max-count=1`)
RUN GOOS=linux go get github.com/jfrogdev/jfrog-cli-go/jfrog
FROM alpine
COPY --from=0 /go/bin/jfrog /usr/bin/
ENTRYPOINT ["jfrog"]
The script you are running begins with:
#!/bin/bash
Bash is not included with alpine by default. You can install it with:
apk update && apk add bash
Note that alpine is fairly stripped down by design, so there may be other missing dependencies that you'll need to add to make this script work.
May be too late, but this probably might help someone else.
RUN curl -Lo /usr/bin/jfrog https://api.bintray.com/content/jfrog/jfrog-cli-go/\$latest/jfrog-cli-linux-386/jfrog?bt_package=jfrog-cli-linux-386 \
&& chmod a+x /usr/bin/jfrog
(Click Here for Reference Link)
I have a Dockerfile for a custom Jenkins master like so:
FROM jenkins
MAINTAINER me
USER root
RUN echo 2.0 > /usr/share/jenkins/ref/jenkins.install.UpgradeWizard.state
RUN apt-get update \
&& apt-get install -y sudo \
&& apt-get install -y vim \
&& rm -rf /var/lib/apt/lists/*
RUN echo "jenkins ALL=NOPASSWD: ALL" >> /etc/sudoers
USER jenkins
# COPY plugins.txt /usr/share/jenkins/plugins.txt
# RUN /usr/local/bin/plugins.sh /usr/share/jenkins/plugins.txt
ENV JAVA_OPTS="-Xmx8192m"
ENV JENKINS_OPTS="--handlerCountStartup=100 --handlerCountMax=300"
RUN /usr/local/bin/install-plugins.sh git:2.6.0
Everything works fine until the RUN /usr/local/bin/install-plugins.sh git:2.6.0 line. I get an error installing the plugins:
Creating initial locks...
Analyzing war...
Downloading plugins...
Downloading plugin: git from https://updates.jenkins.io/download/plugins/git/2.6.0/git.hpi
Downloading plugin: git-plugin from https://updates.jenkins.io/download/plugins/git-plugin/2.6.0/git-plugin.hpi
Failed to download plugin: git or git-plugin
WAR bundled plugins:
Installed plugins:
*:
Some plugins failed to download!
Not downloaded: git
The command '/bin/sh -c /usr/local/bin/install-plugins.sh git:2.6.0' returned a non-zero code: 1
Am I doing something wrong or is this an issue with Jenkins/Docker?
For those who are pulling the jenkins image from dockerHub, dont pull:
docker pull jenkins
or
docker pull jenkinsci/jenkins
rather pull the latest version using:
docker pull jenkins/jenkins
This is the latest one according to https://jenkins.io/blog/2018/12/10/the-official-Docker-image/
Your Dockerfile works for me, installs all plugins and builds the image successfully:
Analyzing war...
Downloading plugins...
Downloading plugin: git from https://updates.jenkins.io/download/plugins/git/2.6.0/git.hpi
> git depends on workflow-scm-step:1.14.2,mailer:1.17,matrix-project:1.7.1,ssh-credentials:1.12,parameterized-trigger:2.4;resolution:=optional,scm-api:1.2,token-macro:1.11;resolution:=optional,promoted-builds:2.27;resolution:=optional,credentials:2.1.4,git-client:1.21.0
Downloading plugin: workflow-scm-step from https://updates.jenkins.io/download/plugins/workflow-scm-step/latest/workflow-scm-step.hpi
...
Removing intermediate container 4f895c203944
Successfully built 31d58d1f586f
Try docker build --no-cache in case there's an issue with one of the layers in your image cache, or set up an automated build on Docker Hub and build it on Docker's servers.
I recall having problems installing with that script myself. Instead, I used the following:
RUN install-plugins.sh \
disable-failed-job \
disk-usage \
greenballs \
...
And hopefully it doesn't make a difference for this, but I have my plugin install inside of the root portion of my Dockerfile, before dropping back to running commands as USER jenkins.
Dockerfile
FROM jenkins/jenkins:latest
ENV CURL_OPTIONS -sSfLk
ENV JENKINS_OPTS --httpPort=-1
The curl timeouts for downloading plugins were insufficient in some cases, that was just fixed for image 2.19.1, and it is now configurable too using CURL_CONNECTION_TIMEOUT and other options
I had the same problem on OS X.
In my case the problem was caused by a bad DNS configuration (obtained by DHCP). When I changed the DNS to Googles DNS 8.8.8.8 it all worked perfectly.
I encountered error messages such as:
Failed to resolve host name "ftp.icm.edu.pl". Perhaps you need to configure HTTP proxy
I had a very similar issue and the solution for me was to specify the proxy within the Docker file prior to plugin install. Below is the snippet of my Dockerfile
FROM jenkins:latest
MAINTAINER Jose Estrada
USER root
ENV JAVA_OPTS="--handlerCountStartup=100 --handlerCountMax=300 --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war -Dhttps.proxyHost=proxy-wsa.esl.cisco.com -Dhttps.proxyPort=80"
ENV http_proxy <PROXY Settings>
ENV https_proxy <PROXY Settings>
RUN /usr/local/bin/install-plugins.sh cisco-spark-notifier:latest
This could be a DNS issue. Please restart docker daemon and try. (sudo service docker restart)