Flaky pipeline on Jenkins with docker agents - docker

Question

We have several Jenkins agents running as physical machines.
So far I ran my Jenkins pipeline on the agents themselves, I now try to move the building and test execution into docker containers using the Jenins docker plugin.
Below you can find a simplified version of our Jenkinsfile which uses gradle to build, test, and package a Java Spring Boot application.
node {
stage('Preparation') {
cleanWs()
checkout scm
notifyBitbucket()
}
}
pipeline {
agent {
docker {
image "our-custom-registry.com/jenkins-build:latest"
registryUrl 'https://our-custom-registry.com'
registryCredentialsId '...'
alwaysPull true
args "-u jenkins -v /var/run/docker.sock:/var/run/docker.sock" // the pipeline itself required docker
}
}
stages {
stage('Build') {
steps {
sh './gradlew assemble classes testClasses'
}
}
stage('Test') {
parallel {
stage('Unit Tests') {
when { expression { return build_params.ENABLE_UNITTEST } }
steps {
sh './gradlew test'
junit UNIT_TEST_RESULT_DIR
}
}
stage('Integration Tests') {
when { expression { return build_params.ENABLE_INTEGRATION } }
steps {
sh './gradlew integration'
junit INTEGRATION_TEST_RESULT_DIR
}
}
}
}
stage('Finalize') {
stage('Docker Push') {
when { expression { return build_params.ENABLE_DOCKER_PUSH } }
steps {
sh './gradlew pushDockerImage'
}
}
}
}
post {
cleanup {
cleanWs()
}
always {
script {
node {
currentBuild.result = currentBuild.result ?: 'SUCCESS'
notifyBitbucket()
}
}
}
}
}
Below is the Dockerfile I use for the build image. As you can see I manually create a Jenkins user and add the to the docker groups (unfortunately the GID is 998 or 999 depending on the Jenkins agent).
FROM openjdk:8-jdk-stretch
USER root
# prerequisites:
# - a user and a group Jenkins with UID/GID=1001 exist
# - the user home is /var/jenkins
# - the user is in the docker group
# - on some agents docker has the gid 998, on some it is 999
RUN apt-get update \
&& apt-get -y install apt-transport-https ca-certificates curl gnupg2 software-properties-common rsync tree \
&& curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - \
&& add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/debian \
$(lsb_release -cs) \
stable" \
&& apt-get update \
&& apt-get -y install docker-ce docker-ce-cli containerd.io \
&& groupadd -g 1001 jenkins \
&& groupadd -f -g 998 docker1 \
&& groupadd -f -g 999 docker2 \
&& useradd -d "/var/jenkins" -u 1001 -g 1001 -m -s /bin/bash jenkins \
&& usermod -a -G 998 jenkins \
&& usermod -a -G 999 jenkins
USER jenkins
Jenkins then executes the following command
docker run -t -d -u 1001:1001 -u jenkins -v /var/run/docker.sock:/var/run/docker.sock -w /var/jenkins/workspace/JOB_NAME -v /var/jenkins/workspace/JOB_NAME:/var/jenkins/workspace/JOB_NAME:rw,z -v /var/jenkins/workspace/JOB_NAME#tmp:/var/jenkins/workspace/JOB_NAME#tmp:rw,z -e ******** ... our-custom-registry.com/base/jenkins-build:latest cat
This pipeline works just fine... sometimes!
Most of the times, however, some files get mysteriously lost.
For example, my build.gradle consists of multiple other files that are included.
At some point during the build, one of these files seems to be missing.
+ ./gradlew pushDockerImage
FAILURE: Build failed with an exception.
* Where:
Build file '/var/****/workspace/JOB_NAME/build.gradle' line: 35
* What went wrong:
A problem occurred evaluating root project 'foo'.
> Could not read script '/var/****/workspace/JOB_NAME/build/gradle/scripts/springboot-plugin.gradle' as it does not exist.
It is always a different file that goes missing.
I started running tree just before the ./gradlew just to make sure the file is not actually removed.
Does anybody have an idea what might be going on here?
Update
Forget everything I said about docker, this is a pure Gradle and Jenkins problem.
When I replace the docker agent with a plain Jenkins agent the same problems occur.
The problem seems to be that you cannot run multiple gradle tasks in parallel on the same directory.