Jenkins pipeline docker.image call generates a timeout - docker

I recently upgraded my jenkins/Docker installation and since, I randomly experience timeout when i run a docker in my jenkins pipeline scripts, in which I start a docker like:
docker.image("${dockerImage}").inside(' -v /opt/software/:/software/:Z'){
.....
}
This occurs randomly and my job crash with the following error message ( see extended log at the bottom of this thread):
ERROR: Timeout after 10 seconds
UPDATE:
My pipeline script:
stage('Build'){
node("master"){
withEnv(['GIT_SSL_NO_VERIFY=true']){
checkout([$class: 'GitSCM', branches: [[name: "$myGitSHA"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId:"${myGitCredentialsId}" , url:"${myGitUrl}" ]]])
}
docker.image("${dockerImage}").inside(' -v /opt/software/:/software/:Z'){
// Cleaning any the previous build
sh "make all"
} // docker
} // node
} // stage
I wonder whether there is an option to increase the default timeout value set currently to 10s.
thanks
$ docker run -t -d -u 995:993 -v /opt/software/:/software/:Z -w /var/lib/jenkins/jobs/RegressionTests/jobs/rg_centos5-x86_64-gcc-4.1.2/workspace
-v /var/lib/jenkins/jobs/RegressionTests/jobs/rg_centos5-x86_64-gcc-4.1.2/workspace:/var/lib/jenkins/jobs/RegressionTests/jobs/rg_centos5-x86_64-gcc-4.1.2/workspace:rw
-v /var/lib/jenkins/jobs/RegressionTests/jobs/rg_centos5-x86_64-gcc-4.1.2/workspace#tmp:/var/lib/jenkins/jobs/RegressionTests/jobs/rg_centos5-x86_64-gcc-4.1.2/workspace#tmp:rw
-e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ********
-e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat centos5_labs_ompi14
ERROR: Timeout after 10 seconds
java.io.IOException: Failed to run image 'centos5_labs_ompi14'. Error:
at org.jenkinsci.plugins.docker.workflow.client.DockerClient.run(DockerClient.java:127)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Execution.start(WithContainerStep.java:178)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:184)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:126)
at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:18)
at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/var/lib/jenkins/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:128)
at org.jenkinsci.plugins.docker.workflow.Docker.node(jar:file:/var/lib/jenkins/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:63)
at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/var/lib/jenkins/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:116)
at Script1.extMain(Script1.groovy:305)
at ___cps.transform___(Native Method)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
at sun.reflect.GeneratedMethodAccessor349.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
at com.cloudbees.groovy.cps.Next.step(Next.java:74)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:165)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:328)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

the issue is due to the hardcoded 10s timeout value, as one can read it from this link;
docker workflow plugin
https://github.com/jenkinsci/docker-workflow-plugin/blob/master/src/main/java/org/jenkinsci/plugins/docker/workflow/client/DockerClient.java#L268
In some cases this is far too short. A solution would be to have this timeout value as a parameter which is set by the user.

I also run into the same problem since yesterday. my docker pipeline version is 1.10.
The issue happens when it try to initiate the docker container from the fol
def abcd_image = docker.image('172.16.181.203:5000/abcd:1.0.0.7')
abcd_image.inside("--privileged -u root -v ${actual_workspace}:${chroot}/code") {
I tried ssh into the docker host and restart docker service. After that, it works but the issue comes back after couple hours.

You can manually set the timeout limit using the "timeout" option.
Set a timeout period for the Pipeline run, after which Jenkins should abort the Pipeline. For example:
options { timeout(time: 1, unit: 'HOURS') }
https://jenkins.io/doc/pipeline/steps/workflow-basic-steps/#code-timeout-code-enforce-time-limit

Related

How to build and run a Docker image in a scripted Jenkinsfile pipeline?

I'm trying to build and run a docker image on a specific build node from a scripted Jenkinsfile. Switching to declarative syntax is something I would rather like to avoid.
My code is quite close to the example from the documentation. The image builds as expected. But running the container fails Jenkins complaining the physical machine of the node is not running inside a container and the echo and make commands from the innermost block that I would expect to run inside the container are not executed and do not appear in the log.
As far as I understand Jenkins considers containers to be build nodes on their own and that nesting of node statements are not allowed. At the same time a node is required to build and run the Docker image.
What am I missing to build and run the image? As Im quite new to Jenkins as well as to Docker any hints or recommendations are appreciated.
The code:
node('BuildMachine1')
{
withEnv(envList)
{
dir("/some/path")
{
docker.build("build-image:${env.BUILD_ID}", "-f ${env.WORKSPACE}/build/Dockerfile .").inside
{
echo "Echo from Docker"
sh script: 'make'
}
}
}
}
The log:
Successfully built 8c57cad188ed
Successfully tagged build-image:79
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] isUnix
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
+ docker inspect -f . build-image:79
.
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] withDockerContainer
BuildMachine1 does not seem to be running inside a container
$ docker run -t -d -u 1004:1005 -w /data/Jenkins_Node/workspace/myFeature/buildaarch64Release -v /data/Jenkins_Node/workspace/myFeature/buildaarch64Release:/data/Jenkins_Node/workspace/myFeature/buildaarch64Release:rw,z -v /data/Jenkins_Node/workspace/myFeature/buildaarch64Release#tmp:/data/Jenkins_Node/workspace/myFeature/buildaarch64Release#tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** build-image:87 cat
$ docker top 2242078968bc1ee5ddfd08c73a2e1551eda36c2595f0e4c9fb6e9b3b0af15b8b -eo pid,comm
[Pipeline] // withDockerContainer
Looks like the Entrypoint of the container was configured in a way that worked for manual usage in terminal but not inside the Jenkins pipeline.
It was set as
ENTRYPOINT ["/usr/bin/env", "bash"]
After changing it to
ENTRYPOINT [ "/bin/bash", "-l", "-c" ]
the resulting container is used by the Jenkinsfile as intended.

Jenkins Pipeline NPM Integration creates empty package.json

We're running a dedicated build server on Debian 10. Jenkins is installed on there as well as Docker.
The Jenkins user of course owns the jenkins folder and is part of the Docker group. So all rights should be set.
There are already a couple pipelines building Docker images just fine.
Now we're trying to set up a new pipeline to build a web front-end. It's using the "Pipeline NPM Integration" plugin.
The pipeline looks like this
pipeline {
environment {
applicationVersion = '0'
}
agent any
stages {
stage('Clone Git') {
steps {
git url: '<githuburl>',
credentialsId: '<githubcreds>',
branch: 'develop'
}
}
stage('Build module (npm)') {
agent {
docker {
image 'node:15.2-buster-slim'
}
}
steps {
withNPM(npmrcConfig:'af7db72c-3235-4827-8eb4-69819a44e612') {
sh 'npm install'
sh 'npm run build:dev'
}
}
}
}
}
The pipeline always fails at the npm build with the following output
Running on Jenkins in /var/lib/jenkins/workspace/test#2
+ docker inspect -f . node:15.2-buster-slim
.
Jenkins does not seem to be running inside a container
$ docker run -t -d -u 108:114 -w /var/lib/jenkins/workspace/test#2 -v /var/lib/jenkins/workspace/test#2:/var/lib/jenkins/workspace/test#2:rw,z -v /var/lib/jenkins/workspace/test#2#tmp:/var/lib/jenkins/workspace/test#2#tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** node:15.2-buster-slim cat
$ docker top c227e7fe6de64d5d7046f5565f109e90ecdd9ece1caf4203e6a6f810cc72f40c -eo pid,comm
Using settings config with name npmrc
A workscape local .npmrc already exists and will be overwrriten for the build.
Writing .npmrc file: /var/lib/jenkins/workspace/test#2/.npmrc
+ npm install
up to date in 178ms
found 0 vulnerabilities
+ npm run build:dev
npm ERR! missing script: build:dev
$ docker stop --time=1 c227e7fe6de64d5d7046f5565f109e90ecdd9ece1caf4203e6a6f810cc72f40c
$ docker rm -f c227e7fe6de64d5d7046f5565f109e90ecdd9ece1caf4203e6a6f810cc72f40c
Removed some lines here to keep it short.
After a couple hours I found out that the package.json while being there is actually empty!
cat /var/lib/jenkins/workspace/test#2/package.json
{}
I have no idea why the plugin creates an empty copy of the file.
Is there any way to get more output from the plugin about what it's doing and what could possibly cause this?

Why when using Jenkins dockerfile agent does it run container with invalid user?

I have a pretty standard Dockerfile:
FROM python:3.6.5
WORKDIR /src
COPY . /src
... install and configure things ...
and a pretty standard Jenkinsfile
pipeline {
agent none
stages {
stage('Test') {
agent { dockerfile true }
steps {
dir('/src') {
sh 'pwd' // any command
}
}
}
}
}
However when this is run in Jenkins (I should mention I'm currently testing using a local Jenkins instance in a docker container) the Dockerfile builds successfully but when Jenkins attempts to run the step I get the following output:
[Pipeline] withDockerContainer
Jenkins seems to be running inside container aaee62f2a28e29b94c13fcdc08c1a82ef7baed48beabe54579db07b2fbd26b23
$ docker run -t -d -u 1000:1000 -w "/var/jenkins_home/workspace/My Project" --volumes-from aaee62f2a28e29b94c13fcdc08c1a82ef7baed48beabe54579db07b2fbd26b23 -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** 12724611f0bf2363c9eee7288654e43eca2aabaf cat
$ docker top 4e29bc102d8f4e6b4ffc142fc06eb706e95b00fa6190b2927f4f79f0cfa53af5 -eo pid,comm
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] dir
Running in /src
[Pipeline] {
[Pipeline] sh
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
$ docker stop --time=1 4e29bc102d8f4e6b4ffc142fc06eb706e95b00fa6190b2927f4f79f0cfa53af5
$ docker rm -f 4e29bc102d8f4e6b4ffc142fc06eb706e95b00fa6190b2927f4f79f0cfa53af5
which is not expected, and when the step is "finished" there is the following failure message:
java.nio.file.AccessDeniedException: /src
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
at java.nio.file.Files.createDirectory(Files.java:674)
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
at java.nio.file.Files.createDirectories(Files.java:767)
at hudson.FilePath.mkdirs(FilePath.java:3273)
at hudson.FilePath.access$1300(FilePath.java:213)
at hudson.FilePath$Mkdirs.invoke(FilePath.java:1254)
at hudson.FilePath$Mkdirs.invoke(FilePath.java:1250)
at hudson.FilePath.act(FilePath.java:1078)
at hudson.FilePath.act(FilePath.java:1061)
at hudson.FilePath.mkdirs(FilePath.java:1246)
at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.<init>(FileMonitoringTask.java:181)
at org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java:221)
at org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java:210)
at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:131)
at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:99)
at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:305)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:268)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:176)
at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157)
at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:155)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
at WorkflowScript.run(WorkflowScript:23)
at ___cps.transform___(Native Method)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
at sun.reflect.GeneratedMethodAccessor143.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
at com.cloudbees.groovy.cps.Next.step(Next.java:83)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:136)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:182)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE
The problem appears to be with this line: docker run -t -d -u 1000:1000 there is no user with UID 1000 on the container built by the Dockerfile, commands are run by root and therefore with the directory change to /src the "user" that jenkins specifies has no access to the files.
How do you prevent Jenkins from trying to use -u 1000:1000 by default? or what steps should I take to resolve this issue. I could in theory add a chmod 777 into the dockerfile but this seems like an ugly workaround
Update:
I added RUN chmod -R 777 /src to my Dockerfile but the issue remains. If I run the container manually from inside the docker Jenkins container:
docker run -it -u 1000:1000 -w "/var/jenkins_home/workspace/My Project" --volumes-from aaee62f2a28e29b94c13fcdc08c1a82ef7baed48beabe54579db07b2fbd26b23 12724611f0bf2363c9eee7288654e43eca2aabaf /bin/bash
and cd /src I have access to the files and can do e.g. touch newfile without issues. The java.nio.file.AccessDeniedException: /src still appears in the logs though
It seems that you can override/pass docker arguments according to this, so you might try this in order to override the default using uid/gid
args: A string. Runtime arguments to pass to docker run.
This option is valid for docker and dockerfile.
agent {
dockerfile {
args '-u 0:0' // or try to make it -u root
}
}

Jenkins pipeline dir() directive causing docker container to exit

Following on from the issue I had in this question: Why when using Jenkins dockerfile agent does it run container with invalid user?
I successfully managed to run the container as root user, however the actual java stack trace failure I received remained the same- implying it is not a permissions error with the user in the container.
The actual issue appears to be with the dir() {} step within my stage. To re-iterate, when I have the following:
steps {
dir('/src') {
sh 'pwd' // any command
}
}
the container exits without running the command:
[Pipeline] withDockerContainer
Jenkins seems to be running inside container aaee62f2a28e29b94c13fcdc08c1a82ef7baed48beabe54579db07b2fbd26b23
$ docker run -t -d -u 1000:1000 -w "/var/jenkins_home/workspace/My Project" --volumes-from aaee62f2a28e29b94c13fcdc08c1a82ef7baed48beabe54579db07b2fbd26b23 -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** 12724611f0bf2363c9eee7288654e43eca2aabaf cat
$ docker top 4e29bc102d8f4e6b4ffc142fc06eb706e95b00fa6190b2927f4f79f0cfa53af5 -eo pid,comm
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] dir
Running in /src
[Pipeline] {
[Pipeline] sh
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
$ docker stop --time=1 4e29bc102d8f4e6b4ffc142fc06eb706e95b00fa6190b2927f4f79f0cfa53af5
$ docker rm -f 4e29bc102d8f4e6b4ffc142fc06eb706e95b00fa6190b2927f4f79f0cfa53af5
and the failure stack trace at the end of the build is:
java.nio.file.AccessDeniedException: /src
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
at java.nio.file.Files.createDirectory(Files.java:674)
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
at java.nio.file.Files.createDirectories(Files.java:767)
at hudson.FilePath.mkdirs(FilePath.java:3273)
at hudson.FilePath.access$1300(FilePath.java:213)
at hudson.FilePath$Mkdirs.invoke(FilePath.java:1254)
at hudson.FilePath$Mkdirs.invoke(FilePath.java:1250)
at hudson.FilePath.act(FilePath.java:1078)
at hudson.FilePath.act(FilePath.java:1061)
at hudson.FilePath.mkdirs(FilePath.java:1246)
at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.<init>(FileMonitoringTask.java:181)
at org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java:221)
at org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java:210)
at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:131)
at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:99)
at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:305)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:268)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:176)
at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157)
at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:155)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
at WorkflowScript.run(WorkflowScript:23)
at ___cps.transform___(Native Method)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
at sun.reflect.GeneratedMethodAccessor143.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
at com.cloudbees.groovy.cps.Next.step(Next.java:83)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:136)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:182)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE
However if I use the following:
steps {
sh 'cd /src && pwd'
}
The command runs as expected. Is this a bug within Jenkins or am I misunderstanding what the dir() directive is used for?
Generally speaking, The dir() directive's purpose is to change your current working directory to another directory on the jenkins agent where the build itself is running.
In your case specifically there is a limitation from the docker plugin side as explained in the following issue's comment:
dir with an absolute path is not supported inside a Docker container. Simply start your sh script with cd. Or avoid using the withDockerContainer step alogether - if it works perfectly for you out of the box, great, otherwise forget about it.
So your current workaround sh 'cd /src' - if we considered it a workaround - is the recommended way to do it.
I agree that this is a bug. And reported the issue here: https://issues.jenkins-ci.org/browse/JENKINS-46055
The core maintainers seemed to feel that there isn't actually a problem because there is a workaround. (the one you described of cd <dir> && <cmd>)
My opinion is that you're using dir correctly, it's just broken.

How do I run jenkins as a single master agent with a docker pipeline?

I am trying to run a single node Jenkins with:
Jenkins master as a docker container
Jenkins jobs running in the same container (single node)
Jenkins docker pipeline tasks, e.g. https://jenkins.io/doc/pipeline/tour/hello-world/
I am using the jenkins/jenkins:lts image. The issues I found and resolved so far were:
Missing docker binary (added it either via extending the image or bind-mounting)
Missing docker plugin not part of the standard recommended plugins (added via the dashboard)
It is able to run docker, but I still get the "Jenkins does not seem to be running inside a container", and the script will not execute.
The failure logs are here:
Jenkins does not seem to be running inside a container
$ docker run -t -d -u 1000:1000 -w "/var/jenkins_home/jobs/Test Pipeline/branches/master/workspace" -v "/var/jenkins_home/jobs/Test Pipeline/branches/master/workspace:/var/jenkins_home/jobs/Test Pipeline/branches/master/workspace:rw,z" -v "/var/jenkins_home/jobs/Test Pipeline/branches/master/workspace#tmp:/var/jenkins_home/jobs/Test Pipeline/branches/master/workspace#tmp:rw,z" -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat node:6.3
[Pipeline] {
[Pipeline] stage
[Pipeline] { (build)
[Pipeline] sh
[workspace] Running shell script
sh: 1: cannot create /var/jenkins_home/jobs/Test Pipeline/branches/master/workspace#tmp/durable-3f202d99/jenkins-log.txt: Directory nonexistent
sh: 1: cannot create /var/jenkins_home/jobs/Test Pipeline/branches/master/workspace#tmp/durable-3f202d99/jenkins-result.txt: Directory nonexistent
Why does it not believe it is running in a container? Is it trying to run the agent worker locally and failing?

Resources