Jenkins Pipeline's deleteDir() fails with java.nio.file.AccessDeniedException exception - jenkins

Question

I have a Jenkins Pipeline job for building my Android project.
The basic steps are to checkout the project's repository, run docker container (map host's repository folder to appropriate folder in container), execute the script within the container and extract the artifacts.
The very first step deletes the workspace using deleteDir() function:
node("jenkins-slaves") {
deleteDir() // <-------------- HERE
stage('checkout repo') {
// REDACTED
}
// REDACTED
}
However, during one of the first attempts to run it, I've received the following error:
[Pipeline] node
Running on jenkins-slave14 in /home/jenkins/workspace/REDACTED
[Pipeline] {
[Pipeline] deleteDir
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
java.nio.file.AccessDeniedException: /home/jenkins/workspace/REDACTED/app/all-apk/apks
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
at java.nio.file.Files.deleteIfExists(Files.java:1165)
at hudson.Util.tryOnceDeleteFile(Util.java:290)
at hudson.Util.deleteFile(Util.java:245)
at hudson.FilePath.deleteRecursive(FilePath.java:1211)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.access$1000(FilePath.java:197)
at hudson.FilePath$14.invoke(FilePath.java:1181)
at hudson.FilePath$14.invoke(FilePath.java:1178)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2750)
at hudson.remoting.UserRequest.perform(UserRequest.java:208)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:360)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused: java.io.IOException: Unable to delete '/home/jenkins/workspace/REDACTED/app/all-apk/apks'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
at hudson.Util.deleteFile(Util.java:250)
at hudson.FilePath.deleteRecursive(FilePath.java:1211)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
at hudson.FilePath.deleteRecursive(FilePath.java:1202)
at hudson.FilePath.access$1000(FilePath.java:197)
at hudson.FilePath$14.invoke(FilePath.java:1181)
at hudson.FilePath$14.invoke(FilePath.java:1178)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2750)
at hudson.remoting.UserRequest.perform(UserRequest.java:208)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:360)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at ......remote call to jenkins-slave14(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:311)
at hudson.remoting.Channel.call(Channel.java:905)
at hudson.FilePath.act(FilePath.java:987)
Caused: java.io.IOException: remote file operation failed: /home/jenkins/workspace/REDACTED at hudson.remoting.Channel#16d096ce:jenkins-slave14
at hudson.FilePath.act(FilePath.java:994)
at hudson.FilePath.act(FilePath.java:976)
at hudson.FilePath.deleteRecursive(FilePath.java:1178)
at org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:77)
at org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:69)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49)
at hudson.security.ACL.impersonate(ACL.java:260)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE
What is the root cause of this problem and how can I solve that?

Answer 1

Delete error is generally caused because of either insufficient permissions or someone/something locks the file.
I've noticed that the folder that Jenkins unsuccessfully tried to delete was dynamically created in container by the build script. The folder and all the files beneath were created with root/root (user/group).
This helped me to understand the root cause of this problem - the pipeline is running with jenkins/jenkins (user/group) and unable to delete files/folders created with another user (root/root in my case).
The solution I came up with was to create the docker image with same user (belonging to same group) as the host system uses (my container OS is based on Alpine):
RUN addgroup -S -g 6002 jenkins
RUN adduser -S -u 6002 -G jenkins jenkins
USER jenkins
Note that the GID and UID (both equal 6002 in above example) have to match the ids in your host machine. In order to find them, you can use the following commands:
id -u jenkins
id -g jenkins
From documentation:
The USER instruction sets the user name (or UID) and optionally the
user group (or GID) to use when running the image and for any RUN, CMD
and ENTRYPOINT instructions that follow it in the Dockerfile.
Once this step performed, any files/folders that are being created by a build script within the container are using a matching host user - which allows host OS to manipulate/delete them without any issues.