Jenkins concurrent builds on docker slaves - jenkins

Question

I have a Jenkins Server (2.204.1) with Docker plugin (1.1.9) and a docker cloud API.
I work with Jenkins docker agents (slaves)
And i map the docker slave build workspace between the container and the host in order to be able to path
Artifacts to the downstream jobs.
in Jenkins Configuration - Docker Cloud Details - Container settings:
Volumes /var/lib/jenkins:/var/lib/jenkins
This works fine for a single build , The problem starts when i run concurrent builds,
They are all mapped to the same workspace on the Docker host and interfering each other.
What would be the best practice when using docker slaves and mapping workspace as a volume ?
I wouldn't like to use $CustomWorkspace or coping artifacts during the build as this is hard to manage and purge.
I prefer the Jenkins regular slave approach of adding #2 to a second concurrent build but this is not the behavior when running concurrent builds on docker slaves

Answer 1

One remote Jenkins agent has no way of knowing whether a given workspace directory is in use by another agent running on the same machine. This is equally true for docker-based agents that share a common directory via volume mounting. Ideally, all agents working from the same machine would have some way of talking to each other to keep from stepping on each other's toes (e.g. a lockfile in the workspace that gets removed upon job termination), but this is not currently the case.
Solution #1: Unique Build Workspaces
If we are using Jenkins pipelines, we can append a unique subdirectory to the workspace directory on a per-build basis. This solution is clean, simple, and easy to implement.
agent {
node {
customWorkspace "${env.BUILD_NUMBER}"
}
}
Ref: https://www.jenkins.io/doc/book/pipeline/syntax/#agent
Solution #2: Unique Agent Workspaces
If this is not possible or desirable, another potential solution is to change the root working directory of the Jenkins agent itself, which can be done by supplying an additional argument to the agent's startup command:
-workDir FILE : Declares the working directory of the
remoting instance (stores cache and logs by
default)
Source: java -jar agent.jar -help
When spinning up multiple agents dynamically on the same machine, we can set this -workDir value to something with a bit more uniqueness to give each agent its own directory to work out of, effectively mitigating workspace collisions. Something like this should work well:
java -classpath agent.jar hudson.remoting.jnlp.Main -headless \
-workDir /var/lib/jenkins/workspace/$(date +%3N) ...
The magic is in the $(date +%3N), which returns the system clock nanoseconds to three digits of precision. We may want to use more or fewer digits because there's a tradeoff: more precision will result in a higher maximum number of workspace directories but decrease the risk of workspace collisions; less precision will have the opposite effect - fewer directories, increased collision risk.
How this command is configured will vary based on your Jenkins setup. For example, we are using the Docker Swarm plugin (v1.9) on Jenkins 2.249.3. Our agent command is configurable at Manage Jenkins >> Manage Nodes and Clouds >> Configure Clouds >> Docker Swarm Cloud Configuration >> Docker Agent templates >> Command.
Ref: https://man7.org/linux/man-pages/man1/date.1.html