I have latest Jenkins and using it's latest Swarm Plugin.
I have written Ansible modules/roles/playbooks to setup install various tools/configuration on a given target node (which I would like to use as a Swarm slave node).
After Ansible playbook run is complete, I now see a new Slave is created and attached to my Jenkins master but Swarm Plugin's docs (Available Options) doesn't mention how to create ENVIRONMENT variables in the slave. https://wiki.jenkins-ci.org/display/JENKINS/Swarm+Plugin
My question is:
How can I have multiple slaves created on a same target machine and they all have their own individual settings for setting various tools like JAVA_HOME, M2_HOME, GRADLE_HOME, PATH etc.
How can I set ENVIRONMENT variables for a slave using Swarm plugin?
This is required as if I created a slave whose default JAVA is jdk1.7.0_67, then I would like to create another slave whose default JAVA_HOME is jdk1.8.0_45. Similarly, the end goal is to have various flavors of such slaves with various tools if possible, where each slave's tools are slightly different. I'll assign the LABEL(s) accordingly and use it in a Jenkins job's configuration so that a job runs only using / on these slave if the associated label is assigned/tied to the job.
I tried using https://github.com/MovingBlocks/GroovyJenkins/blob/master/src/main/groovy/AddNodeToJenkins.groovy but not sure how I can automatically define/set ENVIRONMENT variables in the slave's configuration.
I'm assuming you're running on Linux here.
You can have a shell script to export the new environment before calling the swarm-client. These variables will be inherited by the new swarm slave
https://unix.stackexchange.com/questions/130985/if-processes-inherit-the-parents-environment-why-do-we-need-export
Alternatively you could run docker and have a separate swarm slave containers https://hub.docker.com/r/csanchez/jenkins-swarm-slave/ and put your specific install into the Dockerfile and add a new ENTRYPOINT in the bottom of the Dockerfile
ENTRYPOINT ["/usr/local/bin/jenkins-slave.sh" \
"-labels", "label1", "-labels", "label2"]
Related
I have a Jenkins Server (2.204.1) with Docker plugin (1.1.9) and a docker cloud API.
I work with Jenkins docker agents (slaves)
And i map the docker slave build workspace between the container and the host in order to be able to path
Artifacts to the downstream jobs.
in Jenkins Configuration - Docker Cloud Details - Container settings:
Volumes /var/lib/jenkins:/var/lib/jenkins
This works fine for a single build , The problem starts when i run concurrent builds,
They are all mapped to the same workspace on the Docker host and interfering each other.
What would be the best practice when using docker slaves and mapping workspace as a volume ?
I wouldn't like to use $CustomWorkspace or coping artifacts during the build as this is hard to manage and purge.
I prefer the Jenkins regular slave approach of adding #2 to a second concurrent build but this is not the behavior when running concurrent builds on docker slaves
One remote Jenkins agent has no way of knowing whether a given workspace directory is in use by another agent running on the same machine. This is equally true for docker-based agents that share a common directory via volume mounting. Ideally, all agents working from the same machine would have some way of talking to each other to keep from stepping on each other's toes (e.g. a lockfile in the workspace that gets removed upon job termination), but this is not currently the case.
Solution #1: Unique Build Workspaces
If we are using Jenkins pipelines, we can append a unique subdirectory to the workspace directory on a per-build basis. This solution is clean, simple, and easy to implement.
agent {
node {
customWorkspace "${env.BUILD_NUMBER}"
}
}
Ref: https://www.jenkins.io/doc/book/pipeline/syntax/#agent
Solution #2: Unique Agent Workspaces
If this is not possible or desirable, another potential solution is to change the root working directory of the Jenkins agent itself, which can be done by supplying an additional argument to the agent's startup command:
-workDir FILE : Declares the working directory of the
remoting instance (stores cache and logs by
default)
Source: java -jar agent.jar -help
When spinning up multiple agents dynamically on the same machine, we can set this -workDir value to something with a bit more uniqueness to give each agent its own directory to work out of, effectively mitigating workspace collisions. Something like this should work well:
java -classpath agent.jar hudson.remoting.jnlp.Main -headless \
-workDir /var/lib/jenkins/workspace/$(date +%3N) ...
The magic is in the $(date +%3N), which returns the system clock nanoseconds to three digits of precision. We may want to use more or fewer digits because there's a tradeoff: more precision will result in a higher maximum number of workspace directories but decrease the risk of workspace collisions; less precision will have the opposite effect - fewer directories, increased collision risk.
How this command is configured will vary based on your Jenkins setup. For example, we are using the Docker Swarm plugin (v1.9) on Jenkins 2.249.3. Our agent command is configurable at Manage Jenkins >> Manage Nodes and Clouds >> Configure Clouds >> Docker Swarm Cloud Configuration >> Docker Agent templates >> Command.
Ref: https://man7.org/linux/man-pages/man1/date.1.html
I have a swarm of three nodes (one manager, two workers). In my swarm, I am running a jenkins service with docker-swarm-plugin (https://github.com/jenkinsci/docker-swarm-plugin) installed. I want to use the plugin to create a build agent container in my swarm for every jenkins job. For the agents I want to user the jenkins/jnlp-slave docker image (https://hub.docker.com/r/jenkinsci/jnlp-slave/). The image expects two arguments for the start:
secret (can be set via JENKINS_SECRET environment variable)
agent name (can be set via JENKINS_AGENT_NAME environment variable)
The docker-swarm-plugin creates three environment variables:
$DOCKER_SWARM_PLUGIN_JENKINS_AGENT_SECRET (I use this to set the secret)
$DOCKER_SWARM_PLUGIN_JENKINS_AGENT_JAR_URL
$DOCKER_SWARM_PLUGIN_JENKINS_AGENT_JNLP_URL (this contains the agent name)
I pass the secret to the agent via JENKINS_SECRET environment variable (in ENV section of Jenkins plugin configuration):
JENKINS_SECRET=$DOCKER_SWARM_PLUGIN_JENKINS_AGENT_SECRET
I tried to pass the agent name by using a regular expression (also in ENV section of Jenkins plugin configuration):
JENKINS_AGENT_NAME=`echo $DOCKER_SWARM_PLUGIN_JENKINS_AGENT_JNLP_URL | sed ...`
But the command is not executed (I understand that this is for security reasons to avoid code injection).
What do I want to reach:
I want to run jenkins on my docker swarm and I want jenkins to run every job in an own build agent container that is dropped after the job finished. And I want the build agent containers to spread across the swarm (jenkins docker-plugin launches them on the node where the jenkins master is running). I understood that the docker-swarm-plugin should do exactly what I want to do. And I think the jenkins/jnlp-slave image is there to build agent containers as I want to use. But I can't find a solution how to get them work together.
Can anyone give me some advice?
Should I maybe use another image that is working better with the plugin?
I opened issue https://github.com/jenkinsci/docker-swarm-plugin/issues/37 on docker-swarm-plugin for this and now with PR https://github.com/jenkinsci/docker-swarm-plugin/pull/39 a new environment variable is added with the created agents name. This can be passed to the docker image and everything works fine!
I'd like to be able to dynamically provision docker child nodes for builds and have the configuration / setup of those nodes be part of the Jenkinsfile groovy script it uses.
Limitations of the current setup of jobs means Jenkins has one node/executor (master) and I'd like to support using Docker for nodes to alleviate this bottleneck.
I've noticed there's two ways of using a docker container as a node:
You can use the agent section in your pipeline file which allows you to specify an image to use. As part of this, you can target a specific node which supports running docker images, but I haven't gotten that far as to see what happens.
You can use the Jenkins Docker Plugin which allows you to add a Docker Cloud in Jenkins' configuration. It allows you to specify a label which, when used as part of a build, will spawn a container in that "cloud" from the image chosen in the cloud configuration. In this case, the "cloud" is the docker instance running on the Jenkins server.
Unfortunately, it doesn't seem like you can use both together - using the label but specifying a docker image in the configuration (1) where the label matches a docker cloud template configuration (2) does not seem to work and instead produces a label not found error during the build.
Ideally I'd prefer the control to be in the pipeline groovy file so the configuration is stored with the application (1), not with the Jenkins server (2). However, it suggests that if I use the agent section and provide a docker image, it still must target an existing executor first (i.e. master) which will cause other builds to queue until the current build is complete.
I'm at a point of migrating builds, so not all builds can support using a docker container as the node yet, and builds will have issues when ran in parallel on the master node.
Is there a way for a docker pipeline file to determine the image of the child node it runs on?
There are a few options I have considered but not attempted yet:
Migrate jobs to run on the "docker cloud" until all jobs support running on child container nodes, then move the configuration from Jenkins to the pipeline build file for each job and turn on parallel builds on the master node.
Attempt to add a new node configuration which is effectively a copy of master (uses the same server, just different location). Configure it to support parallel builds, and have all migrated jobs target the node explicitly during builds.
We have the following jenkins setup:
Jenkins master
Jenkins Slave1
Jenkins Slave2
Jenkins Slave3
Those are all virtual machines and the slaves do always exist. They don't spawn automatically up and down.
Now we have builds which needs a lot of tools (maven, python, aws cli, ...). We can install every tool on every slave and everything will work fine.
But we want to build a docker approach.
Nearly all the tutorials I've seen are using slaves in Docker. They use some orchestration tool like Kubernetes and are creating slaves in Docker, do their stuff and delete the pod again.
We don't have the possibility to do this:
Question: Is it a decent approach to use an 'old' Jenkins setup with
real VM slaves on which we use docker?
What I'm thinking about is writing a pipeline and in each stage we use a docker container:
start build (it will choose a slave, e.g. Slave1)
pipeline will start
stage1: spin up e.g. a python container: git clone and execute python commands. mount volume to workspace??
stage2: sping up e.g. aws container and mount the content of the workspace and execute new commands etc.
Can someone evaluate this approach?
This is a very good approach. In fact the way to do that is documented under jenkins docs under Using multiple containers section.
In each stage you basically spin up a container with the necessary tools available and you can use a volume to presist output from the stage into the workspace so that other
stages can use it.
I want to setup Jenkins master on server A and slave on server B with use of Docker.
Both servers are virtual machines dedicated for Jenkins.
Currently I have started Docker container on server A for master, based on the official Jenkins docker image. But what docker image should I use for Jenkins slave?
That actually depends on the environment and tools you need in your build environment. For example, if you build a C project, you would need an image containing a C compiler and possibly make if you use Makefiles. If you build a Java project, you would need a JDK with a Java compiler and possibly Ant / Maven / Gradle if you use them as part of your build.
You can use the evarga/jenkins-slave as a good starting point for your build slave.
This image already contains JDK. If you simply need JDK and Maven on your build slave, you can build your Docker image with the following Dockerfile:
FROM evarga/jenkins-slave
run apt-get install maven
Using Docker images for build slaves is actually a good idea. Some of the reasons appear at Templating Jenkins Build Environments with Docker Containers:
Docker has established itself as a popular and convenient way to
bootstrap isolated and reproducible environments, which enables Docker
containers to be the most maintainable slave environments. Docker
containers’ tooling and other configurations can be version controlled
in an environment definition called a Dockerfile, and Dockerfiles
allows multiple identical containers can be created quickly using this
definition or for more customized off-shoots to be created by using
that Dockerfile’s image as a base.
I suggest you take trying to use dynamic|ephemeral docker nodes, instead of manually creating nodes and connecting to them via ssh. Take a look at https://engineering.riotgames.com/news/putting-jenkins-docker-container, it's very powerful and I think it's one of killer usecases for Docker.