Can I assign a node dynamically within an iterative loop in Jenkins? - jenkins

Question

I am building a Jenkins pipeline for a scenario where I'll have to use specific Jenkins agents inside remote data centers to deploy my code to those data centers. This is due to firewall restrictions on some ports, specifically WinRM is blocked between some of our global data centers.
Our deploys are written so that a single deploy stage can deploy to any number of environments, specified by the user's passed-in parameters. The stage loops through the environments and calls a generic deploy script for each one.
I know how to specify an agent by its label or other closure in a stage's definition:
stage ('a stage') {
agent { label 'some agent label' }
steps { ...
but in this case, i am solving for deploying to multiple environments in one deploy stage, each of which will require its own agent.
I can, of course, specify a unique stage for each env, and use a when clause to run it when appropriate, but that's messy.
What I'd like to do is tell the pipeline what agent(s) to use for the deploy stage inside inside the deploy stage, and be able to use multiple agents within that single stage, determined dynamically based on the parameters of the run.

Answer 1

I'd originally found this answer on SO, which gave me the idea of acquiring a node inside the stage, and not with the agent declaration. It doesn't show the acquisition inside a script block, but I'd initially read it that way, and that gave me the idea to try acquiring the node inside a script. And once you're there, it's a small leap to try doing it in a loop.
To prove it, I print some local environment variables from the agent to prove that we're switching agents, inside the stage, inside the loop. I'm also passing a file to each agent to prove that I can pass the files through the firewall.
Note that to even connect to the agent behind the firewall, we had to open the port that is defined in the Jenkins global security config, inbound to the agent from the controller (aka master), and https (443) outbound to the controller. The inbound port is configured to be static.
pipeline {
agent none
stages {
stage ('init') {
agent any
steps {
writeFile file: 'tester', text: 'i am a test file'
stash includes: 'tester', name: 'tester'
}
}
stage ('get agents') {
steps {
script {
['Agent1', 'Agent2'].each { agent ->
node (agent) {
echo "I am agent `${NODE_NAME}`\nMy labels are `${NODE_LABELS}`"
unstash 'tester'
echo "the content of the file is `${readFile 'tester'}`"
}
}
}
}
}
}
}
Which outputs:
Started by user Maximilian Cascone Admin
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] stage
[Pipeline] { (init)
[Pipeline] node
Running on Agent1 in /mnt/data/jenkins/workspace/Sandbox/mcascone/dynamic-agents
[Pipeline] {
[Pipeline] writeFile
[Pipeline] stash
Stashed 1 file(s)
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (get agents)
[Pipeline] script
[Pipeline] {
[Pipeline] node
Running on Agent2 in D:\workspace\Sandbox\mcascone\dynamic-agents
[Pipeline] {
[Pipeline] echo
I am agent `Agent2`
My labels are `Cider Redgate Windows Worker02 ant chef npm relativity wix`
[Pipeline] unstash
[Pipeline] readFile
[Pipeline] echo
the content of the file is `i am a test file`
[Pipeline] }
[Pipeline] // node
[Pipeline] node
Running on Agent1 in C:\jenkins\workspace\Sandbox\mcascone\dynamic-agents
[Pipeline] {
[Pipeline] echo
I am agent `Agent1`
My labels are `Itar Agent1`
[Pipeline] unstash
[Pipeline] readFile
[Pipeline] echo
the content of the file is `i am a test file`
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
Finished: SUCCESS
So I've proven that I can get multiple agents dynamically within a single stage. Next step would be elevating this into a shared step, so it can be called without the script block and make the pipeline nice and neat. But as a POC, this is a great achievement. I don't believe I've seen this elsewhere.

Answer 2

The answer might not fit all needs you have, but dynamic generation of stage and on that basis you can assign / execute the generate stages with following way.
def agents = ['master', 'agent1', 'agent2']
def generateStage(nodeLabel) {
return {
stage("Runs on ${nodeLabel}") {
node(nodeLabel) {
echo "Running on ${nodeLabel}"
}
}
}
}
def parallelStagesMap = agents.collectEntries {
["${it}" : generateStage(it)]
}
pipeline {
agent none
stages {
stage('non-parallel stage') {
steps {
echo 'This stage will be executed first.'
}
}
stage('parallel stage') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
Furthermore you can use collectEntries, out of the function box parallelStagesMap, in this way you can use each collect entry for different stage and can dynamically assign nodes in the stage, and in function generateStage you need to do modification as per your requirement.
If you wanted to execute these stages sequentially, then remove parallel from script.generateStage contains return which is imporant without that, pipeline will not work as expected.