I have a declarative Jenkins Pipeline with a lock, e. g.
pipeline {
environment {
BRANCH = 'master'
}
agent any
stages{
stage('stage') {
options {
lock(resource: "lock-${env.BRANCH}")
}
steps {
echo "Something"
}
}
}
}
But when I execute the pipeline, in the log it says
[Pipeline] lock
Trying to acquire lock on [lock-null]
Lock acquired on [lock-null]
[Pipeline] {
[Pipeline] echo
master
[Pipeline] }
Lock released on resource [lock-null]
The environment variable seems to be not set when the lock-name is evaluated, but when the echo argument is evaluated, it is set correctly.
This answer to a somewhat related question gave the hint to use a lazily evaluated GString instead of a normal GString. Trying this:
pipeline {
environment {
BRANCH = 'master'
}
agent any
stages{
stage('stage') {
options {
lock(resource: "lock-${->env.BRANCH}" as String)
}
steps {
echo "${->env.BRANCH}" as String
}
}
}
}
gives me the following log messages
[Pipeline] lock
Trying to acquire lock on [[no resource/label specified - probably a bug]]
Lock acquired on [[no resource/label specified - probably a bug]]
[Pipeline] {
[Pipeline] echo
master
[Pipeline] }
Lock released on resource [[no resource/label specified - probably a bug]]
So, it looks like the variable can't be resolved correctly.
The problem I want to solve is, creating a multibranch-pipeline which has a lock on a stage. But when the lock has a name, which is not dependend on the branchname, only one branch of the pipeline can run in parallel in this stage.
How can I solve this?
You could just use lock as a step instead of an option:
pipeline {
environment {
BRANCH = 'master'
}
agent any
stages{
stage('stage') {
steps {
lock("lock-${env.BRANCH}" as String) {
echo "${env.BRANCH}" as String
}
}
}
}
}
As within steps the variable env.BRANCH is set, this should work.
Also see documentation for lock step.
I recently ran into this myself. ${env} is not accessible in the options block, but ${currentBuild} is. So what I did was first was println("$currentBuild") and get the name of the class. From that, I googled the Java source code docs and followed the functions until I got what I needed.
In my case, I wanted the ${env.NODE_NAME} and I ended up with ${currentBuild.getRawBuild().getExecutor().getOwner().getDisplayName()}
Related
I am building a Jenkins pipeline for a scenario where I'll have to use specific Jenkins agents inside remote data centers to deploy my code to those data centers. This is due to firewall restrictions on some ports, specifically WinRM is blocked between some of our global data centers.
Our deploys are written so that a single deploy stage can deploy to any number of environments, specified by the user's passed-in parameters. The stage loops through the environments and calls a generic deploy script for each one.
I know how to specify an agent by its label or other closure in a stage's definition:
stage ('a stage') {
agent { label 'some agent label' }
steps { ...
but in this case, i am solving for deploying to multiple environments in one deploy stage, each of which will require its own agent.
I can, of course, specify a unique stage for each env, and use a when clause to run it when appropriate, but that's messy.
What I'd like to do is tell the pipeline what agent(s) to use for the deploy stage inside inside the deploy stage, and be able to use multiple agents within that single stage, determined dynamically based on the parameters of the run.
I'd originally found this answer on SO, which gave me the idea of acquiring a node inside the stage, and not with the agent declaration. It doesn't show the acquisition inside a script block, but I'd initially read it that way, and that gave me the idea to try acquiring the node inside a script. And once you're there, it's a small leap to try doing it in a loop.
To prove it, I print some local environment variables from the agent to prove that we're switching agents, inside the stage, inside the loop. I'm also passing a file to each agent to prove that I can pass the files through the firewall.
Note that to even connect to the agent behind the firewall, we had to open the port that is defined in the Jenkins global security config, inbound to the agent from the controller (aka master), and https (443) outbound to the controller. The inbound port is configured to be static.
pipeline {
agent none
stages {
stage ('init') {
agent any
steps {
writeFile file: 'tester', text: 'i am a test file'
stash includes: 'tester', name: 'tester'
}
}
stage ('get agents') {
steps {
script {
['Agent1', 'Agent2'].each { agent ->
node (agent) {
echo "I am agent `${NODE_NAME}`\nMy labels are `${NODE_LABELS}`"
unstash 'tester'
echo "the content of the file is `${readFile 'tester'}`"
}
}
}
}
}
}
}
Which outputs:
Started by user Maximilian Cascone Admin
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] stage
[Pipeline] { (init)
[Pipeline] node
Running on Agent1 in /mnt/data/jenkins/workspace/Sandbox/mcascone/dynamic-agents
[Pipeline] {
[Pipeline] writeFile
[Pipeline] stash
Stashed 1 file(s)
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (get agents)
[Pipeline] script
[Pipeline] {
[Pipeline] node
Running on Agent2 in D:\workspace\Sandbox\mcascone\dynamic-agents
[Pipeline] {
[Pipeline] echo
I am agent `Agent2`
My labels are `Cider Redgate Windows Worker02 ant chef npm relativity wix`
[Pipeline] unstash
[Pipeline] readFile
[Pipeline] echo
the content of the file is `i am a test file`
[Pipeline] }
[Pipeline] // node
[Pipeline] node
Running on Agent1 in C:\jenkins\workspace\Sandbox\mcascone\dynamic-agents
[Pipeline] {
[Pipeline] echo
I am agent `Agent1`
My labels are `Itar Agent1`
[Pipeline] unstash
[Pipeline] readFile
[Pipeline] echo
the content of the file is `i am a test file`
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
Finished: SUCCESS
So I've proven that I can get multiple agents dynamically within a single stage. Next step would be elevating this into a shared step, so it can be called without the script block and make the pipeline nice and neat. But as a POC, this is a great achievement. I don't believe I've seen this elsewhere.
The answer might not fit all needs you have, but dynamic generation of stage and on that basis you can assign / execute the generate stages with following way.
def agents = ['master', 'agent1', 'agent2']
def generateStage(nodeLabel) {
return {
stage("Runs on ${nodeLabel}") {
node(nodeLabel) {
echo "Running on ${nodeLabel}"
}
}
}
}
def parallelStagesMap = agents.collectEntries {
["${it}" : generateStage(it)]
}
pipeline {
agent none
stages {
stage('non-parallel stage') {
steps {
echo 'This stage will be executed first.'
}
}
stage('parallel stage') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
Furthermore you can use collectEntries, out of the function box parallelStagesMap, in this way you can use each collect entry for different stage and can dynamically assign nodes in the stage, and in function generateStage you need to do modification as per your requirement.
If you wanted to execute these stages sequentially, then remove parallel from script.generateStage contains return which is imporant without that, pipeline will not work as expected.
How do I mark a stage as skipped when using scripted pipeline.
I have no problem skipping a stage in declarative pipeline. I just set
when {
expression {<some boolean expression>}
}
And if the expression is evaluate to false that stage is skipped.
The problem is that if you try to do this with scrippted pipeline you get:
java.lang.NoSuchMethodError: No such DSL method 'when' found among steps
error message. This is because the DSL of declarative pipeline is not the same as scripted pipeline
So, how can it be done?
Solving this issue takes a little bit of hacking... (don't worry, nothing fancy)
The way to do this is by using Jenkins' module that can be found here.
So to mark a stage as skipped you need to call static method markStageSkippedForConditional passing the name of the stage you are skipping.
Lets say you have a stage named "mystage". and you want to skip it and mark it as "skipped".
Your code should look something like:
import org.jenkinsci.plugins.pipeline.modeldefinition.Utils
node() {
stage('a'){
echo 'stage 1'
}
stage('mystage'){
if(true){
echo 'skipping stage...'
Utils.markStageSkippedForConditional('mystage')
}else{
echo 'This stage may be skipped'
}
}
stage('b'){
echo 'stage 2'
}
}
If you're using a version of Jenkins older than mid 2019, you must uncheck Use Groovy Sandbox checkbox, since the Utils method was not yet whitelisted for external use.
You can find an implementation (in the form of a shared pipeline step) in comquent/imperative-when on GitHub.
This allows to access the method Utils.markStageSkippedForConditional that you found yourself in a nice way, like the following:
stage('Zero') {
when(BRANCH_NAME != 'master') {
echo 'Performing steps of stage Zero'
}
}
In declarative pipeline you can use:
stage('deploy') {
when { <some boolean expression> }
......
}
In Scripted pipeline you can use:
if(<some boolean expression>) {
stage('deploy') {
......
}
}
both above works fine. i tested
I have multipe Jenkins node agents, including "master", "tiering_agent1", and "cirrus". I am trying to set the node on which a Stage is executed by a parameters{} setting.
I have this pipeline code
def BuildAgentLabel='tiering_agent1'
pipeline {
agent { label 'master' }
parameters {
string(
name: 'NEW_LABEL',
defaultValue: '',
description: ''
)
}
stages {
stage( 'Init') {
steps {
script {
if ( params.NEW_LABEL != '' ){
echo "Setting BuildAgentLabel to '${params.NEW_LABEL}'"
BuildAgentLabel = params.NEW_LABEL
echo "BuildAgentLabel is now '${BuildAgentLabel}'"
}
}
}
}
stage( "Build") {
agent { label BuildAgentLabel }
steps {
echo "Performing Stage '${STAGE_NAME}' on NODE '${env.NODE_NAME}'"
echo "BuildAgentLabel=${BuildAgentLabel}"
}
}
}
}
Though the 'Init' stage sets the global variable BuildAgentLabel to a different value( if the NEW_LABEL parameters is a string with length > 0), the 'Build' stage always gets executed on whatever node the "def BuildAgentLabel" statement is originally set to.
The console output of the run echoes this:
[Pipeline] node
Running on Jenkins in ...
[Pipeline] {
.
.
.
[Pipeline] stage
[Pipeline] { (Init)
[Pipeline] script
[Pipeline] {
[Pipeline] echo
Setting BuildAgentLabel to 'master'
[Pipeline] echo
BuildAgentLabel is now 'master'
.
.
.
[Pipeline] stage
[Pipeline] { (Build)
[Pipeline] node
Running on tiering_agent1 in /opt/jenkins-agent/workspace/ine-multibranch-test_master-RGJIAQXOIAPL7XDIJW6DOGF4KUE5KBRXCAZ7U4IUW2YOTZVQTWCA
[Pipeline] {
.
.
.
[Pipeline] {
[Pipeline] echo
Performing Stage 'Build' on NODE 'tiering_agent1'
[Pipeline] echo
BuildAgentLabel=master
.
.
.
[Pipeline] End of Pipeline
Finished: SUCCESS
It's like the stage{} objects are instantiated almost concurrently and snag their agent label at that time, but stage execution comes after that.
I would like the 'Init' stage to be able to affect the node on which the 'Build' stage is performed upon, but cannot seem to make it work. How can I get the result I want?
The issue is agent { label BuildAgentLabel } doesn't resolve the variable BuildAgentLabel to it's value possibly due to this bug - https://issues.jenkins-ci.org/browse/JENKINS-9665 and I'm guessing your node agent 'tiering_agent1' is configured as Use this node as much as possible and it's defaulting to this agent.
However, setting label to the parameter directly works agent { label "${params.NEW_LABEL}" }
If I'm not mistaken, you can't change a params item. At least, when I have tried, I get some kind of "static/inaccessible map" error. You can, however, change environment variables, as long as they have not been instantiated in a parent environment declarative (see this link for complete details). And I also believe that you can access any param.name variable as env.name, and that might be changeable. It might depend on the circumstance. If it's not, you can always set a new env var as the value of the incoming parameter, and that will definitely be mutable.
I have a project which has multiple build pipelines to allow for different types of builds against it (no, I don't have the ability to make one build out of it; that is outside my control).
Each of these pipelines is represented by a Jenkinsfile in the project repo, and each one must use the same build agent label (they need to share other pieces of configuration as well, but it's the build agent label which is the current problem). I'm trying to put the label into some sort of a configuration file in the project repo, so that all the Jenkinsfiles can read it.
I expected this to be simple, as you don't need this config data until you have already checked out a copy of the sources to read the Jenkinsfile. As far as I can tell, it is impossible.
It seems to me that a Jenkinsfile cannot read files from SCM until the project has done its SCM step. However, that's too late: the argument to agent{label} is read before any stages get run.
Here's a minimal case:
final def config
pipeline {
agent none
stages {
stage('Configure') {
agent {
label 'master'
}
steps {
checkout scm // we don't need all the submodules here
echo "Reading configuration JSON"
script { config = readJSON file: 'buildjobs/buildjob-config.json' }
echo "Read configuration JSON"
}
}
stage('Build and Deploy') {
agent {
label config.agent_label
}
steps {
echo 'Got into Stage 2'
}
}
}
}
When I run this, I get:
java.lang.NullPointerException: Cannot get property 'agent_label' on null object I don't get either of the echoes from the 'Configure' stage.
If I change the label for the 'Build and Deploy' stage to 'master', the build succeeds and prints out all three echo statements.
Is there any way to read a file from the Git workspace before the agent labels need to be set?
Please see https://stackoverflow.com/a/52807254/7983309. I think you are running into this issue. label is unable to resolve config.agent_label to its updated value. Whatever is set in the first line is being sent to your second stage.
EDIT1:
env.agentName = ''
pipeline {
agent none
stages {
stage('Configure') {
agent {
label 'master'
}
steps {
script {
env.agentName = 'slave'
echo env.agentName
}
}
}
stage('Finish') {
steps {
node (agentName as String) { println env.agentName }
script {
echo agentName
}
}
}
}
}
Source - In a declarative jenkins pipeline - can I set the agent label dynamically?
I'm not a Jenkins guru so please be patient. :-)
I have a pipeline, something nearly as simple as this:
def hash = ''
node {
stage('Checkout') {
…
}
stage('Build') {
…
}
stage('Tests') {
…
}
}
stage('Ask deploy') {
input 'Deploy?'
}
node {
stage('Deploy') {
}
}
I want to set the value of the hash variable in the first node and read it in the next if the manual input is positive. Is this possible and safe? Is this the correct approach?
Note that there are multiple executors and manual input involved. In the Jenkins docs it is hinted for a node that:
As soon as an executor is free on a node, the steps will run.
This means that the two nodes may run in different executors, correct? Do they still share the same global variables? Thanks in advance for any clarifications!
If you have multiple slaves in Jenkins, the pipeline will be launch in one of this slaves. Every slave is different.
Every stage in you pipeline will be launch in the same slave so if you have the variable "hash" at the first line of your pipeline you wouldn't have problem to read it in all your pipeline but if you have to access to this variable value from a different build you can not access.
If you need a global variable to read it in different builds you can define a global variable using the Global Variables String Parameter Plugin
The hash variable is global and its value is available in the different executors which seems logical to me. So it looks like what I do is OK and it will work this way unless I miss something.
Here is how I've verified that (details skipped for brevity):
I've created a similar pipeline and killed the executor which ran the first node:
def gitHash;
node {
withCredentials(...) {
//Step 1:
//Check out from the SCM
stage('Prepare') {
echo "Checking out the project from source control.."
scmInfo = checkout scm
gitHash = scmInfo.GIT_COMMIT
echo "Project checked out, the GIT hash of the last commit is: ${gitHash}"
}
}
}
stage('Ask deploy') {
input 'Deploy?'
}
node {
withCredentials(...) {
stage('Deploy') {
echo "TODO, hash ${gitHash}"
}
}
}
The output from Jenkins is the following (details skipped):
Obtained Jenkinsfile from 7adc4bb98524b31de93e0c1ae16bf967ca3df47c
Running on jnlp-13775fa128a47 in /root/workspace/...
[Pipeline] {
[Pipeline] withCredentials
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Prepare)
[Pipeline] echo
Project checked out, the GIT hash of the last commit is: 7adc4bb98524b31de93e0c1ae16bf967ca3df47c
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] stage
[Pipeline] { (Ask deploy)
[Pipeline] input
Deploy?
Proceed or Abort
Approved by admin
[Pipeline] }
[Pipeline] // stage
[Pipeline] node
Running on jnlp-1383bdf520c9d in /root/workspace/...
[Pipeline] {
[Pipeline] withCredentials
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Deploy)
[Pipeline] echo
TODO, hash 7adc4bb98524b31de93e0c1ae16bf967ca3df47c
[Pipeline] End of Pipeline
Finished: SUCCESS
As seen the first node runs on executor jnlp-13775fa128a47 the second is on jnlp-1383bdf520c9d but the value of the globally scoped variable can be read there.