I have the following declarative pipeline where I write a global build variable
during a parallel matrix, the write in stage Build Detection is probably (wasn't clear to me) a race condition but I am not sure. I have 3 questions regarding the below simple pipeline:
Is it correct that since Build-Detection uses the same agent (note only Build uses a different agent), it is definitely a race condition ?
If I would have one agent for each parallel line, it would not be a
race condition as the global build is different in each agent?
Is there a way to make a variable copy of build inside the stage such that its not global anymore?
How should we deal with global variable communicating stuff (for when steps etc)
and parallel matrix feature?
Map<String,Boolean> build
pipeline {
stages {
stage('Test') {
failFast false
matrix {
axes {
axis {
name 'CONTAINER'
values 'A', 'B'
}
}
stages {
stage('Build Detection') {
steps {
script {
build[CONTAINER] = CONATAINER == 'A'
echo "Should Build: ${build[CONTAINER]}"
}
}
}
stage('Build') {
agent {
kubernetes {
yamlFile '.jenkins/pods/build-kaniko.yaml'
}
}
when {
beforeAgent true
expression { return build[CONTAINER] }
}
steps {
echo "BUILDING....."
}
}
}
}
}
}
}
No, it has nothing to do with build agents. The JVM that's executing the compiled groovy code is running on the Jenkins master, not a build agent. Therefore, using a global variable is shared by each thread running in the Jenkins master JVM. Whether there's a possible race condition is not related to stages using the same or different build agents.
Same answer as 1.
Yes, simply define a variable using "def" or a specific type in the stage's script block. Just be sure to not reference a new variable without a type because in Groovy that causes it to be declared globally.
Using a map with a key that is specific to each thread like you're doing seems like a good way to me. If you really want to make sure there is no possibility of two unsafe thread operations modifying the map at the same time, then make sure that a threadsafe map is used. You could print out the class of the map to find out what implementation is getting instantiated. I would hope it's something threadsafe like ConcurrentHashMap.
Related
I have a bunch of repositories which use (parts of) the same Jenkins shared library for running tests, docker builds, etc. So far the shared library has greatly reduced the maintenance costs for these repos.
However, it turned out that basically all pipelines use the same set of options, e.g.:
#Library("myExample.jenkins.shared.library") _
import org.myExample.Constants
pipeline {
options {
disableConcurrentBuilds()
parallelsAlwaysFailFast()
}
agent {
label 'my-label'
}
stages {
stage {
runThisFromSharedLibrary(withParameter: "foo")
runThatFromSharedLibrary(withAnotherParameter: "bar")
...
...
In other words, I need to copy-and-paste the same option snippets in any new specific pipeline that I create.
Also, this means that I need to edit separately each Jenkinsfile (along with any peer-review processes we use internally) when I decide to change the set of options.
I'd very much like to remove this maintenance overhead somehow.
How can I delegate the option-setting to a shared library, or otherwise configure the required options for all pipelines at once?
Two options will help you the most:
Using global variables on Master/Agent level.
go to Jenkins-->Manage Jenkins-->Configure System--> Global properties.
Mark the Environment variables box then add name and value for the variable.
then you will be able to use it normally in your Jenkins pipelines as below code snippets.
Wrap the whole pipeline in a function inside shared-library.
Jenkinsfile will look like below:
#Library('shared-library') _
customServicePipeline(agent: 'staging',
timeout: 3,
server:'DEV')
shared library function
// customServicePipeline.groovy
def call(Map pipelineParams = [:]) {
pipeline {
agent { label "${pipelineParams.agent}" }
tools {
maven 'Maven-3.8.6'
jdk 'JDK 17'
}
options {
timeout(time: "${pipelineParams.timeout}", unit: 'MINUTES')
}
stages {
stage('Prep') {
steps {
echo 'prep started'
pingServer(pipelineParams.get("server"))
}
}
}
}
}
I have a jenkins declarative pipeline which I am interested to be able to perform a stage only if a specific environment variable contains a specific substring(not fully equals to it, just contains it).
Does anyone got any idea on how can I implement it(maybe using the when condition if possible).
Thanks in advance,
Alon
As you mentioned, in declarative pipeline you can use the when directive to establish a condition in which the stage will be executed.
Among the built in condition options like triggeredBy,branch and tag there is the generic expression option, which allows you to run any groovy code and calculate the relevant Boolean value according to your needs.
So for your case for example you can just use the groovy contains methods to achieve what you want, something like:
pipeline {
agent any
stages {
stage('Conditional Stage') {
when {
expression { return env.MyParamter.contains('MySubstring') }
}
steps {
echo "Running the conditional stage"
}
}
}
}
I put this in the script section of a Jenkins UI job's configuration -
pipeline {
agent any
stages{
stage('Project') {
...
That works, however -
pipeline {
agent any
stages{
stage('Project ' + 'Josh') {
...
throws and displays an incorrect error message because the parser gets all confused due to the constructed string inside the stage.
Moreover,
String description = 'Project' + ' Josh'
pipeline {
agent any
stages{
stage(description) {
...
does not fail, but displays 'description' as the stage's description.
Now, if you try to load a groovy PaC file with this in it:
node {
stage('Project' + 'Josh') {
...
it works without a hitch.
Is it possible that there are two different Groovy parsers employed, one for the UI and another for loaded PaC's? This means that the UI one has this really horrible bug in it...
Ideas?
.a.
Your example has nothing to do with Jenkins UI. You have shown two different pipeline types - a declarative and scripted one.
Declarative pipeline
A declarative pipeline
pipeline {
agent any
stages {
stage('Build') {
steps {
// do something here
}
}
}
}
introduces more simplified, limited and opinionated syntax. This type of a pipeline sets boundaries for Groovy code execution - it is only available inside a dedicated script block, e.g.
pipeline {
agent any
stages {
stage('Build') {
steps {
script {
def name = 'Joe'
echo "My name is ${name}"
}
}
}
}
}
This is why stage block expects a literal and not a variable nor expression.
Scripted pipeline
The second example you have shown is a scripted pipeline. This kind of pipeline is more powerful comparing to a declarative pipeline - the whole pipeline script is more or less a Groovy script so you can put any code almost everywhere. A scripted pipeline starts with node block and it allows you to put any Groovy code inside this block. Consider following example:
node {
stage("Test") {
echo "1,2,3"
}
for (int i = 0; i < 5; i++) {
stage("Stage ${i}") {
echo "This is ${i}"
}
}
}
This pipeline script generates 6 stages:
As you can see there are actually no limits what kind of stuff you put inside node block. Declarative pipeline does not allow you doing that - its syntax is strict and you have to follow it directly.
Differences
As a final note I will quote Jenkins official docs:
Where they differ however is in syntax and flexibility. Declarative limits what is available to the user with a more strict and pre-defined structure, making it an ideal choice for simpler continuous delivery pipelines. Scripted provides very few limits, insofar that the only limits on structure and syntax tend to be defined by Groovy itself, rather than any Pipeline-specific systems, making it an ideal choice for power-users and those with more complex requirements. As the name implies, Declarative Pipeline encourages a declarative programming model. Whereas Scripted Pipelines follow a more imperative programming model.
Source: https://jenkins.io/doc/book/pipeline/syntax/#compare
The script you configured via UI is using declarative pipeline syntax, while the other uses the scripted node syntax. I'd say that's probably where the other parser comes in and would agree that the one for pipeline has a bug.
Is it possible to create a Jenkins pipeline with an optional input stage?
The below snippet doesn't achieve this goal.
Expected behaviour
The stage (and therefore the input prompt) should only run for specific branches.
Actual behaviour
This stage runs for all branches. The when filter is ignored when an input step is used.
stage('Approve') {
when {
expression { BRANCH_NAME ==~ /^qa[\w-_]*$/ }
}
input {
message "Approve release?"
ok "y"
submitter "admin"
parameters {
string(name: 'IS_APPROVED', defaultValue: 'y', description: 'Deploy to master?')
}
}
steps {
script {
if (IS_APPROVED != 'y') {
currentBuild.result = "ABORTED"
error "User cancelled"
}
}
}
}
The filter is not ignored, it is just evaluated after the input step. In your example, you would always be asked whether to deploy, and in the case that you are not on a QA branch, nothing would happen.
Now you could ask why Jenkins isn't evaluating the 'when' directive first. In that case, you could not use the input parameter in your when condition.
And having multiple when directives would be like scripting within the declarative pipeline.
However, there is an expression that allows you controlling when the 'when' directive is evaluated. This is beforeAgent. It allows you to evaluate the when statement before the agent is allocated. Similar to that, you would need something like beforeInput. You could create a feature request for that.
I stepped away from using the input directive and I use input within a scripting block now, because that provides much more flexibility, e.g. I am sending Slack notifications when somebody has to approve something, which is impossible with the declarative approach. You would need a notify directive for that. And if there was one, is that going to be evaluated before or after the input step?
You see, doing everything declarative is not always the best way. So my recommended approach is the following (disclaimer: this is untested!):
pipeline {
// We want to use agents per stage to avoid blocking our build agents
// while we are waiting for user input.
agent none
...
// The question mark naming convention is helpful to show you which
// approval stage belongs to which work stage.
stage('Release?') {
// Don't allocate an agent because we don't want to block our
// slaves while waiting for user input.
agent none
when {
// You forgot the 'env.' in your example above ;)
expression { env.BRANCH_NAME ==~ /^qa[\w-_]*$/ }
}
options {
// Optionally, let's add a timeout that we don't allow ancient
// builds to be released.
timeout time: 14, unit: 'DAYS'
}
steps {
// Optionally, send some notifications to the approver before
// asking for input. You can't do that with the input directive
// without using an extra stage.
slackSend ...
// The input statement has to go to a script block because we
// want to assign the result to an environment variable. As we
// want to stay as declarative as possible, we put noting but
// this into the script block.
script {
// Assign the 'DO_RELEASE' environment variable that is going
// to be used in the next stage.
env.DO_RELEASE = input ...
}
// In case you approved multiple pipeline runs in parallel, this
// milestone would kill the older runs and prevent deploying
// older releases over newer ones.
milestone 1
}
}
stage('Release') {
// We need a real agent, because we want to do some real work.
agent any
when {
// Evaluate the 'when' directive before allocating the agent.
beforeAgent true
// Only execute the step when the release has been approved.
environment name: 'DO_RELEASE', value: 'yes'
}
steps {
// Make sure that only one release can happen at a time.
lock('release') {
// As using the first milestone only would introduce a race
// condition (assume that the older build would enter the
// milestone first, but the lock second) and Jenkins does
// not support inter-stage locks yet, we need a second
// milestone to make sure that older builds don't overwrite
// newer ones.
milestone 2
// Now do the actual work here.
...
}
}
}
The correct syntax would be more like (completely untested):
stage('Approve') {
when {
expression { BRANCH_NAME ==~ /^qa[\w-_]*$/ }
}
steps {
script {
def IS_APPROVED = input(
message: "Approve release?"
ok: "y"
submitter: "admin"
parameters: [
string(name: 'IS_APPROVED', defaultValue: 'y', description: 'Deploy to master?')
]
)
if (IS_APPROVED != 'y') {
currentBuild.result = "ABORTED"
error "User cancelled"
}
}
}
}
So essentially, you're hitting the limits of declarative pipelines have to fall back to groovy scripting / scripted pipelines.
I'm using declarative Jenkins pipelines to run some of my build pipelines and was wondering if it is possible to define multiple agent labels.
I have a number of build agents hooked up to my Jenkins and would like for this specific pipeline to be able to be built by various agents that have different labels (but not by ALL agents).
To be more concrete, let's say I have 2 agents with a label 'small', 4 with label 'medium' and 6 with label 'large'. Now I have a pipeline that is very resource-low and I want it to be executed on only a 'small'- or 'medium'-sized agent, but not on a large one as it may cause larger builds to wait in the queue for an unnecessarily long time.
All the examples I've seen so far only use one single label.
I tried something like this:
agent { label 'small, medium' }
But it failed.
I'm using version 2.5 of the Jenkins Pipeline Plugin.
You can see the 'Pipeline-syntax' help within your Jenkins installation and see the sample step "node" reference.
You can use exprA||exprB:
node('small||medium') {
// some block
}
EDIT: I misunderstood the question. This answer is only if you know
which specific agent you want to run for each stage.
If you need multiple agents you can declare agent none and then declare the agent at each stage.
https://jenkins.io/doc/book/pipeline/jenkinsfile/#using-multiple-agents
From the docs:
pipeline {
agent none
stages {
stage('Build') {
agent any
steps {
checkout scm
sh 'make'
stash includes: '**/target/*.jar', name: 'app'
}
}
stage('Test on Linux') {
agent {
label 'linux'
}
steps {
unstash 'app'
sh 'make check'
}
post {
always {
junit '**/target/*.xml'
}
}
}
stage('Test on Windows') {
agent {
label 'windows'
}
steps {
unstash 'app'
bat 'make check'
}
post {
always {
junit '**/target/*.xml'
}
}
}
}
}
This syntax appears to work for me:
agent { label 'linux && java' }
As described in Jenkins pipeline documentation and by Vadim Kotov one can use operators in label definition.
So in your case if you want to run your jobs on nodes with specific labels, the declarative way goes like this:
agent { label('small || medium') }
And here are some examples from Jenkins page using different operators
// with AND operator
agent { label('windows && jdk9 )') }
// a more complex one
agent { label('postgres && !vm && (linux || freebsd)') }
Notes
When constructing those definitions one just needs to consider following rules/restrictions:
All operators are left-associative
Labels or agent names can be surrounded with quotation marks if they contain characters that would conflict with the operator syntax
Expressions can be written without whitespace
Jenkins will ignore whitespace when evaluating expressions
Matching labels or agent names with wildcards or regular expressions is not supported
An empty expression will always evaluate to true, matching all agents
Create a another label call 'small-or-medium' that has 6 all agents. Then in Jenkinsfile:
agent { label 'small-or-medium' }