I'm building jobs that will be using Jenkins piplines extensively. Our nodes are designated per project by their tags, but unlike regular jobs the pipeline build does not seem to have the "Restrict where this project can be run" checkbox. How can I specify on which node the pipeline will run the way I do for regular jobs?
You specify the desired node or tag when you do the node step:
node('specialSlave') {
// Will run on the slave with name or tag specialSlave
}
See https://jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#node-allocate-node for an extended explanation of the arguments to node.
Edit 2019: This answer (and question) was made back in 2017, back then there only was one flavor of Jenkins pipeline, scripted pipeline, since then declarative pipeline has been added. So above answer is true for scripted pipeline, for answers regarding declarative pipeline please see other answers below.
Choosing a node which has the label X in a declarative pipeline with json format:
pipeline {
agent { label 'X' }
...
...
}
You also can apply multiple labels with or (||) or with and (&&) operator.
Running the job on any of the nodes which has label X or label Y:
agent { label 'X || Y' }
Running the job only on nodes which have both label:
agent { label 'X && Y' }
More in the Jenkins Pipeline reference guide.
ps: if you are reading this you probably have just started using Jenkins pipeline and you are not sure if you should use declarative or scripted pipeline. Short answer: it's better to start with declarative. From jenkins.io:
Declarative and Scripted Pipelines are constructed fundamentally
differently. Declarative Pipeline is a more recent feature of Jenkins
Pipeline which:
provides richer syntactical features over Scripted Pipeline syntax, and
is designed to make writing and reading Pipeline code easier.
To be clear, because Pipeline has two Syntax, there are two ways to achieve that.
Declarative
pipeline {
agent none
stages {
stage('Build') {
agent { label 'slave-node' }
steps {
echo 'Building..'
sh '''
'''
}
}
}
post {
success {
echo 'This will run only if successful'
}
}
}
Scripted
node('your-node') {
try {
stage 'Build'
node('build-run-on-this-node') {
sh ""
}
} catch(Exception e) {
throw e
}
}
Agent or Node where we should not execute the jenkins job :
This is the negation of the problem statment i.e. node where not to run
It was most weird solution to me but issue has been already raised to jenkins community
agent { label '!build-agent-name' }
If you need to run entire jenkins pipeline to run on single node, use the following format
pipeline {
agent {
label 'test1'
}
stages {
stage('Build') {
steps {
echo 'Building..'
}
}
stage('Test') {
steps {
echo 'Testing..'
}
}
stage('Deploy') {
steps {
echo 'Deploying....'
}
}
}
}
if you need to execute each stage in different nodes, use the below format,
pipeline {
agent any
stages {
stage('Build') {
steps {
echo 'Building..'
}
}
stage('Test') {
steps {
node("test1"){
echo 'Testing..'
}
}
}
stage('Deploy') {
steps {
echo 'Deploying....'
}
}
}
}
Related
Now I'm building using execute shell in Jenkins.
(currently) The code below is built in order. I want to implement this in parallel.
now code status
(I want) build action -> test1.py ~ test4.py executed in parallel
Is there a way to build in parallel in this way(execute shell) or other strategy?
You have several options to run things in parallel within a Jenkins pipeline.
The first option is to use the static Parallel Directive Stages which allow you to easily define parallel stages inside your declarative pipeline, something like:
pipeline {
agent any
stages {
stage('Non-Parallel Stage') {
steps {
echo 'This stage will be executed first.'
}
}
stage('Parallel Stages') {
parallel {
stage('Test 1') {
steps {
sh "python3 $WORKSPACE/folder/test1.py"
}
}
stage('Test 2') {
steps {
sh "python3 $WORKSPACE/folder/test2.py"
}
}
.....
}
}
}
}
A second and more dynamic option is to use the built in parallel keyword which takes a map from branch names to closures:
parallel firstBranch: {
// do something
}, secondBranch: {
// do something else
},
failFast: true|false
and use it to dynamically create your parallel execution steps, something like:
tests = ['test1','test2','test3', 'test4']
parallel tests.collectEntries{ test ->
["Running test ${test}" : {
sh "python3 $WORKSPACE/folder/${test}.py"
}]
}
This code can reside anywhere in a scripted pipeline, and in a script directive in a declarative pipeline.
I've looked at this Pass Artifact or String to upstream job in Jenkins Pipeline and this Pass variables between Jenkins stages and this How do I pass variables between stages in a declarative Jenkins pipeline?, but none of these questions seem to deal with my specific problem.
Basically I have a pipeline consisting of multiple stages, each run in its own agent.
In the first stage I run a shell script. Here two variables are generated. I would like to use these variables in the next stage. The methods I've seen so far seem to only work when passing variables within the same agent.
pipeline {
stages {
stage("stage 1") {
agent {
docker {
image 'my_image:latest'
}
}
steps {
sh ("""
export VAR1=foo
export VAR2=bar
""")
}
}
stage("stage 2") {
agent {
docker {
image 'my_other_image:latest'
}
}
steps {
sh ("echo "$VAR1 $VAR2")
//expecting to see "foo bar" printed here
}
}
I want to run Jenkins scripted pipeline job at the specified slave at the moment when no other job is running on it.
After my job will be started, no other jobs should be performed on this slave, they will have to wait for my job ending running
All the tutorials that I found allowed me to run job on the node when it is free but did not protect me from launching other jobs on this node
Could you tell me how can I do this?
Because Pipeline has two Syntax, there are two ways to achieve that. For scripted pipeline please check the second one.
Declarative
pipeline {
agent none
stages {
stage('Build') {
agent { label 'slave-node' }
steps {
echo 'Building..'
sh '''
'''
}
}
}
post {
success {
echo 'This will run only if successful'
}
}
}
Scripted
node('your-node') {
try {
stage 'Build'
node('build-run-on-this-node') {
sh ""
}
} catch(Exception e) {
throw e
}
}
I have a pipeline with multiple stages, some of them are in parallel. Up until now I had a single code block indicating where the job should run.
pipeline {
triggers { pollSCM '0 0 * * 0' }
agent { dockerfile { label 'jenkins-slave'
filename 'Dockerfile'
}
}
stages{
stage('1'){
steps{ sh "blah" }
} // stage
} // stages
} // pipeline
What I need to do now is run a new stage on a different slave, NOT in docker.
I tried by adding an agent statement for that stage but it seems like it tries to run that stage withing a docker container on the second slave.
stage('test new slave') {
agent { node { label 'e2e-aws' } }
steps {
sh "ifconfig"
} // steps
} // stage
I get the following error message
13:14:23 unknown flag: --workdir
13:14:23 See 'docker exec --help'.
I tried setting the agent to none for the pipeline and using an agent for every step and have run into 2 issues
1. My post actions show an error
2. The stages that have parallel stages also had an error.
I can't find any examples that are similar to what I am doing.
You can use the node block to select a node to run a particular stage.
pipeline {
agent any
stages {
stage('Init') {
steps {
node('master'){
echo "Run inside a MASTER"
}
}
}
}
}
I am getting started with Jenkins Pipeline. My pipeline has one simple step that is supposed to run on a different agent - like the "Restrict where this project can be run" option.
My problem is that it is running on master.
They are both Windows machines.
Here's my Jenkinsfile:
pipeline {
agent {label 'myLabel'}
stages {
stage('Stage 1') {
steps {
echo pwd()
writeFile(file: 'test.txt', text: 'Hello, World!')
}
}
}
}
pwd() prints C:\Jenkins\workspace\<pipeline-name>_<branch-name>-Q762JIVOIJUFQ7LFSVKZOY5LVEW5D3TLHZX3UDJU5FWYJSNVGV4Q.
This folder is on master. This is confirmed by the presence of the test.txt file.
I expected test.txt to be created on the slave agent.
Note 1
I can confirm that the pipeline finds the agent because the logs contain:
[Pipeline] node
Running on MyAgent in C:\Jenkins\workspace\<pipeline-name>_<branch-name>-Q762JIVOIJUFQ7LFSVKZOY5LVEW5D3TLHZX3UDJU5FWYJSNVGV4Q
But this folder does not exist on MyAgent, which seems related to the problem.
Note 2
This question is similar to Jenkins pipeline not honoring agent specification
, except that I'm not using the build instruction so I don't think the answer applies.
Note 3
pipeline {
agent any
stages {
stage('Stage 1') {
steps {
echo "${env.NODE_NAME}"
}
}
stage('Stage 2') {
agent {label 'MyLabel'}
steps {
echo "${env.NODE_NAME}"
}
}
}
}
This prints the expected output - master and MyAgent. If this is correct, then why is the workspace located in a different folder on master instead of being on MyAgent?
here is an example
pipeline {
agent none
stages {
stage('Example Build') {
agent { label 'build-label' }
steps {
sh 'env'
sh ' sleep 8'
}
}
stage('Example Test') {
agent { label 'deploy-label' }
steps {
sh 'env'
sh ' sleep 5'
}
}
}
}
I faced similar issue and the following pipeline code worked for me (i.e. the file got created on the Windows slave instead of Windows master),
pipeline {
agent none
stages {
stage("Stage 1") {
steps {
node('myLabel'){
script {
writeFile(file: 'test.txt', text: 'Hello World!', encoding: 'UTF-8')
}
// This should print the file content on slave (Hello World!)
bat "type test.txt"
}
}
}
}
}
I'm debugging a completely unrelated issue and this fact was thrown in my face. Apparently the pipeline is processed in the built-in node (previously known as the master node), with the steps being forwarded to the agent.
So even though echo runs on the agent, but pwd() will run on the built-in node. You can do sh 'pwd' to get the path on the agent.