I've looked at this Pass Artifact or String to upstream job in Jenkins Pipeline and this Pass variables between Jenkins stages and this How do I pass variables between stages in a declarative Jenkins pipeline?, but none of these questions seem to deal with my specific problem.
Basically I have a pipeline consisting of multiple stages, each run in its own agent.
In the first stage I run a shell script. Here two variables are generated. I would like to use these variables in the next stage. The methods I've seen so far seem to only work when passing variables within the same agent.
pipeline {
stages {
stage("stage 1") {
agent {
docker {
image 'my_image:latest'
}
}
steps {
sh ("""
export VAR1=foo
export VAR2=bar
""")
}
}
stage("stage 2") {
agent {
docker {
image 'my_other_image:latest'
}
}
steps {
sh ("echo "$VAR1 $VAR2")
//expecting to see "foo bar" printed here
}
}
Related
I have two Jenkins Pipelines :
Pipeline A : In a stage, I defined an environment variable called MAVEN_PROFILE (the user can choose a value from a list)
Pipeline B : I need to get the MAVEN_PROFILE environment variable value that was set in Pipeline A
I need two pipelines because I can't do it in a single Pipeline for process reason.
I saw there was some answers on how to share variable between stages within a single Pipeline but this not my case.
I want to share environment variable value between different Pipelines.
Pipeline A
pipeline {
agent any
...
stages {
stage('Profile Selection'){
steps {
script {
env.MAVEN_PROFILE = input message: 'Choose the profile :',
parameters: [choice(name: 'MAVEN_PROFILE',
choices: 'all\nserver\nclient', description: 'Profiles')]
}
}
}
stage(...){
steps {
script {
bat "mvn deploy -P ${env.MAVEN_PROFILE}"
}
}
}
... other stages
}
}
Pipeline B
pipeline {
agent any
...
stages {
... other stages
stage(...){
steps {
script {
bat "mvn release ... -P ${env.environmentVariableValueFromPipelineA}"
}
}
}
}
}
They're not running in the same environment, so they can't directly share environment variables. The easiest is probably to write these values to a file in the workspace in pipeline A, and read them back in in pipeline B. Something like this:
Pipeline A:
sh "echo ${MAVEN_PROFILE} > .MAVEN_PROFILE"
Pipeline B:
def MAVEN_PROFILE = sh(script: 'cat .MAVEN_PROFILE', returnStdout: true).trim()
I have a pipeline with multiple stages, some of them are in parallel. Up until now I had a single code block indicating where the job should run.
pipeline {
triggers { pollSCM '0 0 * * 0' }
agent { dockerfile { label 'jenkins-slave'
filename 'Dockerfile'
}
}
stages{
stage('1'){
steps{ sh "blah" }
} // stage
} // stages
} // pipeline
What I need to do now is run a new stage on a different slave, NOT in docker.
I tried by adding an agent statement for that stage but it seems like it tries to run that stage withing a docker container on the second slave.
stage('test new slave') {
agent { node { label 'e2e-aws' } }
steps {
sh "ifconfig"
} // steps
} // stage
I get the following error message
13:14:23 unknown flag: --workdir
13:14:23 See 'docker exec --help'.
I tried setting the agent to none for the pipeline and using an agent for every step and have run into 2 issues
1. My post actions show an error
2. The stages that have parallel stages also had an error.
I can't find any examples that are similar to what I am doing.
You can use the node block to select a node to run a particular stage.
pipeline {
agent any
stages {
stage('Init') {
steps {
node('master'){
echo "Run inside a MASTER"
}
}
}
}
}
I am getting started with Jenkins Pipeline. My pipeline has one simple step that is supposed to run on a different agent - like the "Restrict where this project can be run" option.
My problem is that it is running on master.
They are both Windows machines.
Here's my Jenkinsfile:
pipeline {
agent {label 'myLabel'}
stages {
stage('Stage 1') {
steps {
echo pwd()
writeFile(file: 'test.txt', text: 'Hello, World!')
}
}
}
}
pwd() prints C:\Jenkins\workspace\<pipeline-name>_<branch-name>-Q762JIVOIJUFQ7LFSVKZOY5LVEW5D3TLHZX3UDJU5FWYJSNVGV4Q.
This folder is on master. This is confirmed by the presence of the test.txt file.
I expected test.txt to be created on the slave agent.
Note 1
I can confirm that the pipeline finds the agent because the logs contain:
[Pipeline] node
Running on MyAgent in C:\Jenkins\workspace\<pipeline-name>_<branch-name>-Q762JIVOIJUFQ7LFSVKZOY5LVEW5D3TLHZX3UDJU5FWYJSNVGV4Q
But this folder does not exist on MyAgent, which seems related to the problem.
Note 2
This question is similar to Jenkins pipeline not honoring agent specification
, except that I'm not using the build instruction so I don't think the answer applies.
Note 3
pipeline {
agent any
stages {
stage('Stage 1') {
steps {
echo "${env.NODE_NAME}"
}
}
stage('Stage 2') {
agent {label 'MyLabel'}
steps {
echo "${env.NODE_NAME}"
}
}
}
}
This prints the expected output - master and MyAgent. If this is correct, then why is the workspace located in a different folder on master instead of being on MyAgent?
here is an example
pipeline {
agent none
stages {
stage('Example Build') {
agent { label 'build-label' }
steps {
sh 'env'
sh ' sleep 8'
}
}
stage('Example Test') {
agent { label 'deploy-label' }
steps {
sh 'env'
sh ' sleep 5'
}
}
}
}
I faced similar issue and the following pipeline code worked for me (i.e. the file got created on the Windows slave instead of Windows master),
pipeline {
agent none
stages {
stage("Stage 1") {
steps {
node('myLabel'){
script {
writeFile(file: 'test.txt', text: 'Hello World!', encoding: 'UTF-8')
}
// This should print the file content on slave (Hello World!)
bat "type test.txt"
}
}
}
}
}
I'm debugging a completely unrelated issue and this fact was thrown in my face. Apparently the pipeline is processed in the built-in node (previously known as the master node), with the steps being forwarded to the agent.
So even though echo runs on the agent, but pwd() will run on the built-in node. You can do sh 'pwd' to get the path on the agent.
I have several pipeline jobs, which are configured very similarly.
They all have the same stages (of which there are about 10).
I am now I am thinking about moving to the declarative pipeline (https://jenkins.io/blog/2016/09/19/blueocean-beta-declarative-pipeline-pipeline-editor/).
But I do not want to define the ~10 stages in every pipeline. I want to define them at one place, and "import" them somehow.
Is this possible with declarative pipelines at all? I see that there are Libraries, but it does not seem like I could include the stage definition using them.
You will have to create a shared-library to implement what i am about to suggest. For shared-library implementation, you may check the following posts:
Using Building Blocks in Jenkins Declarative Pipeline
Upload file in Jenkins input step to workspace (Mainly for images so one can easily figure out things)
Now if you want to use a Jenkinsfile (kind of a template) which can be reused across multiple projects (jobs), then that is indeed possible.
Once you have created a shared-library repository with vars directory in it, then you just have to create a Groovy file (let's say, commonPipeline.groovy) inside vars directory.
Here's an example that works because I have used it earlier in multiple jobs.
$ cat shared-lib/vars/commonPipeline.groovy
// You can create function(s) as shown below, if required
def someFunctionA() {
// Your code
}
// This is where you will define all the stages that you want
// to run as a whole in multiple projects (jobs)
def call(Map config) {
pipeline {
agent {
node { label 'slaveA || slaveB' }
}
environment {
myvar_Y = 'apple'
myvar_Z = 'orange'
}
stages {
stage('Checkout') {
steps {
deleteDir()
checkout scm
}
}
stage ('Build') {
steps {
script {
check_something = someFunctionA()
if (check_something) {
echo "Build!"
# your_build_code
} else {
error "Something bad happened! Exiting..."
}
}
}
}
stage ('Test') {
steps {
echo "Running tests..."
// your_test_code
}
}
stage ('Deploy') {
steps {
script {
sh '''
# your_deploy_code
'''
}
}
}
}
post {
failure {
sh '''
# anything_you_need_to_perform_in_failure_step
'''
}
success {
sh '''
# anything_you_need_to_perform_in_success_step
'''
}
}
}
}
With above Groovy file in place, all you have to do now is to call it in your various Jenkins projects. Since you might already be having an existing Jenkinsfile (if not, create it) in your Jenkins project, you just have to replace the existing content of that file with the following:
$ cat Jenkinsfile
// Assuming you have named your shared-library as `my-shared-lib` & `Default version` to `master` branch in
// `Manage Jenkins` » `Configure System` » `Global Pipeline Libraries` section
#Library('my-shared-lib#master')_
def params = [:]
params=[
jenkins_var: "${env.JOB_BASE_NAME}",
]
commonPipeline params
Note: As you can see above, I am calling commonPipeline.groovy file. So, all your bulky Jenkinsfile will get reduced to just five or six lines of code, and those few lines are also going to be common across all those projects. Also note that I have used jenkins_var above. It can be any name. It's not actually used but is required for pipeline to run. Some Groovy expert can clarify that part.
Ref: https://www.jenkins.io/blog/2017/10/02/pipeline-templates-with-shared-libraries/
I'm building jobs that will be using Jenkins piplines extensively. Our nodes are designated per project by their tags, but unlike regular jobs the pipeline build does not seem to have the "Restrict where this project can be run" checkbox. How can I specify on which node the pipeline will run the way I do for regular jobs?
You specify the desired node or tag when you do the node step:
node('specialSlave') {
// Will run on the slave with name or tag specialSlave
}
See https://jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#node-allocate-node for an extended explanation of the arguments to node.
Edit 2019: This answer (and question) was made back in 2017, back then there only was one flavor of Jenkins pipeline, scripted pipeline, since then declarative pipeline has been added. So above answer is true for scripted pipeline, for answers regarding declarative pipeline please see other answers below.
Choosing a node which has the label X in a declarative pipeline with json format:
pipeline {
agent { label 'X' }
...
...
}
You also can apply multiple labels with or (||) or with and (&&) operator.
Running the job on any of the nodes which has label X or label Y:
agent { label 'X || Y' }
Running the job only on nodes which have both label:
agent { label 'X && Y' }
More in the Jenkins Pipeline reference guide.
ps: if you are reading this you probably have just started using Jenkins pipeline and you are not sure if you should use declarative or scripted pipeline. Short answer: it's better to start with declarative. From jenkins.io:
Declarative and Scripted Pipelines are constructed fundamentally
differently. Declarative Pipeline is a more recent feature of Jenkins
Pipeline which:
provides richer syntactical features over Scripted Pipeline syntax, and
is designed to make writing and reading Pipeline code easier.
To be clear, because Pipeline has two Syntax, there are two ways to achieve that.
Declarative
pipeline {
agent none
stages {
stage('Build') {
agent { label 'slave-node' }
steps {
echo 'Building..'
sh '''
'''
}
}
}
post {
success {
echo 'This will run only if successful'
}
}
}
Scripted
node('your-node') {
try {
stage 'Build'
node('build-run-on-this-node') {
sh ""
}
} catch(Exception e) {
throw e
}
}
Agent or Node where we should not execute the jenkins job :
This is the negation of the problem statment i.e. node where not to run
It was most weird solution to me but issue has been already raised to jenkins community
agent { label '!build-agent-name' }
If you need to run entire jenkins pipeline to run on single node, use the following format
pipeline {
agent {
label 'test1'
}
stages {
stage('Build') {
steps {
echo 'Building..'
}
}
stage('Test') {
steps {
echo 'Testing..'
}
}
stage('Deploy') {
steps {
echo 'Deploying....'
}
}
}
}
if you need to execute each stage in different nodes, use the below format,
pipeline {
agent any
stages {
stage('Build') {
steps {
echo 'Building..'
}
}
stage('Test') {
steps {
node("test1"){
echo 'Testing..'
}
}
}
stage('Deploy') {
steps {
echo 'Deploying....'
}
}
}
}