Glob patterns in Jenkinsfile sh (shell) steps - jenkins

I'm trying to set up a Jenkinsfile to run our CI pipeline. One of the steps will involve collecting files from across our directory tree and copying them into a single directory, for zipping up.
I'm attempting to do this using the Jenkins sh step and using glob patterns, but I can't seem to get this to work.
A simple example Jenkinsfile would be:
pipeline {
agent any
stages {
stage('List with Glob'){
steps{
sh 'ls **/*.xml'
}
}
}
}
I would expect that to list any .xml files in the workspace, but instead I receive:
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (List with Glob)
[Pipeline] sh
[jenkinsfile-pipeline] Running shell script
+ ls '**/*.xml'
ls: cannot access **/*.xml: No such file or directory
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE
I think i'm missing something with Groovy string interpolation, but I need some help for this specific case (running in a Jenkins pipeline via a Jenkinsfile)
Any help much appreciated!

As far as I can tell **/*.xml' isn't a valid glob pattern (see this). Instead what you have there is an ant naming pattern, which, as far as I know, isn't supported by bash (or sh). Instead what you wan't to do is to use find:
pipeline {
agent any
stages {
stage('List with find'){
steps{
sh "find . -type f -name '*.xml'"
}
}
}
}

Related

How can I pass environment variables to Jenkins Pipeline Tools section

In Jenkins > Global Tool Configuration > JDK installation > I have added JDK7 and its name is oracle-7u80; Similarly under Maven installation, I have added Maven 3.5 install and named it mvn.
Now I am using the above two installs in the Pipeline script:
pipeline {
agent any
tools {
maven 'mvn'
jdk 'oracle-7u80'
}
stages {
stage('Example') {
steps {
}
}
}
}
I do not want to hard code the jdk and Maven values in the Tools section in the pipeline. I want to pass these values via environment variables or properties so that I can manage them externally.
Is there a way to pass the values (mvn or oracle-7u80) that is defined to Maven and jdk in the tools using environment variables?
Like if I need to inject a value within Steps/Script section, in Jenkins pipeline, I can define globally in the environment variables or using Jenkins project
Configure
General
Check mark Prepare an environment for the run
Check mark Keep Jenkins environment variables
I can provide the environment variable in the properties content with Properties File definition.
My intention is to get a format like this:
pipeline {
agent any
tools {
maven '${MVN_VERSION}'
jdk '${ORACLE_VERSION'}
}
stages {
stage('Example') {
steps {
}
}
}
}
Pipeline projects are often used with a Jenkinsfile (Pipeline script from SCM in the Pipeline → Definition drop-down list) to bind a source code version and its build configuration to each other for reproducable builds.
Injecting build tool versions from external before the build contradicts this idea.
I'm also not sure whether this is even possible conceptually since (environment) variables' values from external are set in stages ... script which is a totally different declaration branch than tools. But hey, it's called declarative pipeline, not imperative, so order shouldn't matter ... in theory. I'll give it a try.
For passing external values into internal variables in general see Pipeline: Nodes and Processes, sh: Shell Script and also the answer to the question How to access Shell variable value into Groovy pipeline script.
Maven version injection try
pipeline {
agent any
tools {
maven "${MVN_VERSION}"
}
stages {
stage('Try: Maven version injected') {
steps {
script {
env.MVN_VERSION = sh script: 'echo "Maven 3.8.1"', returnStdout: true
}
echo "${MVN_VERSION}"
}
}
}
}
As expected:
[Pipeline] stage
[Pipeline] { (Declarative: Tool Install)
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
groovy.lang.MissingPropertyException: No such property: MVN_VERSION for class: groovy.lang.Binding
...
Another idea that came into my mind is to make this project parameterized with two parameters (e.g. MVN_GLOBAL_TOOL_NAME, JDK_GLOBAL_TOOL_NAME) via Choice parameter s, for instance, and this works:
pipeline {
agent any
tools {
maven "${MVN_GLOBAL_TOOL_NAME}" // coming from parameterized project's build parameter
}
stages {
stage('Maven tool as build parameter') {
steps {
echo "MVN_GLOBAL_TOOL_NAME=${MVN_GLOBAL_TOOL_NAME}"
}
}
}
}
Console Outpout
[Pipeline] stage
[Pipeline] { (Declarative: Tool Install)
[Pipeline] tool
[Pipeline] envVarsForTool
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Maven version as build parameter)
[Pipeline] tool
[Pipeline] envVarsForTool
[Pipeline] withEnv
[Pipeline] {
[Pipeline] script
[Pipeline] {
[Pipeline] echo
MVN_GLOBAL_TOOL_NAME=Maven 3.8.1
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
See also ${JENKINS_URL}/job/${JOB_NAME}/api/:
Perform a build
If the build has parameters, post to this URL [Link note: ${JENKINS_URL}/job/${JOB_NAME}/buildWithParameters] and provide the parameters as form data.
See also: ${JENKINS_URL}/env-vars.html/.

Can I assign a node dynamically within an iterative loop in Jenkins?

I am building a Jenkins pipeline for a scenario where I'll have to use specific Jenkins agents inside remote data centers to deploy my code to those data centers. This is due to firewall restrictions on some ports, specifically WinRM is blocked between some of our global data centers.
Our deploys are written so that a single deploy stage can deploy to any number of environments, specified by the user's passed-in parameters. The stage loops through the environments and calls a generic deploy script for each one.
I know how to specify an agent by its label or other closure in a stage's definition:
stage ('a stage') {
agent { label 'some agent label' }
steps { ...
but in this case, i am solving for deploying to multiple environments in one deploy stage, each of which will require its own agent.
I can, of course, specify a unique stage for each env, and use a when clause to run it when appropriate, but that's messy.
What I'd like to do is tell the pipeline what agent(s) to use for the deploy stage inside inside the deploy stage, and be able to use multiple agents within that single stage, determined dynamically based on the parameters of the run.
I'd originally found this answer on SO, which gave me the idea of acquiring a node inside the stage, and not with the agent declaration. It doesn't show the acquisition inside a script block, but I'd initially read it that way, and that gave me the idea to try acquiring the node inside a script. And once you're there, it's a small leap to try doing it in a loop.
To prove it, I print some local environment variables from the agent to prove that we're switching agents, inside the stage, inside the loop. I'm also passing a file to each agent to prove that I can pass the files through the firewall.
Note that to even connect to the agent behind the firewall, we had to open the port that is defined in the Jenkins global security config, inbound to the agent from the controller (aka master), and https (443) outbound to the controller. The inbound port is configured to be static.
pipeline {
agent none
stages {
stage ('init') {
agent any
steps {
writeFile file: 'tester', text: 'i am a test file'
stash includes: 'tester', name: 'tester'
}
}
stage ('get agents') {
steps {
script {
['Agent1', 'Agent2'].each { agent ->
node (agent) {
echo "I am agent `${NODE_NAME}`\nMy labels are `${NODE_LABELS}`"
unstash 'tester'
echo "the content of the file is `${readFile 'tester'}`"
}
}
}
}
}
}
}
Which outputs:
Started by user Maximilian Cascone Admin
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] stage
[Pipeline] { (init)
[Pipeline] node
Running on Agent1 in /mnt/data/jenkins/workspace/Sandbox/mcascone/dynamic-agents
[Pipeline] {
[Pipeline] writeFile
[Pipeline] stash
Stashed 1 file(s)
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (get agents)
[Pipeline] script
[Pipeline] {
[Pipeline] node
Running on Agent2 in D:\workspace\Sandbox\mcascone\dynamic-agents
[Pipeline] {
[Pipeline] echo
I am agent `Agent2`
My labels are `Cider Redgate Windows Worker02 ant chef npm relativity wix`
[Pipeline] unstash
[Pipeline] readFile
[Pipeline] echo
the content of the file is `i am a test file`
[Pipeline] }
[Pipeline] // node
[Pipeline] node
Running on Agent1 in C:\jenkins\workspace\Sandbox\mcascone\dynamic-agents
[Pipeline] {
[Pipeline] echo
I am agent `Agent1`
My labels are `Itar Agent1`
[Pipeline] unstash
[Pipeline] readFile
[Pipeline] echo
the content of the file is `i am a test file`
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
Finished: SUCCESS
So I've proven that I can get multiple agents dynamically within a single stage. Next step would be elevating this into a shared step, so it can be called without the script block and make the pipeline nice and neat. But as a POC, this is a great achievement. I don't believe I've seen this elsewhere.
The answer might not fit all needs you have, but dynamic generation of stage and on that basis you can assign / execute the generate stages with following way.
def agents = ['master', 'agent1', 'agent2']
def generateStage(nodeLabel) {
return {
stage("Runs on ${nodeLabel}") {
node(nodeLabel) {
echo "Running on ${nodeLabel}"
}
}
}
}
def parallelStagesMap = agents.collectEntries {
["${it}" : generateStage(it)]
}
pipeline {
agent none
stages {
stage('non-parallel stage') {
steps {
echo 'This stage will be executed first.'
}
}
stage('parallel stage') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
Furthermore you can use collectEntries, out of the function box parallelStagesMap, in this way you can use each collect entry for different stage and can dynamically assign nodes in the stage, and in function generateStage you need to do modification as per your requirement.
If you wanted to execute these stages sequentially, then remove parallel from script.generateStage contains return which is imporant without that, pipeline will not work as expected.

Jenkins pipeline with parallel stages causes "process apparently never started" error

I'm trying to set up a Jenkins pipeline (using the declarative syntax) that runs unit and feature tests on two separate, on-demand AWS EC2 instances. The pipeline works perfectly when run on a single instance and without the parallel stages. As soon as I switch to parallel stages, any shell script fails with this cryptic message:
process apparently never started in
/home/admin/workspace/GSWebRuby_Test#tmp/durable-b0d8c4b4 (running
Jenkins temporarily with
-Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true
might make the problem clearer)
I've googled extensively and came across several bug reports of the Durable Task plugin that appears to be responsible for this message. I'm using the latest version of the plugin v. 1.33 and none of the problems seem to apply to my case, e.g. failures on unusual architectures or when running Docker containers. I've also down- and re-upgaded the plugin (toggling between versions 1.30 and 1.33). Also, to re-iterate, sh commands work without issue when I don't use the parallel stages.
I've created a simplified pipeline to debug the problem. Note that the shell commands are also simple, e.g. "env | sort", or "pwd".
pipeline {
agent none
environment {
DB_USER = credentials('db-user')
DB_PASS = credentials('db-pass')
}
stages {
stage('Setup'){
failFast false
parallel {
stage('foo') {
agent {
label 'jenkins-slave-ondemand'
}
steps {
echo 'In stage foo'
sh 'env|sort'
}
}
stage('bar') {
agent {
label 'jenkins-slave-ondemand'
}
steps {
echo 'In stage bar'
sh 'pwd'
}
}
}
}
}
}
This is the console output:
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] withCredentials
Masking supported pattern matches of $DB_PASS or $DB_USER
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Setup)
[Pipeline] parallel
[Pipeline] { (Branch: foo)
[Pipeline] { (Branch: bar)
[Pipeline] stage
[Pipeline] { (foo)
[Pipeline] stage
[Pipeline] { (bar)
[Pipeline] node
[Pipeline] node
Still waiting to schedule task
All nodes of label ‘jenkins-slave-ondemand’ are offline
Still waiting to schedule task
All nodes of label ‘jenkins-slave-ondemand’ are offline
Running on EC2 (Jenkins AWS EC2) - Jenkins slave (i-0982299c572100c71) in /home/admin/workspace/GSWebRuby_Test
[Pipeline] {
[Pipeline] echo
In stage foo
[Pipeline] sh
Running on EC2 (Jenkins AWS EC2) - Jenkins slave (i-092ecac8e6c257270) in /home/admin/workspace/GSWebRuby_Test
[Pipeline] {
[Pipeline] echo
In stage bar
[Pipeline] sh
process apparently never started in /home/admin/workspace/GSWebRuby_Test#tmp/durable-b0d8c4b4
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch foo
process apparently never started in /home/admin/workspace/GSWebRuby_Test#tmp/durable-b6cfcff9
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch bar
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] End of Pipeline
ERROR: script returned exit code -2
Finished: FAILURE
Am I doing something wrong in the way I've set up the pipeline? Any pointers would be greatly appreciated.
Edit:
After setting this JENKINS_JAVA_OPTIONS org.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true, I see this additional output:
In stage bar
[Pipeline] sh
nohup: failed to run command 'sh': No such file or directory
process apparently never started in /home/admin/workspace/GSWebRuby_Test#tmp/durable-099a2e56

jenkins pipeline not build remotely

I am trying to build a job by pipeline to my other slave in the master
the pipeline is like this
pipeline {
agent {
label "virtual"
}
stages {
stage("test one") {
steps {
echo " test test test"
}
}
stage("test two") {
steps {
echo " testttttttttt "
}
}
}
}
they syntax not getting the error but it doesn't build on my slave server,
but when I run on freestyle job by Restrict where this project can be run with that label then execute sheel by echo "test test"
it was built on my slave server,
what is wrong with my pipeline ? do I missing something?
after build
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on virtual in /home/virtual/jenkins/workspace/demoo
[Pipeline] {
[Pipeline] stage
[Pipeline] { (test one)
[Pipeline] echo
test test test
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (test two)
[Pipeline] echo
testttttttttt
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Add the path you want in the Remote root directory (yellow column) as shown below:-
The build works like you did it already. The steps will be executed on the slave. If you add something like clone a repository to your step, your workspace directory will be created.
Pipeline and Freestylejobs are working here different. The Freestylejob will make the directory in workspace as soon as it runs at the first time. The Pipelinejob will create the directory as soon as it needs this this directory.
I created a simple Pipeline:
pipeline {
agent {
label "linux"
}
stages {
stage("test one") {
steps {
sh "echo 'test test test' > text.txt"
}
}
}
}
I converted your echo to a sh command because my Slave is a linux slave. The sh step creates a text.txt file. As soon as I run this job, the directory will be created:
[<user>#<server> test-pipeline]$ pwd
/var/lib/jenkins/workspace/test-pipeline
[<user>#<server> test-pipeline]$ ls -l
total 4
-rw-r----- 1 <user> <group> 15 Oct 7 16:49 text.txt

#NonCPS stops after first build step

In a #NonCPS annotated function, only code up to the very first jenkins build step is executed. Does anyone have the same problem? Am I missing something? I am using Jenkins LTS... just sayin' (2.73.2).
This is my code:
#NonCPS
def hello() {
println 'Output "hello":'
sh 'echo Hello'
println 'Output "World":'
sh 'echo World'
}
node {
stage('Test') {
hello()
}
}
I would expect this code to run properly, but the output is the following:
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/Sandbox/pipeline-test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] echo
Output "hello":
[Pipeline] sh
[pipeline-test] Running shell script
+ echo Hello
Hello
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
You cannot run build steps inside #NonCPS methods. Pipeline scripts are considered "serializable", allowing them to be durable across system failures etc. Only a subset of the capabilities of groovy used by pipeline scripts is serializable - for anything that is not, you use #NonCPS to execute it.
Essentially, your #NonCPS method needs to do its business and return data back to the "safe", serialized execution stack.
In your particular example code I see no reason why hello() has to be #NonCPS at all - I can only assume your real function is doing something more complex.
(Edit)
Having just looked at your question history and the original script; I don't know if this is still the case with the latest versions but when I was writing our scripts ~6 months ago, each { thing -> iteration was not serializable.

Resources