I noticed that in groovy, an expression like null || 'world' evaluates to true:
$ groovy -v
Groovy Version: 2.4.7 JVM: 1.8.0_60 Vendor: Oracle Corporation OS: Linux
$ cat hello.groovy
print "hello " + (null || "world")
$ groovy hello.groovy
hello true
But while experimenting with jenkins pipelines, I stumbled on null || 'world' evaluating to 'world'. e.g. with a pipeline like this:
pipeline {
agent { label 'test' }
stages {
stage('Run script') {
steps {
print "hello " + (null || "world")
}
}
}
}
I get output like this:
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on mynode in /path/to/ws
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Run script)
[Pipeline] echo
hello world
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
In general I'm not super familiar with either groovy or jenkins pipelines, so I just wonder what explains this difference? The latter behavior is the one I prefer -- is it safe to use and rely on?
Related
In Jenkins > Global Tool Configuration > JDK installation > I have added JDK7 and its name is oracle-7u80; Similarly under Maven installation, I have added Maven 3.5 install and named it mvn.
Now I am using the above two installs in the Pipeline script:
pipeline {
agent any
tools {
maven 'mvn'
jdk 'oracle-7u80'
}
stages {
stage('Example') {
steps {
}
}
}
}
I do not want to hard code the jdk and Maven values in the Tools section in the pipeline. I want to pass these values via environment variables or properties so that I can manage them externally.
Is there a way to pass the values (mvn or oracle-7u80) that is defined to Maven and jdk in the tools using environment variables?
Like if I need to inject a value within Steps/Script section, in Jenkins pipeline, I can define globally in the environment variables or using Jenkins project
Configure
General
Check mark Prepare an environment for the run
Check mark Keep Jenkins environment variables
I can provide the environment variable in the properties content with Properties File definition.
My intention is to get a format like this:
pipeline {
agent any
tools {
maven '${MVN_VERSION}'
jdk '${ORACLE_VERSION'}
}
stages {
stage('Example') {
steps {
}
}
}
}
Pipeline projects are often used with a Jenkinsfile (Pipeline script from SCM in the Pipeline → Definition drop-down list) to bind a source code version and its build configuration to each other for reproducable builds.
Injecting build tool versions from external before the build contradicts this idea.
I'm also not sure whether this is even possible conceptually since (environment) variables' values from external are set in stages ... script which is a totally different declaration branch than tools. But hey, it's called declarative pipeline, not imperative, so order shouldn't matter ... in theory. I'll give it a try.
For passing external values into internal variables in general see Pipeline: Nodes and Processes, sh: Shell Script and also the answer to the question How to access Shell variable value into Groovy pipeline script.
Maven version injection try
pipeline {
agent any
tools {
maven "${MVN_VERSION}"
}
stages {
stage('Try: Maven version injected') {
steps {
script {
env.MVN_VERSION = sh script: 'echo "Maven 3.8.1"', returnStdout: true
}
echo "${MVN_VERSION}"
}
}
}
}
As expected:
[Pipeline] stage
[Pipeline] { (Declarative: Tool Install)
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
groovy.lang.MissingPropertyException: No such property: MVN_VERSION for class: groovy.lang.Binding
...
Another idea that came into my mind is to make this project parameterized with two parameters (e.g. MVN_GLOBAL_TOOL_NAME, JDK_GLOBAL_TOOL_NAME) via Choice parameter s, for instance, and this works:
pipeline {
agent any
tools {
maven "${MVN_GLOBAL_TOOL_NAME}" // coming from parameterized project's build parameter
}
stages {
stage('Maven tool as build parameter') {
steps {
echo "MVN_GLOBAL_TOOL_NAME=${MVN_GLOBAL_TOOL_NAME}"
}
}
}
}
Console Outpout
[Pipeline] stage
[Pipeline] { (Declarative: Tool Install)
[Pipeline] tool
[Pipeline] envVarsForTool
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Maven version as build parameter)
[Pipeline] tool
[Pipeline] envVarsForTool
[Pipeline] withEnv
[Pipeline] {
[Pipeline] script
[Pipeline] {
[Pipeline] echo
MVN_GLOBAL_TOOL_NAME=Maven 3.8.1
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
See also ${JENKINS_URL}/job/${JOB_NAME}/api/:
Perform a build
If the build has parameters, post to this URL [Link note: ${JENKINS_URL}/job/${JOB_NAME}/buildWithParameters] and provide the parameters as form data.
See also: ${JENKINS_URL}/env-vars.html/.
I am trying in one of the steps to write some data in file, and in step after that to read that data using and assign it to a variable. This is my declarative Jenkins pipeline:
pipeline {
agent {label 'build-slave-aws'}
stages {
stage('Notify about start') {
steps {
sh 'echo "Some fatct with brackets ()" > /tmp/facts.issues'
}
}
stage('Gather the facts') {
steps {
script {
factsIssues = sh( script: "cat /tmp/facts.issues", returnStdout: true )
}
sh "echo these are facts: ${factsIssues}"
}
}
}
}
The output of this run is following:
Started by user 123
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on i-000df827977fd5175 in /workspace/workspace/test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Notify about start)
[Pipeline] sh
+ echo 'Some fatct with brackets ()'
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Gather the facts)
[Pipeline] script
[Pipeline] {
[Pipeline] sh
+ cat /tmp/facts.issues
[Pipeline] }
[Pipeline] // script
[Pipeline] sh
/workspace/workspace/test#tmp/durable-2a1f8cdf/script.sh: line 1: syntax error near unexpected token `('
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE
Any other text that doesn't contain ( simply work just fine. Do you have any suggestion on how I can write some data in file with ( and reading that back to a variable?
You are missing quotes around the argument of the echo command within the last sh script.
Fix:
sh "echo 'these are facts: ${factsIssues}'"
Suppose the yaml file is like this：
#test.yaml
0.6.5.1.0:
module:
- mysql
- zk
0.7.1.0.0:
module:
- java
Now I want to get the module list of specified version, and the version is a variable, I try to write the jenkins pipeline like this:
yamlFile = readYaml file: test.yaml
version = '0.7.1.0.0'
moduleList = yamlFile.get("${version}").get(module)
but this can't work, yamlFile.get("${version}") is a null object, how can I achieve this?
This works for me:
pipeline {
agent any
stages {
stage ('read') {
steps {
script {
def data = readYaml text: """
0.6.5.1.0:
module:
- mysql
- zk
0.7.1.0.0:
module:
- java
"""
version = '0.7.1.0.0'
println data.get(version).get('module')
}
}
}
}
}
The output:
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on server in /home/user/workspace/task
[Pipeline] {
[Pipeline] stage
[Pipeline] { (read)
[Pipeline] script
[Pipeline] {
[Pipeline] readYaml
[Pipeline] echo
[java]
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
I'm trying to set up a Jenkins pipeline (using the declarative syntax) that runs unit and feature tests on two separate, on-demand AWS EC2 instances. The pipeline works perfectly when run on a single instance and without the parallel stages. As soon as I switch to parallel stages, any shell script fails with this cryptic message:
process apparently never started in
/home/admin/workspace/GSWebRuby_Test#tmp/durable-b0d8c4b4 (running
Jenkins temporarily with
-Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true
might make the problem clearer)
I've googled extensively and came across several bug reports of the Durable Task plugin that appears to be responsible for this message. I'm using the latest version of the plugin v. 1.33 and none of the problems seem to apply to my case, e.g. failures on unusual architectures or when running Docker containers. I've also down- and re-upgaded the plugin (toggling between versions 1.30 and 1.33). Also, to re-iterate, sh commands work without issue when I don't use the parallel stages.
I've created a simplified pipeline to debug the problem. Note that the shell commands are also simple, e.g. "env | sort", or "pwd".
pipeline {
agent none
environment {
DB_USER = credentials('db-user')
DB_PASS = credentials('db-pass')
}
stages {
stage('Setup'){
failFast false
parallel {
stage('foo') {
agent {
label 'jenkins-slave-ondemand'
}
steps {
echo 'In stage foo'
sh 'env|sort'
}
}
stage('bar') {
agent {
label 'jenkins-slave-ondemand'
}
steps {
echo 'In stage bar'
sh 'pwd'
}
}
}
}
}
}
This is the console output:
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] withCredentials
Masking supported pattern matches of $DB_PASS or $DB_USER
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Setup)
[Pipeline] parallel
[Pipeline] { (Branch: foo)
[Pipeline] { (Branch: bar)
[Pipeline] stage
[Pipeline] { (foo)
[Pipeline] stage
[Pipeline] { (bar)
[Pipeline] node
[Pipeline] node
Still waiting to schedule task
All nodes of label ‘jenkins-slave-ondemand’ are offline
Still waiting to schedule task
All nodes of label ‘jenkins-slave-ondemand’ are offline
Running on EC2 (Jenkins AWS EC2) - Jenkins slave (i-0982299c572100c71) in /home/admin/workspace/GSWebRuby_Test
[Pipeline] {
[Pipeline] echo
In stage foo
[Pipeline] sh
Running on EC2 (Jenkins AWS EC2) - Jenkins slave (i-092ecac8e6c257270) in /home/admin/workspace/GSWebRuby_Test
[Pipeline] {
[Pipeline] echo
In stage bar
[Pipeline] sh
process apparently never started in /home/admin/workspace/GSWebRuby_Test#tmp/durable-b0d8c4b4
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch foo
process apparently never started in /home/admin/workspace/GSWebRuby_Test#tmp/durable-b6cfcff9
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch bar
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] End of Pipeline
ERROR: script returned exit code -2
Finished: FAILURE
Am I doing something wrong in the way I've set up the pipeline? Any pointers would be greatly appreciated.
Edit:
After setting this JENKINS_JAVA_OPTIONS org.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true, I see this additional output:
In stage bar
[Pipeline] sh
nohup: failed to run command 'sh': No such file or directory
process apparently never started in /home/admin/workspace/GSWebRuby_Test#tmp/durable-099a2e56
I am trying to build a job by pipeline to my other slave in the master
the pipeline is like this
pipeline {
agent {
label "virtual"
}
stages {
stage("test one") {
steps {
echo " test test test"
}
}
stage("test two") {
steps {
echo " testttttttttt "
}
}
}
}
they syntax not getting the error but it doesn't build on my slave server,
but when I run on freestyle job by Restrict where this project can be run with that label then execute sheel by echo "test test"
it was built on my slave server,
what is wrong with my pipeline ? do I missing something?
after build
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on virtual in /home/virtual/jenkins/workspace/demoo
[Pipeline] {
[Pipeline] stage
[Pipeline] { (test one)
[Pipeline] echo
test test test
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (test two)
[Pipeline] echo
testttttttttt
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Add the path you want in the Remote root directory (yellow column) as shown below:-
The build works like you did it already. The steps will be executed on the slave. If you add something like clone a repository to your step, your workspace directory will be created.
Pipeline and Freestylejobs are working here different. The Freestylejob will make the directory in workspace as soon as it runs at the first time. The Pipelinejob will create the directory as soon as it needs this this directory.
I created a simple Pipeline:
pipeline {
agent {
label "linux"
}
stages {
stage("test one") {
steps {
sh "echo 'test test test' > text.txt"
}
}
}
}
I converted your echo to a sh command because my Slave is a linux slave. The sh step creates a text.txt file. As soon as I run this job, the directory will be created:
[<user>#<server> test-pipeline]$ pwd
/var/lib/jenkins/workspace/test-pipeline
[<user>#<server> test-pipeline]$ ls -l
total 4
-rw-r----- 1 <user> <group> 15 Oct 7 16:49 text.txt