Jenkins Pipeline sleep(10) prevents function from being completed - jenkins

I have found a strange issue with Groovy code in Jenkinsfile:
#NonCPS
def test() {
println "Start"
sleep(10)
println "Stop"
}
Thing is, that after sleeping for 10s code never gets to println "Stop".
It just seems, that sleep returns after 10 seconds and runs next pipeline steps.
Output is just:
[Pipieline] echo
Start
[Pipeline] sleep
Sleeping for 10 sec
[Pipeline] }
... next pipeline steps
Did anyone had same problem?

When you call sleep(10) inside your Groovy script, Workflow CPS plugin executes SleepStep instead of DefaultGroovyStaticMethods.sleep(Object self, long time). The problem in your case is caused #NonCPS function (thanks mkobit for a suggestion!). Consider following example:
node {
stage("Test") {
test()
}
}
#NonCPS
def test() {
echo "Start"
sleep(5)
echo "Stop"
}
Output:
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/test-pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] echo
Start
[Pipeline] sleep
Sleeping for 5 sec
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
There is no Stop echoed from the pipeline run. Now, if we only remove #NonCPS annotation from the function we call in the pipeline:
node {
stage("Test") {
test()
}
}
def test() {
echo "Start"
sleep(5)
echo "Stop"
}
Things get change and Stop gets echoed as expected:
Started by user admin
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/test-pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] echo
Start
[Pipeline] sleep
Sleeping for 5 sec
[Pipeline] echo
Stop
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
For more information about usage of #NonCPS please read following Best Practices For Pipeline Code article.

This issue is well documented in https://wiki.jenkins.io/display/JENKINS/Pipeline+CPS+method+mismatches
Use of Pipeline steps from #NonCPS
Sometimes users will apply the #NonCPS annotation to a method definition, which bypasses the CPS transform inside that method. This can be done to work around limitations in Groovy language coverage (since the body of the method will execute using the native Groovy semantics), or to get better performance (the interpreter imposes a substantial overhead). However such methods must not call CPS-transformed code such as Pipeline steps.
As suggested by Szymon removing the #NonCPS tag will indeed solve the issue,
however if you can't remove the #NonCPS tag because it is needed (to solve serialization issues for example) you can overcome this by using the Thread.sleep(long millis) Java method instead.
Notice - an administrator will need to approve this signature if running in sandbox.

Related

How to know elapsed time of a Jenkins Job

I have a Jenkins job in which at the end of the job (maybe around post actions or in the last stage of that job I want to find how much time has been elapsed since the job started.
How can find that? Is there any easy straightforward way of knowing it ?
Sample pipeline script
pipeline {
agent any
stages {
stage('Hello') {
steps {
echo 'Hello World'
sleep 10
}
}
}
post {
always {
println "${currentBuild.durationString}"
}
}
}
Output:
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Hello)
[Pipeline] echo
Hello World
[Pipeline] sleep
Sleeping for 10 sec
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] echo
14 sec and counting
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
if you don't like the and counting part you can always do
"${currentBuild.durationString}".replaceAll(' and counting', "")
Install Timestamper plugin.
The Timestamper plugin adds timestamps to the console output of Jenkins jobs. For example:
21:51:15 Started by user anonymous
21:51:15 Building on master
21:51:17 Finished: SUCCESS
/timestamps/ By default, display the elapsed time in seconds with three places after the decimal point.
/timestamps/?time=HH:mm:ss&appendLog Display the system clock time and append the line from the log.
/timestamps/?elapsed=HH:mm:ss.S&appendLog Display the elapsed time and append the line from the log.
/timestamps/?time=HH:mm:ss&elapsed=HH:mm:ss.S Display both the system clock time and the elapsed time.
/timestamps/?currentTime&time=HH:mm:ss Display the current time on the Jenkins controller.
Apart from all the answers above there is a very nice way of doing it:
import hudson.Util
.
.
.
String time = Util.getTimeSpanString(System.currentTimeMillis() - currentBuild.startTimeInMillis)
If you display time it will come in this format:
2 hr 15 min 47 sec
which is pretty neat and does not require any extra variable to be setup initially, also less editing needed (in my case I wanted in this exact way).

Can I assign a node dynamically within an iterative loop in Jenkins?

I am building a Jenkins pipeline for a scenario where I'll have to use specific Jenkins agents inside remote data centers to deploy my code to those data centers. This is due to firewall restrictions on some ports, specifically WinRM is blocked between some of our global data centers.
Our deploys are written so that a single deploy stage can deploy to any number of environments, specified by the user's passed-in parameters. The stage loops through the environments and calls a generic deploy script for each one.
I know how to specify an agent by its label or other closure in a stage's definition:
stage ('a stage') {
agent { label 'some agent label' }
steps { ...
but in this case, i am solving for deploying to multiple environments in one deploy stage, each of which will require its own agent.
I can, of course, specify a unique stage for each env, and use a when clause to run it when appropriate, but that's messy.
What I'd like to do is tell the pipeline what agent(s) to use for the deploy stage inside inside the deploy stage, and be able to use multiple agents within that single stage, determined dynamically based on the parameters of the run.
I'd originally found this answer on SO, which gave me the idea of acquiring a node inside the stage, and not with the agent declaration. It doesn't show the acquisition inside a script block, but I'd initially read it that way, and that gave me the idea to try acquiring the node inside a script. And once you're there, it's a small leap to try doing it in a loop.
To prove it, I print some local environment variables from the agent to prove that we're switching agents, inside the stage, inside the loop. I'm also passing a file to each agent to prove that I can pass the files through the firewall.
Note that to even connect to the agent behind the firewall, we had to open the port that is defined in the Jenkins global security config, inbound to the agent from the controller (aka master), and https (443) outbound to the controller. The inbound port is configured to be static.
pipeline {
agent none
stages {
stage ('init') {
agent any
steps {
writeFile file: 'tester', text: 'i am a test file'
stash includes: 'tester', name: 'tester'
}
}
stage ('get agents') {
steps {
script {
['Agent1', 'Agent2'].each { agent ->
node (agent) {
echo "I am agent `${NODE_NAME}`\nMy labels are `${NODE_LABELS}`"
unstash 'tester'
echo "the content of the file is `${readFile 'tester'}`"
}
}
}
}
}
}
}
Which outputs:
Started by user Maximilian Cascone Admin
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] stage
[Pipeline] { (init)
[Pipeline] node
Running on Agent1 in /mnt/data/jenkins/workspace/Sandbox/mcascone/dynamic-agents
[Pipeline] {
[Pipeline] writeFile
[Pipeline] stash
Stashed 1 file(s)
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (get agents)
[Pipeline] script
[Pipeline] {
[Pipeline] node
Running on Agent2 in D:\workspace\Sandbox\mcascone\dynamic-agents
[Pipeline] {
[Pipeline] echo
I am agent `Agent2`
My labels are `Cider Redgate Windows Worker02 ant chef npm relativity wix`
[Pipeline] unstash
[Pipeline] readFile
[Pipeline] echo
the content of the file is `i am a test file`
[Pipeline] }
[Pipeline] // node
[Pipeline] node
Running on Agent1 in C:\jenkins\workspace\Sandbox\mcascone\dynamic-agents
[Pipeline] {
[Pipeline] echo
I am agent `Agent1`
My labels are `Itar Agent1`
[Pipeline] unstash
[Pipeline] readFile
[Pipeline] echo
the content of the file is `i am a test file`
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
Finished: SUCCESS
So I've proven that I can get multiple agents dynamically within a single stage. Next step would be elevating this into a shared step, so it can be called without the script block and make the pipeline nice and neat. But as a POC, this is a great achievement. I don't believe I've seen this elsewhere.
The answer might not fit all needs you have, but dynamic generation of stage and on that basis you can assign / execute the generate stages with following way.
def agents = ['master', 'agent1', 'agent2']
def generateStage(nodeLabel) {
return {
stage("Runs on ${nodeLabel}") {
node(nodeLabel) {
echo "Running on ${nodeLabel}"
}
}
}
}
def parallelStagesMap = agents.collectEntries {
["${it}" : generateStage(it)]
}
pipeline {
agent none
stages {
stage('non-parallel stage') {
steps {
echo 'This stage will be executed first.'
}
}
stage('parallel stage') {
steps {
script {
parallel parallelStagesMap
}
}
}
}
}
Furthermore you can use collectEntries, out of the function box parallelStagesMap, in this way you can use each collect entry for different stage and can dynamically assign nodes in the stage, and in function generateStage you need to do modification as per your requirement.
If you wanted to execute these stages sequentially, then remove parallel from script.generateStage contains return which is imporant without that, pipeline will not work as expected.

Jenkins pipeline with parallel stages causes "process apparently never started" error

I'm trying to set up a Jenkins pipeline (using the declarative syntax) that runs unit and feature tests on two separate, on-demand AWS EC2 instances. The pipeline works perfectly when run on a single instance and without the parallel stages. As soon as I switch to parallel stages, any shell script fails with this cryptic message:
process apparently never started in
/home/admin/workspace/GSWebRuby_Test#tmp/durable-b0d8c4b4 (running
Jenkins temporarily with
-Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true
might make the problem clearer)
I've googled extensively and came across several bug reports of the Durable Task plugin that appears to be responsible for this message. I'm using the latest version of the plugin v. 1.33 and none of the problems seem to apply to my case, e.g. failures on unusual architectures or when running Docker containers. I've also down- and re-upgaded the plugin (toggling between versions 1.30 and 1.33). Also, to re-iterate, sh commands work without issue when I don't use the parallel stages.
I've created a simplified pipeline to debug the problem. Note that the shell commands are also simple, e.g. "env | sort", or "pwd".
pipeline {
agent none
environment {
DB_USER = credentials('db-user')
DB_PASS = credentials('db-pass')
}
stages {
stage('Setup'){
failFast false
parallel {
stage('foo') {
agent {
label 'jenkins-slave-ondemand'
}
steps {
echo 'In stage foo'
sh 'env|sort'
}
}
stage('bar') {
agent {
label 'jenkins-slave-ondemand'
}
steps {
echo 'In stage bar'
sh 'pwd'
}
}
}
}
}
}
This is the console output:
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] withCredentials
Masking supported pattern matches of $DB_PASS or $DB_USER
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Setup)
[Pipeline] parallel
[Pipeline] { (Branch: foo)
[Pipeline] { (Branch: bar)
[Pipeline] stage
[Pipeline] { (foo)
[Pipeline] stage
[Pipeline] { (bar)
[Pipeline] node
[Pipeline] node
Still waiting to schedule task
All nodes of label ‘jenkins-slave-ondemand’ are offline
Still waiting to schedule task
All nodes of label ‘jenkins-slave-ondemand’ are offline
Running on EC2 (Jenkins AWS EC2) - Jenkins slave (i-0982299c572100c71) in /home/admin/workspace/GSWebRuby_Test
[Pipeline] {
[Pipeline] echo
In stage foo
[Pipeline] sh
Running on EC2 (Jenkins AWS EC2) - Jenkins slave (i-092ecac8e6c257270) in /home/admin/workspace/GSWebRuby_Test
[Pipeline] {
[Pipeline] echo
In stage bar
[Pipeline] sh
process apparently never started in /home/admin/workspace/GSWebRuby_Test#tmp/durable-b0d8c4b4
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch foo
process apparently never started in /home/admin/workspace/GSWebRuby_Test#tmp/durable-b6cfcff9
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch bar
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] End of Pipeline
ERROR: script returned exit code -2
Finished: FAILURE
Am I doing something wrong in the way I've set up the pipeline? Any pointers would be greatly appreciated.
Edit:
After setting this JENKINS_JAVA_OPTIONS org.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true, I see this additional output:
In stage bar
[Pipeline] sh
nohup: failed to run command 'sh': No such file or directory
process apparently never started in /home/admin/workspace/GSWebRuby_Test#tmp/durable-099a2e56

#NonCPS stops after first build step

In a #NonCPS annotated function, only code up to the very first jenkins build step is executed. Does anyone have the same problem? Am I missing something? I am using Jenkins LTS... just sayin' (2.73.2).
This is my code:
#NonCPS
def hello() {
println 'Output "hello":'
sh 'echo Hello'
println 'Output "World":'
sh 'echo World'
}
node {
stage('Test') {
hello()
}
}
I would expect this code to run properly, but the output is the following:
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/Sandbox/pipeline-test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] echo
Output "hello":
[Pipeline] sh
[pipeline-test] Running shell script
+ echo Hello
Hello
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
You cannot run build steps inside #NonCPS methods. Pipeline scripts are considered "serializable", allowing them to be durable across system failures etc. Only a subset of the capabilities of groovy used by pipeline scripts is serializable - for anything that is not, you use #NonCPS to execute it.
Essentially, your #NonCPS method needs to do its business and return data back to the "safe", serialized execution stack.
In your particular example code I see no reason why hello() has to be #NonCPS at all - I can only assume your real function is doing something more complex.
(Edit)
Having just looked at your question history and the original script; I don't know if this is still the case with the latest versions but when I was writing our scripts ~6 months ago, each { thing -> iteration was not serializable.

Jenkins pipeline - How to iterate through a list

I'm required to read values from a file in my pipeline. I'm using split() which puts them into an Array. I need to put them into an Arraylist so I'm using Arrays.asList(). The problem I'm having is I'm unable to use the size() or length() methods so I cannot make a for loop such as
for (ii = 0; ii < var.length; ii++)
or
for (ii = 0; ii < var.size; ii++)
because I get error: unclassified field java.util.Arrays$ArrayList length
So I tried to use a for each loop, but when I take some action (like ls command for example) in my finally block it only iterates 1 time. But if I just run the command 'echo' it iterates for each element like it's supposed to. Any advice on how to modify my code to get it to iterate for each element in the list when using any command?
Works correctly....
node{
wrap([$class: 'ConfigFileBuildWrapper', managedFiles: [[fileId: 'dest_hosts.txt', targetLocation: '', variable: 'DEST_HOST']]]) {
HOST = Arrays.asList(readFile(env.DEST_HOST).split("\\r?\\n"))
deploy(HOST)
}
}
#NonCPS
def deploy(host){
for (String target : host){
try {
echo target
}
finally {
echo target
}
}
}
OUTPUT (iterates for each element):
[Pipeline] node
Running on <obfuscated>
[Pipeline] {
[Pipeline] wrap
provisoning config files...
copy managed file [<obfuscated>] to file:/var/lib/jenkins/<obfuscated>
[Pipeline] {
[Pipeline] readFile
[Pipeline] echo
www.testhost.com
[Pipeline] echo
www.testhost.com
[Pipeline] echo
www.testhost2.com
[Pipeline] echo
www.testhost2.com
[Pipeline] }
Deleting 1 temporary files
[Pipeline] // wrap
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
But if I take some action such as 'ls -l' it only iterates 1 time
node{
wrap([$class: 'ConfigFileBuildWrapper', managedFiles: [[fileId: 'dest_hosts.txt', targetLocation: '', variable: 'DEST_HOST']]]) {
HOST = Arrays.asList(readFile(env.DEST_HOST).split("\\r?\\n"))
deploy(HOST)
}
}
#NonCPS
def deploy(host){
for (String target : host){
try {
echo target
}
finally {
sh 'ls -l'
}
}
}
OUTPUT (only iterates 1 time):
[Pipeline] node
Running on <obfuscated>
[Pipeline] {
[Pipeline] wrap
provisoning config files...
copy managed file [<obfuscated>] to file:/var/lib/jenkins/<obfuscated>
[Pipeline] {
[Pipeline] readFile
[Pipeline] echo
www.testhost.com
[Pipeline] sh
[sandbox%2Fpipeline-test-new1] Running shell script
+ ls -l
total 8
-rw-r--r-- 1 jenkins jenkins 10 Jun 17 16:07 someFile
[Pipeline] }
Deleting 1 temporary files
[Pipeline] // wrap
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
ArrayList (and generally Lists) don't have a length or size field, they have a size() method. So use that in the for:
for (ii = 0; ii < var.size(); ii++)
I prefer this solution:
node('master') {
stage('Test 1: loop of echo statements') {
echo_all(abcs)
}
}
#NonCPS // has to be NonCPS or the build breaks on the call to .each
def echo_all(list) {
list.each { item ->
echo "Hello ${item}"
}
}
If you use a declarative pipeline, you have to wrap the call in a script statement:
stage('master') {
steps {
script {
echo_all(abcs);
}
}
As per this tutorial: https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md#serializing-local-variables
...a method marked with the annotation #NonCPS... will be treated as “native” by the Pipeline engine, and its local variables never saved. However it may not make any calls to Pipeline steps
In your case, the sh call is a pipeline step operation, which you apparently can't perform from within a #NonCPS annotated method.
Regarding turning an array into a List, then since we're in Groovy land you could just use the .toList() method on the array.
I cannot tell you PRECISELY why, as I've not figured out how to find useful information about Jenkins without spending hours googling, but I can tell you this:
For a moment I thought you can make it run fine by adding 'echo line' AFTER the sh 'echo $line', but that turned out to be caused by Jenkins running a PREVIOUS version of the script...
I tried all sorts of things and none of them worked, then I found this:
Why an each loop in a Jenkinsfile stops at first iteration
Its a known bug in Jenkins pipeline!
(the known bug is JENKINS-26481, which says "At least some closures are executed only once inside of Groovy CPS DSL scripts managed by the workflow plugin")

Resources