following code is throwing below error.
if(!SkipLanguageComponentTests){
^
WorkflowScript: : Groovy compilation error(s) in script. Error(s): "Ambiguous expression could be either a parameterless closure expression or an isolated open code block;
solution: Add an explicit closure parameter list,
script {
2 errors
`
def SkipLanguageComponentTests = false;
pipeline {
parameters {
booleanParam(name: 'SkipLanguageComponentTests', defaultValue: false, description: 'XYZ')
}
stages {
stage('Checkout Source') {
steps {
checkout scm
}
}
stage("Component & Language Tests"){
steps{
parallel (
"componentTestsTask":{
//component test start
dir("docker"){
sh script: "docker-compose -f blah blah\""
}
// some xyz step here
//component test ends here
},
"integrationTestTasks":{
// language test script starts
if(!SkipLanguageComponentTests){
//run lang test and publish report
} else {
echo "Skip Language Component Tests"
}
// language test script ends
}
)
}
}
}
`
I have tried as per the documentation https://www.jenkins.io/blog/2017/09/25/declarative-1/
I have tried this based on the answer mentioned in : Running stages in parallel with Jenkins workflow / pipeline
stage("Parallel") { steps { parallel ( "firstTask" : { //do some stuff }, "secondTask" : { // Do some other stuff in parallel } ) } }
Can someone help me to resolve this ?
Ok, here is the working version of your pipeline - with proper IF inside:
parameters {
booleanParam(name: 'SkipLanguageComponentTests', defaultValue: false, description: '')
}
agent { label 'master' }
stages {
stage("Component & Language Tests") {
parallel {
stage("componentTestsTask") {
steps {
//component test start
echo "docker"
// some xyz step here
//component test ends here
}
}
stage("integrationTestTasks") {
steps {
script {
// language test script starts
if (!params.SkipLanguageComponentTests) {
echo "not skipped"
//run lang test and publish report
} else {
echo "Skip Language Component Tests"
}
}
}
// language test script ends
}
}
}
}
}
This pipeline is not optimal, use below information to improve it.
Notes:
you are using declarative pipeline so I think it is better to stay with parallel section expressed in declarative way
help is here: Jenkins doc about parallel
there is scripted pipeline as well Jenkins doc about scripted pipeline
as I stated in original answer, you have to use params to refer to input parameter properly
if you are using code you have to enclose it in script section - this is like putting scripted pipeline inside declarative one
IF statement can also be done declaratively: Jenkins doc about WHEN
I recommend not to mix these 2 styles: so if you are using both for some good reason they should be separated from one another as much as possible for example by using function or library inside declarative pipeline - the goal is to have pipeline as clear and readable as possible
You are using input variable, so try to refer as it should be done for input in Jenkins:
if(!params.SkipLanguageComponentTests)
Related
I have 2 questions regarding jenkins
1. What is the difference between the 2 methods?
Method 1
pipeline {
agent any
stages {
stage('Example') {
steps {
echo 'Hello World'
}
}
}
Method 2
node ("jenkins-nodes") {
stage("git clone"){
echo 'Hello World' }
}
As I understand in the first method I can add Post section that will be running regardless of the result of the job. I wish to add the same post section for the second method, but it is not working. Any ideas?
What is the difference between the 2 methods?
As Noam Helmer wrote, pipeline{} is declarative syntax and and node{} is scripted syntax.
In general I recommend to always use pipeline{} as it makes common tasks easier to write and visualization using Blue Ocean plugin works best with declarative pipeline.
When declarative pipeline becomes too unflexible, you can insert scripted blocks using the script{} step in a declarative pipeline:
pipeline {
agent any
stages {
stage('Example') {
steps {
script {
echo 'Hello World'
}
}
}
}
}
A cleaner approach is to define a function, which is scripted by definition and use that as a custom step in declarative pipeline. Note this works even without script{}!
pipeline {
agent any
stages {
stage('Example') {
steps {
myStep 42
}
}
}
}
void myStep( def x ) {
echo "Hello World $x" // prints "Hello World 42"
}
In complex pipelines that use lots of custom code, I usually have one function per stage. This keeps the pipeline{} clean and makes it easy to see the overall structure of the pipeline, without script{} clutter all over the place.
As I understand in the first method I can add Post section that will
be running regardless of the result of the job. I wish to add the same
post section for the second method, but it is not working. Any ideas?
post{} is only available in declarative pipeline, but not within a scripted pipeline or a scripted section of a declarative pipeline. You can use try{} catch{} finally{} instead. catch{} runs only when an error occurs, finally{} always runs. You can use both or either of catch{} and finally{}.
node ("jenkins-nodes") {
stage("git clone"){
echo 'Hello World'
try {
// some steps that may fail
}
catch( Exception e ) {
echo "An error happened: $e"
// Rethrow exception, to let the build fail.
// If you remove "throw", the error would be ignored by Jenkins.
throw
}
finally {
echo "Cleaning up some stuff"
}
}
}
When i load another groovy file in Jenkinsfile it show me following error.
"Required context class hudson.FilePath is missing
Perhaps you forgot to surround the code with a step that provides this, such as: node"
I made a groovy file which contains a function and i want to call it in my Declarative Jenkinsfile. but it shows an error.
My Jenkinsfile--->
def myfun = load 'testfun.groovy'
pipeline{
agent any
environment{
REPO_PATH='/home/manish/Desktop'
APP_NAME='test'
}
stages{
stage('calling function'){
steps{
script{
myfun('${REPO_PATH}','${APP_NAME}')
}
}
}
}
}
Result--
org.jenkinsci.plugins.workflow.steps.MissingContextVariableException: Required context class hudson.FilePath is missing
Perhaps you forgot to surround the code with a step that provides this, such as: node
Suggest me what is the right way to do it.
You either need to use a scripted pipeline and put "load" instruction inside the node section (see this question) or if you are already using a declarative pipeline (which seems to be the case), you can include it in "environment" section:
environment {
REPO_PATH='/home/manish/Desktop'
APP_NAME='test'
MY_FUN = load 'testfun.groovy'
}
We have to wrap with node {}, so that jenkins executors will execute on node, Incase if we would like to execute on any specific agent node, we can mention like node('agent name'){}
example here :
node {
def myfun = load 'testfun.groovy'
pipeline{
agent any
environment{
REPO_PATH='/home/manish/Desktop'
APP_NAME='test'
}
stages{
stage('calling function'){
steps{
script{
myfun('${REPO_PATH}','${APP_NAME}')
}
}
}
}
}
}
Loading the function in an initial script block inside the pipeline worked for me. Something like below:
def myfun
pipeline {
agent any
environment {
REPO_PATH='/home/manish/Desktop'
APP_NAME='test'
}
stages {
stage('load function') {
steps {
script {
myfun = load 'testfun.groovy'
}
}
}
stage('calling function') {
steps {
script {
myfun("${REPO_PATH}","${APP_NAME}")
}
}
}
}
}
I got this error message when I was calling a sh script that does not not exist in the repository / file system. Please look in the stack trace the following line:
at WorkflowScript.run(WorkflowScript:135)
The 135 marks the line in Jenkinsfile, on which the missing script or error is happening.
Another possibility is that due to earlier/underlying errors, the context has been removed for example by multiple executor machines. This happens if you are missing the node (e.g. script) -block, but especially at the post always -block. You can use the if -check also in other places. After this you get another error, that was causing this error message.
post {
always {
script {
//skip the step if context is missing
if (getContext(hudson.FilePath)) {
echo "It works"
}
}
}
}
See https://docs.cloudbees.com/docs/cloudbees-ci-kb/latest/troubleshooting-guides/how-to-troubleshoot-hudson-filepath-is-missing-in-pipeline-run
What is modern best practice for multi-configuration builds (with Jenkins)?
I want to support multiple branches and multiple configurations.
For example for each version V1, V2 of the software I want builds targeting
platforms P1 and P2.
We have managed to set up multi-branch declarative pipelines. Each build has its own docker so its easy to support multiple platforms.
pipeline {
agent none
stages {
stage('Build, test and deploy for P1) {
agent {
dockerfile {
filename 'src/main/docker/Jenkins-P1.Dockerfile'
}
}
steps {
sh buildit...
}
}
stage('Build, test and deploy for P2) {
agent {
dockerfile {
filename 'src/main/docker/Jenkins-P2.Dockerfile'
}
}
steps {
sh buildit...
}
}
}
}
This gives one job covering multiple platforms but there is no separate red/blue status for each platform.
There is good argument that this does not matter as you should not release unless the build works for all platforms.
However, I would like a separate status indicator for each configuration. This suggests I should use a multi-configuration build which triggers a parameterised build for each configuration as below (and the linked question):
pipeline {
parameters {
choice(name: 'Platform',choices: ['P1', 'P2'], description: 'Target OS platform', )
}
agent {
filename someMagicToGetDockerfilePathFromPlatform()
}
stages {
stage('Build, test and deploy for P1) {
steps {
sh buildit...
}
}
}
}
There are several problems with this:
A declarative pipeline has more constraints over how it is scripted
Multi-configuration builds cannot trigger declarative pipelines (even with the parameterized triggers plugin I get "project is not buildable").
This also begs the question what use are parameters in declarative pipelines?
Is there a strategy that gives the best of both worlds i.e:
pipeline as code
separate status indicators
limited repetition?
This is a partial answer. I think others with better experience will be able to improve on it.
This is currently untested. I may be barking up the wrong tree.
Please comment or add a better answer.
Do not use pipeline parameters except where you need user input
Use a hybrid of a scripted and declarative pipeline
(see also https://stackoverflow.com/a/46675227/1569204)
Have a function which declares a pipeline based on parameters:
(see also https://jenkins.io/doc/book/pipeline/shared-libraries/)
Use nodes to create visible indicators in the pipeline (at least in blue ocean)
So something like the following:
def build(string platform) {
switch(platform) {
case P1:
dockerFile = 'foo'
indicator = 'build for foo'
break
case P2:
dockerFile = 'bar'
indicator = 'build for bar'
break
}
pipeline {
agent {
dockerfile {
filename "$dockerFile"
}
node {
label "$indicator"
}
}
stages {
steps {
echo "build it"
}
}
}
}
The relevant code could be moved to a shared library (even if you don't actually need to share it).
I think the cleanest approach is to have this all in a pipeline similar to the first one you presented, the only modification I would see here is making those parallel, so you would actually try and build/test for both platforms.
To reuse the previous stage's workspace you could do: reuseNode true
Something similar to this flow, that would have parallel build for platforms
pipeline {
agent 'docker'
stages {
stage('Common pre') { ... }
stage('Build all platforms') {
parallel {
stage('Build, test and deploy for P1') {
agent {
dockerfile {
filename 'src/main/docker/Jenkins-P1.Dockerfile'
reuseNode true
}
}
steps {
sh buildit...
}
}
stage('Build, test and deploy for P2') {
agent {
dockerfile {
filename 'src/main/docker/Jenkins-P2.Dockerfile'
reuseNode true
}
}
steps {
sh buildit...
}
}
}
}
stage('Common post parallel') { ... }
}
}
I would like to setup parallel stages as described in the image
In this instance, the setup is pretty heavy so has to be done once before the parallel group starts of G1, G2 and G3. At the same time, the stage : Initial Checks has 2 items that I would like to run in parallel.
Is this possible in the Declarative Pipeline or do I have to resort to a script?
I couldnt see in the documentation the ability for this to work
Stages {
stage ('P1') {
}
stage ('P2 Setup') {}
stage ('P2') {
//Here it can contain either Steps or Parallel. I can only do
parallel {
stage ('g1') {} //Parallel tests
stage ('g2') {}
stage ('g3') {}
}
}
stage ('P2 Cleanup') {}
}
Have you encountered similar situations and what have your solutions been like?
Ofcourse 1 solution is to make Setup and Cleanup as part of every group, but like I said, its pretty intensive and I would only take it on if what the diagram indicates isn't possible.
Solution 2
stage ('p2') {
script {
//Some scripting here to get the result?
}
}
Pipeline
This is not supported by a DSL or declarative pipeline yet. You are essentially looking for nested parallel stages as mentioned here
Issue is still open with Jenkins community which you can watch here
In your given case, you can launch stage P1 and stage setup in parallel. However, it is important to start P1 as a background process because from your graph it appears that P1 is a time-intensive operation. Once group stage completes, you can collect the status of P1 and proceed to S2.
stages{
stage('Build'){
steps{
echo "Build"
}
}
stage('Init'){
parallel{
stage('P1'){steps{ echo "launch p1 in background"}}
stage('setup'){steps{echo "setup"}}
}
}
stage('Group'){
parallel{
stage('P1'){steps{echo "p1"}}
stage('P2'){steps{echo "p2"}}
stage('P3'){steps{echo "p3"}}
}
}
stage('Cleanup'){
steps{
echo "cleanup"
}
}
stage('Check P1 status'){
steps{
echo "Check"
}
}
stage('S2'){
steps{
echo "S2"
}
}
}
I think you are looking for this
node {
stage("P1"){}
stage("p2") {}
stage("p3") {
parallel (
"firstTask" : {
},
"secondTask" : {
}
)
}
}
Please note: the question is based on the old, now called "scripted" pipeline format. When using "declarative pipelines", parallel blocks can be nested inside of stage blocks (see Parallel stages with Declarative Pipeline 1.2).
I'm wondering how parallel steps are supposed to work with Jenkins workflow/pipeline plugin, esp. how to mix them with build stages. I know about the general pattern:
parallel(firstTask: {
// Do some stuff
}, secondTask: {
// Do some other stuff in parallel
})
However, I'd like to run couple of stages in parallel (on the same node, which has multiple executors), so I tried to add stages like this:
stage 'A'
// Do some preparation stuff
parallel(firstTask: {
stage 'B1'
// Do some stuff
}, secondTask: {
stage 'B2'
// Do some other stuff in parallel
})
stage 'C'
// Finalizing stuff
This does not work as expected. The "do stuff" tasks are executed in parallel, but the parallel stages end immediately and do not incorporate the stuff they should contain. As a consequence, the Stage View does not show the correct result and also does not link the logs.
Can I build different stages in parallel, or is the "parallel" step only meant to be used within a single stage?
You may not place the deprecated non-block-scoped stage (as in the original question) inside parallel.
As of JENKINS-26107, stage takes a block argument. You may put parallel inside stage or stage inside parallel or stage inside stage etc. However visualizations of the build are not guaranteed to support all nestings; in particular
The built-in Pipeline Steps (a “tree table” listing every step run by the build) shows arbitrary stage nesting.
The Pipeline Stage View plugin will currently only display a linear list of stages, in the order they started, regardless of nesting structure.
Blue Ocean will display top-level stages, plus parallel branches inside a top-level stage, but currently no more.
JENKINS-27394, if implemented, would display arbitrarily nested stages.
that syntax is now deprecated, you will get this error:
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 14: Expected a stage # line 14, column 9.
parallel firstTask: {
^
WorkflowScript: 14: Stage does not have a name # line 14, column 9.
parallel secondTask: {
^
2 errors
You should do something like:
stage("Parallel") {
steps {
parallel (
"firstTask" : {
//do some stuff
},
"secondTask" : {
// Do some other stuff in parallel
}
)
}
}
Just to add the use of node here, to distribute jobs across multiple build servers/ VMs:
pipeline {
stages {
stage("Work 1"){
steps{
parallel ( "Build common Library":
{
node('<Label>'){
/// your stuff
}
},
"Build Utilities" : {
node('<Label>'){
/// your stuff
}
}
)
}
}
All VMs should be labelled as to use as a pool.
I have just tested the following pipeline and it works
parallel firstBranch: {
stage ('Starting Test')
{
build job: 'test1', parameters: [string(name: 'Environment', value: "$env.Environment")]
}
}, secondBranch: {
stage ('Starting Test2')
{
build job: 'test2', parameters: [string(name: 'Environment', value: "$env.Environment")]
}
}
This Job named 'trigger-test' accepts one parameter named 'Environment'
Job 'test1' and 'test2' are simple jobs:
Example for 'test1'
One parameter named 'Environment'
Pipeline : echo "$env.Environment-TEST1"
On execution, I am able to see both stages running in the same time
I think this has been officially implemented now:
https://jenkins.io/blog/2017/09/25/declarative-1/
As #Quartz mentioned, you can do something like
stage('Tests') {
parallel(
'Unit Tests': {
container('node') {
sh("npm test --cat=unit")
}
},
'API Tests': {
container('node') {
sh("npm test --cat=acceptance")
}
}
)
}