I have written a Jenkins pipeline where the relevant parts looks as follows:
pipeline {
agent {
dockerfile true
}
triggers {
pollSCM('H 1 * * 1-5')
}
options {
buildDiscarder(logRotator(artifactNumToKeepStr: "${NUMBER_OF_ARTIFACTS_TO_KEEP}"))
disableConcurrentBuilds()
timeout(time: 60, unit: 'MINUTES')
timestamps()
}
stages {
stage('Metadata') {
steps {
script {
sh 'java -version'
}
script {
sh './mvnw -v'
}
}
}
stage('Build') {
steps {
script {
sh './mvnw --batch-mode clean install'
}
}
}
stage('Archive artifacts (develop/master)') {
when {
anyOf {
branch 'master'
branch 'develop'
}
}
steps {
script {
sh './package.sh'
}
archive '**/target/*.jar'
archiveArtifacts artifacts: '*.deb'
}
}
}
post {
always {
deleteDir()
}
failure {
sendNotifications currentBuild.result
}
unstable {
sendNotifications currentBuild.result
}
}
}
And my Dockerfile:
FROM alpine
RUN apk add --no-cache dpkg openjdk8
All scripts run fine, except package.sh where I get the following log in the output:
07:47:25 [chx-sync_-sync_master-A2F53LY4I2X54TLDEU2Z2PXI423NI6FODHQDS7CRIKCCNDF5UGOA] Running shell script
07:47:25 + ./package.sh
07:47:25 /home/jenkins/jenkins/workspace/chx-sync_-sync_master-A2F53LY4I2X54TLDEU2Z2PXI423NI6FODHQDS7CRIKCCNDF5UGOA#tmp/durable-dbcb4143/script.sh: line 1: ./package.sh: not found
I can't figure out why all scripts except this one would work. They are all located in the root of the project in Git. Is there some command in my pipeline that would change the working directory, or what is going on here?
EDIT:
I'm guessing the shebang in package.sh might be relevant? It is #!/bin/bash.
I got the answer from this answer on SO. I simply changed the using #!/bin/sh and it is working.
Related
I have two stages on jenkins pipeline, and it is expecting to add steps in execute jmeter stage
Could someone help to resolve this....
I got below error:
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 339: Expected one of "steps", "stages", or "parallel" for stage "Execute Jmeter" # line 339, column 9.
stage('Execute Jmeter') {
Below is the code snippet of jenkins pipeline:
pipeline {
agent {
label 'qatest'
}
tools {
maven 'Maven'
jdk 'JDK8'
}
environment {
VIRTUOSO_URL = 'qa.myapp.com'
}
stages {
stage('BUILD') {
steps {
sh 'mvn clean verify'
}
}
stage('Execute Jmeter') {
post{
always{
dir("scenarioLoadTests/target/jmeter/results/"){
sh 'pwd'
sh 'mv *myapp_UserLoginAndLogout.csv UserLoginAndLogout.csv '
sh 'mv *myapp_myappPortfolioScenario.csv myappPortfolioScenario.csv '
sh 'mv *myapp_myappDesign.csv myappDesign.csv '
perfReport '*.csv'
}
}
}
}
}
}
Shouldn't you use script blocks like:
pipeline {
agent {
label 'qatest'
}
tools {
maven 'Maven'
jdk 'JDK8'
}
environment {
VIRTUOSO_URL = 'qa.myapp.com'
}
stages {
stage('BUILD') {
steps {
script { // <----------------- here
sh 'mvn clean verify'
}
}
}
stage('Execute Jmeter') {
post {
always {
dir('scenarioLoadTests/target/jmeter/results/') {
script { <-------------------------------------- and here
sh 'pwd'
sh 'mv *myapp_UserLoginAndLogout.csv UserLoginAndLogout.csv '
sh 'mv *myapp_myappPortfolioScenario.csv myappPortfolioScenario.csv '
sh 'mv *myapp_myappDesign.csv myappDesign.csv '
perfReport '*.csv'
}
}
}
}
}
}
}
More information:
Pipeline Maven Integration
Running a JMeter Test via Jenkins Pipeline - A Tutorial
I have a series of steps in a stage that I want to run even if the first one fails. I want the stage result to fail and the build to get aborted, but only after all steps have run. For example,
pipeline {
agent any
stages {
stage('Run Test') {
steps {
sh "echo running unit-tests"
sh "echo running linting && false" // failure
sh "echo generating report" // This should still run (It currently doesn't)
publishCoverage adapters: [coberturaAdapter("coverage.xml")] // This should still run (It currently doesn't)
junit 'unit-test.xml' // This should still run (It currently doesn't)
}
}
stage('Deploy') {
steps {
echo "deploying" // This should NOT run
}
}
}
}
The result should be a failed build where the "Run Test" stage failed and the "Deploy" stage did not run. Is this possible?
P.S.
I am NOT asking for the same behavior as in Continue Jenkins pipeline past failed stage. I want to run the steps following the failure, but not any of the stages afterwards. I tried to enclose each of the test steps with catchError (buildResult: 'FAILURE', stageResult: 'FAILURE'), but the "Deploy" stage still runs.
EDIT:
I cannot combine all the steps into one big sh step and capture its return code because some of the steps are not shell commands, but instead jenkins steps like junit and publishCoverage.
A script witha non-zero exit code will always cause a jenkins step to fail. You can use returnStatus as true so that jenkins does not fails the step.
Additionally considering your use case, you could use a post always execution, so that the steps are always carried out.
Please see below reference example:
stage('Run Test') {
steps {
def unit_test_result= sh returnStatus: true, script: 'echo "running unit-tests"'
def lint_result= sh returnStatus: true, script: 'echo "running linting"'
if (unit_test_result!=0 || lint_result!=0 ) {
// If the unit_test_result or lint_result status is not 0 then mark this stage as unstable to continue ahead
// and all later stages will be executed
unstable ('Testing failed')
// You can also mark as failed as below and it will not conintue other stages:
// error ('Testing failed')
}
}
post {
always {
// This block would always be executed inspite of failure
sh "echo generating report"
publishCoverage adapters: [coberturaAdapter("coverage.xml")]
junit 'unit-test.xml'
}
}
}
I found a slightly hacky way to get the behavior I want. The other answers didn't work for me, either because they need all the steps to be sh steps, or they don't stop the deploy stage from running. I used catchError to set the build and stage result. But to prevent the next stage from running, I needed to an explicit call to error if the stage failed.
pipeline {
agent any
stages {
stage('Run Test') {
steps {
script {
// catchError sets the stageResult to FAILED, but does not stop next stages from running
catchError (buildResult: 'FAILURE', stageResult: 'FAILURE') {
sh "echo running unit-tests"
}
catchError (buildResult: 'FAILURE', stageResult: 'FAILURE') {
sh "echo running linting && false" // failure
}
catchError (buildResult: 'FAILURE', stageResult: 'FAILURE') {
sh "echo generating report" // This still runs
}
publishCoverage adapters: [coberturaAdapter("coverage.xml")] // This still runs
junit 'unit-test.xml' // This still runs
if (currentBuild.result == "FAILURE") { // This is needed to stop the next stage from running
error("Stage Failed")
}
}
}
}
stage('Deploy') {
steps {
echo "deploying" // This should NOT run
}
}
}
}
Theoretically you should be able to use sh "<command>||true" It would ignore the error on command and continue. However, Jenkins will not fail as it would ignore the error.
If you don't want Jenkins to ignore the error and want it to stop at the end of the stage, you can do something like: sh "<command>||$error=true" then fail the build based on the $error variable. (sh "$error" might be enough but I am not sure, may require an if statement at the end.) It will be only set to true iff command fails.
Another option is to wrap your build steps in a try-catch block! if there's an exception, i.e. return code of build is not 0 you can catch it, mark the build as unstable and then the rest of the pipeline continues on.
here's an example `
pipeline {
agent {
node {
label 'linux'
}
}
options {
timestamps()
disableConcurrentBuilds()
buildDiscarder(logRotator(numToKeepStr: '3'))
}
tools {
maven 'Maven 3.6.3'
jdk 'jdk11'
}
stages {
stage('CleanWS') {
steps {
cleanWs()
}
}
stage('Build') {
steps {
withMaven(options: [artifactsPublisher(disabled: true)]) {
sh "export NLS_LANG=GERMAN_GERMANY.WE8ISO8859P1 && mvn -f pom.xml clean install -DskipTests -Pregression-test -Dmaven.javadoc.skip=true"
}
}
}
stage('Test') {
steps {
script {
try {
withMaven(options: [artifactsPublisher(disabled: true)]) {
sh "export MAVEN_OPTS=\"-Xmx2048m\" && export NLS_LANG=GERMAN_GERMANY.WE8ISO8859P1 && mvn -B verify -Dmaven.source.skip=true -Dmaven.javadoc.skip=true"
}
} catch (exc) {
currentBuild.result = 'UNSTABLE'
}
}
}
post {
always {
script {
junit "**/surefire-reports/*.xml"
}
}
}
}
stage('Sonar Analyse') {
steps {
script {
withMaven(options: [artifactsPublisher(disabled: true)]) {
withSonarQubeEnv("SonarQube") {
sh "export MAVEN_OPTS=\"-Xmx2048m\" && export NLS_LANG=GERMAN_GERMANY.WE8ISO8859P1 && mvn sonar:sonar"
}
}
}
}
}
stage('Deploy to Nexus') {
steps {
sh "export NLS_LANG=GERMAN_GERMANY.WE8ISO8859P1 && mvn -f pom.xml -B clean deploy -DdeployAtEnd=true -DskipTests"
}
}
}
post {
failure {
script {
emailext(
body: "Please go to ${env.BUILD_URL}/console for more details.",
to: emailextrecipients([developers(), requestor()]),
subject: "Nightly-Build-Pipeline Status is ${currentBuild.result}. ${env.BUILD_URL}"
)
}
}
unstable {
script {
emailext(
body: "Please go to ${env.BUILD_URL}/console for more details.",
to: emailextrecipients([developers(), requestor()]),
subject: "Nightly-Build-Pipeline Build Status is ${currentBuild.result}. ${env.BUILD_URL}"
)
}
}
}
}`
I've a Jenkinsfile, which has a two different stages: Pre-Build and Build. The Pre-Build is executing pylint and uses the warnings-ng-plugin to report that back to Jenkins.
Something like that:
stages {
stage('Pre-build') {
steps {
script {
sh """#!/usr/bin/env bash
pip install .
pylint --exit-zero --output-format=parseable --reports=n myProject > reports/pylint.log
"""
}
}
post {
always {
recordIssues(
enabledForFailure: true,
tool: pyLint(pattern: '**/pylint.log'),
unstableTotalAll: 20,
failedTotalAll: 30,
)
}
failure {
cleanWs()
}
}
}
stage('Build') {
steps {
script {
sh """#!/usr/bin/env bash
set -e
echo 'I AM STAGE TWO AND I SHOULD NOT BE EXECUTED'
"""
}
}
post {
always {
cleanWs()
}
}
}
}
I'm running into a couple of issues here. Currently I'm setting pylint to --exit-zero, as I want the warnings-ng plugin decide if it is good to go or not, based on the report.
Currently this is set to fail at a total of 30 issues. Now, myProject has 45 issues and I want to prevent that the next stage, Build is entered. But currently I can't seem to be able to prevent this behaviour, as it always continuous to the Build stage.
The build is flagged as failure, because of the results determined within recordIssues, but it doesn't abort the job.
I've found a ticket on https://issues.jenkins-ci.org (Ticket), but I can't seem to make sense out of all of this.
I've found a solution to your problem and I think that this is a bug in the pipeline workflow. The warnings-ng correctly sets build status to failed, but next stages are started despite the status in ${currentBuild.currentResult} variable.
You can use when { expression { return currentBuild.currentResult == "SUCCESS" } } to skip later stages, or throwing an error. But I think this should be default behaviour. Your file then should like:
stages {
stage('Pre-build') {
steps {
script {
sh """#!/usr/bin/env bash
pip install .
pylint --exit-zero --output-format=parseable --reports=n myProject > reports/pylint.log
"""
}
}
post {
always {
recordIssues(
enabledForFailure: true,
tool: pyLint(pattern: '**/pylint.log'),
unstableTotalAll: 20,
failedTotalAll: 30,
)
}
}
}
stage('Build') {
when { expression { return currentBuild.currentResult == "SUCCESS" } }
steps {
script {
echo "currentResult: ${currentBuild.currentResult}"
sh """#!/usr/bin/env bash
set -e
echo 'I AM STAGE TWO AND I SHOULD NOT BE EXECUTED'
"""
}
}
}
post {
always {
cleanWs()
}
}
}
I've created an issue in their Jira.
My environment:
Jenkins ver.: 2.222.1
warnings-ng ver.: 8.1
worfklow-api ver.: 2.40
You have used post 2 times which is wrong implementation as post is designed to get executed only once after all stages are done. It should be written after all the stages just before end of pipeline.
To stop or skip the execution of 2nd Build stage, you can create global varaible at the top, capture the output of pylint in that and use if or when condition at start of stage. Something similar to --
pipeline {
def result
stages {
stage('Pre-build') {
steps {
script {
sh """#!/usr/bin/env bash
pip install .
pylint --exit-zero --output-format=parseable --reports=n myProject > reports/pylint.log
"""
}
}
}
}
stage('Pylint result') { // Not sure how recordIssue works. This just an example.
result = recordIssues(
enabledForFailure: true,
tool: pyLint(pattern: '**/pylint.log'),
unstableTotalAll: 20,
failedTotalAll: 30,
)
}
stage('Build') {
if ( result == "pass") {
steps {
script {
sh """#!/usr/bin/env bash
set -e
echo 'I AM STAGE TWO AND I SHOULD NOT BE EXECUTED'
"""
}
}
}
}
}
post { // this should be used after stages
always {
cleanWs()
}
failure {
cleanWs()
}
}
Also, stages are designed in such a way that if they fail, next stage will not be executed so it's a good idea to have the pylint to be executed inside a stage instead of post condition.
Note: The code above is just an example. Please modify it according to your need.
One option that you may consider is to fail the build explicitly with the following code:
post {
always {
recordIssues(
enabledForFailure: true,
tool: pyLint(pattern: '**/pylint.log'),
unstableTotalAll: 20,
failedTotalAll: 30
)
script {
if (currentBuild.currentResult == 'FAILURE') {
error('Ensure that the build fails if the quality gates fail')
}
}
}
}
Here, after you record the issues, you also check if the value of currentBuild.currentResult is FAILURE and in that case you explicitly call the error() function which fails the build correctly.
I have a pipeline with multiple stages, and I want to reuse a docker container between only "n" number of stages, rather than all of them:
pipeline {
agent none
stages {
stage('Install deps') {
agent {
docker { image 'node:10-alpine' }
}
steps {
sh 'npm install'
}
}
stage('Build, test, lint, etc') {
agent {
docker { image 'node:10-alpine' }
}
parallel {
stage('Build') {
agent {
docker { image 'node:10-alpine' }
}
// This fails because it runs in a new container, and the node_modules created during the first installation are gone at this point
// How do I reuse the same container created in the install dep step?
steps {
sh 'npm run build'
}
}
stage('Test') {
agent {
docker { image 'node:10-alpine' }
}
steps {
sh 'npm run test'
}
}
}
}
// Later on, there is a deployment stage which MUST deploy using a specific node,
// which is why "agent: none" is used in the first place
}
}
See reuseNode option for Jenkins Pipeline docker agent:
https://jenkins.io/doc/book/pipeline/syntax/#agent
pipeline {
agent any
stages {
stage('NPM install') {
agent {
docker {
/*
* Reuse the workspace on the agent defined at top-level of
* Pipeline, but run inside a container.
*/
reuseNode true
image 'node:12.16.1'
}
}
environment {
/*
* Change HOME, because default is usually root dir, and
* Jenkins user may not have write permissions in that dir.
*/
HOME = "${WORKSPACE}"
}
steps {
sh 'env | sort'
sh 'npm install'
}
}
}
}
You can use scripted pipelines, where you can put multiple stage steps inside a docker step, e.g.
node {
checkout scm
docker.image('node:10-alpine').inside {
stage('Build') {
sh 'npm run build'
}
stage('Test') {
sh 'npm run test'
}
}
}
(code untested)
For Declarative pipeline, one solution can be to use Dockerfile in the root of the project. For e.g.
Dockerfile
FROM node:10-alpine
// Further Instructions
Jenkinsfile
pipeline{
agent {
dockerfile true
}
stage('Build') {
steps{
sh 'npm run build'
}
}
stage('Test') {
steps{
sh 'npm run test'
}
}
}
I want to use different JDK versions for different stages in Jenkins declarative pipeline. In the first stage I am using Java 8. In the second stage i am using Java 6. How to select multiple JDK version in declarative pipeline in Jenkins?
pipeline {
agent any
tools {
jdk 'jdk_1.8.0_151'
jdk 'jdk_1.6.0_45'
}
stages {
stage('java 8') {
steps {
sh 'java -version'
sh 'javac -version'
}
}
stage('java 6') {
steps {
sh 'java -version'
sh 'javac -version'
}
}
}
}
you can add a tools section for each stage.
pipeline {
agent any
stages {
stage ("first") {
tools {
jdk "jdk-1.8.101"
}
steps {
sh 'java -version'
}
}
stage("second"){
tools {
jdk "jdk-1.8.152"
}
steps{
sh 'java -version'
}
}
}
}
From the Pipeline tools directive:
tools: A section defining tools to auto-install and put on the PATH.
The tool name must be pre-configured in Jenkins under
Manage Jenkins → Global Tool Configuration.
From the pipeline-examples and cloudbess example:
pipeline {
agent any
tools {
jdk 'jdk_1.8.0_151'
}
stages {
stage('jdk 8') {
steps {
sh 'java -version'
sh 'javac -version'
}
}
stage('jdk 6') {
steps {
withEnv(["JAVA_HOME=${tool 'openjdk_1.6.0_45'}", "PATH=${tool 'openjdk_1.6.0_45'}/bin:${env.PATH}"]) {
sh 'java -version'
sh 'javac -version'
}
}
}
stage('global jdk') {
steps {
sh 'java -version'
sh 'javac -version'
}
}
}
}
I would recommend you to use different docker images for each stage if you want to have different JDK versions. You can achieve using the docker hub openjdk images with the correct tag.
https://hub.docker.com/r/library/openjdk/
https://hub.docker.com/r/library/openjdk/tags/
Something like that:
pipeline {
agent none
stages {
stage('openjdk:7-jdk') {
agent {
docker { image 'jdk7_image' }
}
steps {
sh 'java -version'
}
}
stage('java8') {
agent {
docker { image 'openjdk:8-jdk' }
}
steps {
sh 'java -version'
}
}
}
}