I have some code that needs running (build, test, and packages in actuality but for example just running tox) on different OSes. Currently my Jenkinsfile looks like thus:
pipeline {
// Where to run stuff.
agent {
node {
label 'CentOS7'
customWorkspace '/home/build/jenkins/workspace/pipelines/ook'
}
}
// What to run goes here.
stages {
stage('Tox') {
steps {
sh 'tox -v --recreate'
}
}
}
// Clean up after ourselves.
post {
failure {
mail subject: "\u2639 ${env.JOB_NAME} (${env.BUILD_NUMBER}) has failed",
body: """Build ${env.BUILD_URL} is failing!
Somebody should do something about that\u2026""",
to: "devs#example.com",
replyTo: "devs#example.com",
from: 'jenkins#example.com'
}
}
}
}
The middle bit, I want to run on two different nodes: one for OS 1 and one for OS 2.
How do I do that?
Sure, you would want to label your slave nodes somehow. I didn't look up what tox is, but maybe like 'os_linux' and 'os_mac', and then you can use the node step in your Jenkinsfile to run some commands in the context of each slave. So your Tox stage might look like:
stage('Tox') {
steps {
node('os_linux') {
sh 'tox -v --recreate'
}
node('os_mac') {
sh 'tox -v --recreate'
}
}
}
This will run the tasks in serial, and Jenkinsfile syntax also supports doing those two tox commands in parallel on different nodes. Use the "Pipeline Syntax" link in the left nav of your Jenkins UI (only on pipeline jobs) to play around with node and parallel. Rock on.
Related
I created a simple pipline in Jenkins. The remote root directory of my agent is set to my project root path. But when I test, where I am during the build (e.g. by defining a step like sh 'pwd'), I see, that the directory, my steps are executed from is the $WORKSPACE directory (/path_to_remote_root_directory_of_the_agent/workspace/jenkins_project_title). That means, I cannot just start neither my unit tests like sh 'vendor/bin/phpunit ./test/Unit', nor other tasks, that I usually run from the project root folder.
I'm pretty sure, that I simply configured something incorrectly and that in the normal case scripts like this
pipeline {
agent {
label 'devvm-slave-01'
}
stages {
stage('Prepare') {
steps {
sh 'composer install'
...
}
}
...
stage('Checkstyle') {
steps {
sh 'vendor/bin/phpcs --report=checkstyle --report-file=`pwd`/build/logs/checkstyle.xml --standard=PSR2 --extensions=php --ignore=autoload.php --ignore=vendor/ . || exit 0'
checkstyle pattern: 'build/logs/checkstyle.xml'
}
}
}
}
work as expected without any crude workarounds for paths.
What am I doing wrong and how to get it working correctly?
From the section "agent" of the "Jenkins Handbook"'s chapter "Pipeline Syntax":
Parameters
node
agent { node { label 'labelName' } } behaves the same as agent { label 'labelName' }, but node allows for additional options (such as customWorkspace).
So, the solution is the using of the node and its customWorkspace option:
pipeline {
agent {
node {
label 'devvm-slave-01'
customWorkspace '/path/to/my/project'
}
}
...
}
I have a pipeline with multiple stages, some of them are in parallel. Up until now I had a single code block indicating where the job should run.
pipeline {
triggers { pollSCM '0 0 * * 0' }
agent { dockerfile { label 'jenkins-slave'
filename 'Dockerfile'
}
}
stages{
stage('1'){
steps{ sh "blah" }
} // stage
} // stages
} // pipeline
What I need to do now is run a new stage on a different slave, NOT in docker.
I tried by adding an agent statement for that stage but it seems like it tries to run that stage withing a docker container on the second slave.
stage('test new slave') {
agent { node { label 'e2e-aws' } }
steps {
sh "ifconfig"
} // steps
} // stage
I get the following error message
13:14:23 unknown flag: --workdir
13:14:23 See 'docker exec --help'.
I tried setting the agent to none for the pipeline and using an agent for every step and have run into 2 issues
1. My post actions show an error
2. The stages that have parallel stages also had an error.
I can't find any examples that are similar to what I am doing.
You can use the node block to select a node to run a particular stage.
pipeline {
agent any
stages {
stage('Init') {
steps {
node('master'){
echo "Run inside a MASTER"
}
}
}
}
}
So I spent the whole day trying to figure out how to configure a simple Jenkins Pipeline with multiple Docker images and I am not happy at all.
I need a few stages (prepare, build, test, docs) executed on a couple of different docker containers (currently I just picked three standard Python containers). And it would be nice if those would run in parallel, but I only found this solution, which combines all stages into a single one (and thus creates a not so informative overview in the Blue Ocean UI): Jenkins Pipeline Across Multiple Docker Images
So I ended up with the configuration below, which is ugly as hell (code repetition everywhere), but more or less creates an good looking overview in the classic UI:
A not so informative overview in the Blue Ocean UI
And an acceptable test overview from junit, which combines all the tests from each stage but if any test is failing, the corresponding "version" is shown:
The most annoying thing however is, you cannot see which step has failed. If Python 2.7 fails, everything else is also marked as failed and you don't even see which stage failed.
I tried so many different approaches and I am wondering how this should be done. This should be such a common thing to do with Jenkins, so I guess I have some general misunderstandings in this (for me absolutely new) pipeline/nodes/labels/stages/steps/declarative/scripted/groovy/blueocean stuff...
It should be possible to define a list of docker containers some (maybe customisable stages/steps) for each of them and run them in parallel and having it displayed nicely in Blue Ocean and in Classic UI, shouldn't it?
node {
stage("Python 2.7.14") {
checkout scm
docker.image('python:2.7.14').inside { // just a dummy for now
stage("Prepare") { sh 'python --version' }
stage("Build") { sh 'ls -al' }
}
}
stage("Python 3.5.4") {
checkout scm
docker.image('python:3.5.4').inside {
stage("Prepare") { sh 'python -m venv venv' }
stage("Build") {
sh """
. venv/bin/activate
make install-dev
"""
}
stage('Test') {
sh """
. venv/bin/activate
make test
"""
}
stage('Docs') {
sh """
. venv/bin/activate
make doc-dependencies
cd docs
make html
"""
}
}
}
stage("Python 3.6.4") {
checkout scm
docker.image('python:3.5.4').inside {
stage("Prepare") { sh 'python -m venv venv' }
stage("Build") {
sh """
. venv/bin/activate
make install-dev
"""
}
stage('Test') {
sh """
. venv/bin/activate
make test
"""
}
stage('Docs') {
sh """
. venv/bin/activate
make doc-dependencies
cd docs
make html
"""
}
}
}
}
Update: this is how it looks like in the Blue Ocean UI when a step fails, int this case "Test" in both Python 3.5.4 and 3.6.4 failed but it looks like everything has failed.
Also the Python 2.7.14 and 3.5.4 stages are collapsed and cannot be viewed separately. If I click on one of them, all the steps are shown in green although in this case . venv/bin/activate make test failed:
So this is what I ended up with. There are surely better solutions, but I have to move on. I hope to gather some (better) answers in time, I'll not mark this as "the solution" yet ;)
First, some credits to Stephen Kings slides (the title says "Declarative" but there are some nice examples regarding the scripted Pipeline): (Declarative) Jenkins Pipelines
Here is my gist on GitHub with the following snippet:
def docker_images = ["python:2.7.14", "python:3.5.4", "python:3.6.2"]
def get_stages(docker_image) {
stages = {
docker.image(docker_image).inside {
stage("${docker_image}") {
echo 'Running in ${docker_image}'
}
stage("Stage A") {
switch (docker_image) {
case "python:2.7.14":
sh 'exit 123' // for python 2.7.14 we force an error for fun
break
default:
sh 'sleep 10' // for any other docker image, we sleep 10s
}
sh 'echo this is stage A' // this is executed for all
}
stage("Stage B") {
sh 'sleep 5'
sh 'echo this is stage B'
}
stage("Stage C") {
sh 'sleep 8'
sh 'echo this is stage C'
}
}
}
return stages
}
node('master') {
def stages = [:]
for (int i = 0; i < docker_images.size(); i++) {
def docker_image = docker_images[i]
stages[docker_image] = get_stages(docker_image)
}
parallel stages
}
I tried to make it easy to use:
you add your Docker images in a list at the top and then you define the stages in the get_stages() function
add the common stages and steps
if any Docker image needs special treatment (like python:2.7.14 in my example), you can use a simple switch. This could also be realised with a double map for the special cases ('images'->'stage'='steps') and a fallback double map for defaults, but I'll leave it as an exercise for the reader. (to be honest, I could not figure out the correct, supported Groovy-lang syntax)
This is how it looks like when everything is fine in both the Classic and the Blue Ocean UIs (it's known that the Blue Ocean UI fails to display multiple stages in parallel runs, see JENKINS-38442):
Classic UI
Blue Ocean UI
And this is the output if Stage A in python:2.7.14 fails:
Classic UI
Blue Ocean UI
I have create a pipeline like below and please note that I have the script files namely- "backup_grafana.sh" and "gitPush.sh" in source code repository where the Jenkinsfile is present. But I am unable to execute the script because of the following error:-
/home/jenkins/workspace/grafana-backup#tmp/durable-52495dad/script.sh:
line 1: backup_grafana.sh: not found
Please note that I am running jenkins master on kubernetes in a pod. So copying scripts files as suggested by the error is not possible because the pod may be destroyed and recreated dynamically(in this case with a new pod, my scripts will no longer be available in the jenkins master)
pipeline {
agent {
node {
label 'jenkins-slave-python2.7'
}
}
stages {
stage('Take the grafana backup') {
steps {
sh 'backup_grafana.sh'
}
}
stage('Push to the grafana-backup submodule repository') {
steps {
sh 'gitPush.sh'
}
}
}
}
Can you please suggest how can I run these scripts in Jenkinsfile? I would like to also mention that I want to run these scripts on a python slave that I have already created finely.
If the command 'sh backup_grafana.sh' fails to execute when it actually should have successfully executed, here are two possible solutions.
1) Maybe you need a dot slash in front of those executable commands to tell your shell where they are. if they are not in your $PATH, you need to tell your shell that they can be found in the current directory. here's the fixed Jenkinsfile with four non-whitespace characters added:
pipeline {
agent {
node {
label 'jenkins-slave-python2.7'
}
}
stages {
stage('Take the grafana backup') {
steps {
sh './backup_grafana.sh'
}
}
stage('Push to the grafana-backup submodule repository') {
steps {
sh './gitPush.sh'
}
}
}
}
2) Check whether you have declared your file as a bash or sh script by declaring one of the following as the first line in your script:
#!/bin/bash
or
#!/bin/sh
I'm building jobs that will be using Jenkins piplines extensively. Our nodes are designated per project by their tags, but unlike regular jobs the pipeline build does not seem to have the "Restrict where this project can be run" checkbox. How can I specify on which node the pipeline will run the way I do for regular jobs?
You specify the desired node or tag when you do the node step:
node('specialSlave') {
// Will run on the slave with name or tag specialSlave
}
See https://jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#node-allocate-node for an extended explanation of the arguments to node.
Edit 2019: This answer (and question) was made back in 2017, back then there only was one flavor of Jenkins pipeline, scripted pipeline, since then declarative pipeline has been added. So above answer is true for scripted pipeline, for answers regarding declarative pipeline please see other answers below.
Choosing a node which has the label X in a declarative pipeline with json format:
pipeline {
agent { label 'X' }
...
...
}
You also can apply multiple labels with or (||) or with and (&&) operator.
Running the job on any of the nodes which has label X or label Y:
agent { label 'X || Y' }
Running the job only on nodes which have both label:
agent { label 'X && Y' }
More in the Jenkins Pipeline reference guide.
ps: if you are reading this you probably have just started using Jenkins pipeline and you are not sure if you should use declarative or scripted pipeline. Short answer: it's better to start with declarative. From jenkins.io:
Declarative and Scripted Pipelines are constructed fundamentally
differently. Declarative Pipeline is a more recent feature of Jenkins
Pipeline which:
provides richer syntactical features over Scripted Pipeline syntax, and
is designed to make writing and reading Pipeline code easier.
To be clear, because Pipeline has two Syntax, there are two ways to achieve that.
Declarative
pipeline {
agent none
stages {
stage('Build') {
agent { label 'slave-node' }
steps {
echo 'Building..'
sh '''
'''
}
}
}
post {
success {
echo 'This will run only if successful'
}
}
}
Scripted
node('your-node') {
try {
stage 'Build'
node('build-run-on-this-node') {
sh ""
}
} catch(Exception e) {
throw e
}
}
Agent or Node where we should not execute the jenkins job :
This is the negation of the problem statment i.e. node where not to run
It was most weird solution to me but issue has been already raised to jenkins community
agent { label '!build-agent-name' }
If you need to run entire jenkins pipeline to run on single node, use the following format
pipeline {
agent {
label 'test1'
}
stages {
stage('Build') {
steps {
echo 'Building..'
}
}
stage('Test') {
steps {
echo 'Testing..'
}
}
stage('Deploy') {
steps {
echo 'Deploying....'
}
}
}
}
if you need to execute each stage in different nodes, use the below format,
pipeline {
agent any
stages {
stage('Build') {
steps {
echo 'Building..'
}
}
stage('Test') {
steps {
node("test1"){
echo 'Testing..'
}
}
}
stage('Deploy') {
steps {
echo 'Deploying....'
}
}
}
}