I coded a generic pipeline which accepts several parameters in order to deploy releases from a pre-defined GitHub repository to specific nodes. I wanted to host this pipeline on a Jenkinsfile on GitHub, so I configured the job to work with a "Pipeline script from SCM". The fact is - when I try and build the job - the Jenkinsfile gets checked out on every node. Is it possible to checkout and execute the Jenkinsfile only on, say, the master node and run the pipeline as intended?
EDIT: As I stated before, the pipeline works just fine and as intended setting the job to work with a pipeline script. The thing is when I try and change it to be a "Pipeline script from SCM", the Jenkinsfile gets checked out on every agent, which is a problem since I don't have git installed on any agent other than master. I want the Jenkinsfile to be checked out only on master agent and be executed as intended. FYI the pipeline below:
def agents = "$AGENTS".toString()
def agentLabel = "${ println 'Agents: ' + agents; return agents; }"
pipeline {
agent none
stages {
stage('Prep') {
steps {
script {
if (agents == null || agents == "") {
println "Skipping build"
skipBuild = true
}
if (!skipBuild) {
println "Agents set for this build: " + agents
}
}
}
}
stage('Powershell deploy script checkout') {
agent { label 'master' }
when {
expression {
!skipBuild
}
}
steps {
git url: 'https://github.com/owner/repo.git', credentialsId: 'git-credentials', branch: 'main'
stash includes: 'deploy-script.ps1', name: 'deploy-script'
}
}
stage('Deploy') {
agent { label agentLabel }
when {
expression {
!skipBuild
}
}
steps {
unstash 'deploy-script'
script {
println "Execute powershell deploy script on agents set for deploy"
}
}
}
}
}
I think that skipDefaultCheckout is what are you looking for:
pipeline {
options {
skipDefaultCheckout true
}
stages {
stage('Prep') {
steps {
script {
........................
}
}
}
}
}
Take a look to the documentation:
skipDefaultCheckout
Skip checking out code from source control by default in the agent directive.
https://www.jenkins.io/doc/book/pipeline/syntax/
I think you are requesting the impossible.
Now:
your Jenkinsfile is inside your jenkins configuration and is sent as such to each of your agents. No need for git on your agents.
Pipeline script for SCM:
Since you use git, SCM = git. So you are saying: my Pipeline needs to be fetched from a git repository. You are declaring the Deploy step to run on agent { label agentLabel }, so that step is supposed to run on another agent than master.
How would you imagine that agent could get the content of the Jenkinsfile to know what to do, but not use git ?
What happens in Jenkins?
Your master agent gets triggered that it needs to build
the master agent checkouts the Jenkinsfile using git (since it is a Pipeline script from SCM)
jenkins reads the Jenkinsfile and sees what has to be done.
for the Prep stage, I'm not quite sure what happens without agent, I guess that runs on master agent.
the Powershell deploy script checkout is marked to run on master agent, so it runs on master agent (note that the Jenkinsfile will get checked out with git two more times:
before starting the stage, because jenkins needs to know what to execute
one more checkout because you specify git url: 'https://github.com/owner/repo.git'...
the Deploy stage is marked to run on agentLabel, so jenkins tries to checkout your Jenkinsfile on that agent (using git)...
You can use Scripted Pipeline to do this, it should basically look like this
node('master') {
checkout scm
stash includes: 'deploy-script.ps1', name: 'deploy-script'
}
def stepsForParallel = [:]
env.AGENTS.split(' ').each { agent ->
stepsForParallel["deploy ${agent}"] = { ->
node(agent) {
unstash 'deploy-script'
}
}
parallel stepsForParallel
you can find all info about jenkins agent section here.
Shortly: you can call any agent by name or label.
pipeline {
agent {
label 'master'
}
}
If it will not work for you, then you will need to set any label on master node and call it by label
pipeline {
agent {
label 'master_label_here'
}
}
Related
I am starting to work with jenkinsfiles. The jenkinsfile contains an echo message (i.e. Hello world)
This is my case:
I have jenkins (ver 2.190.1) installed on a pc with s.o. windows (master agent).
My slave agent is a pc with s.o. linux.
I put my jenkinsfile in my scm.
I have successfully configured the pipeline to run the jenkinsfile. Which is done successfully.
Jenkins makes repository checkout on *master agent* and not on *slave agent* (what I want) and *option "Lightweight checkout"* is checked.
I want this behaviour because my pipeline must to work on slave agent. And I don't want my repoository on master agent.
I searched on the net for a possible solution but without results.
Could you give me a suggestion on how to checkout my repository directly on the slave agent?
It can be done this way. From the below example just replace agentLabelName to your agent name.
Scripted Pipeline
node('agentLabelName') {
stage('stageName') {
echo "${env.WORKSPACE}"
//checkout scm // If Jenkinsfile availabe with your SCM
git url: 'https://github.com/samitkumarpatel/test0.git', branch: 'main'
}
}
Declarative Pipeline
pipeline {
agent {
label 'agentLabelName'
}
stages {
stage('stageName') {
steps {
echo "Hello World"
echo "${env.WORKSPACE}"
//checkout scm // If Jenkinsfile availabe with your SCM
git url: 'https://github.com/samitkumarpatel/test0.git', branch: 'main'
}
}
}
}
I'm trying to configure job in Jenkins Multibranch pipeline. There are a lot of branches in SVN and I want the job to checkout only the latest one and ignores the rest of them. This job triggers a pipeline that does multiple checks on the whole build... so I always need to trigger this on the latest branch because there I will have the latest revision of the build.
The SVN structure is like this: V01_01_01 till the latest one V01_08_03. Currently I have it set up like the below and in the Jenkins pipeline I have "checkout scm", but if a new branch appears e.g. V01_08_04 I need V01_08_03 to be replaced by V01_08_04. Is there any way to do this ?
My set-up in Jenkins Multibranch pipeline
I found a hack to this. I created a python script that checks the whole repository for the latest folder that was updated.
pipeline
{
agent any
parameters
{
string(name: 'latest_folder', defaultValue: '')
}
stages
{
stage ('find latest folder')
{
steps
{
withPythonEnv('System-CPython-3.8')
{
sh 'pip3 install svn'
script {
def folder_name = sh(script: 'python3 latest_folder_svn.py', returnStdout: true)
env.latest_folder = folder_name
}
}
}
}
stage ('Checkout Step')
{
steps
{
echo "${env.latest_folder}"
}
}
}
}
This variable I will add it in the checkout step in order to have always the latest branch.
The python script is pretty straightforward. I use svn library to parse the repository and extract what I need.
I have a stage in Jenkins declarative pipeline that I want to run conditionally when manually triggered, and only on the master branch.
stage('Deploy to prod') {
when {
branch 'master'
}
input {
message "Deploy to prod?"
ok "Deploy"
}
agent any
steps {
..
}
}
I'd like this stage to be skipped altogether for branches other than master, but what happens in practice is that it gets paused for all branches. Is there a way to get the behaviour I'm after?
According to the declarative pipeline doc for input
The stage will pause after any options have been applied, and before entering the stage's agent or evaluating its when condition.
To make the pipeline work the way you specified, I would convert input to a (non-declarative style) step like this:
stage('Deploy to prod') {
when {
branch 'master'
}
agent any
steps {
input message: "Deploy to prod?", ok: "Deploy"
..
}
}
With Jenkins using the Declarative Pipeline Syntax how do i get the Dockerfile (Dockerfile.ci in this example) from the SCM (Git) since the agent block is executed before all the stages?
pipeline {
agent {
dockerfile {
filename 'Dockerfile.ci'
}
}
stage ('Checkout') {
steps {
git(
url: 'https://www.github.com/...',
credentialsId: 'CREDENTIALS',
branch: "develop"
)
}
}
[...]
}
In all the examples i've seen, the Dockerfile seems to be already present in the workspace.
You could try to declare agent for each stage separately, for checkout stage you could use some default agent and docker agent for others.
pipeline {
agent none
stage ('Checkout') {
agent any
steps {
git(
url: 'https://www.github.com/...',
credentialsId: 'CREDENTIALS',
branch: "develop"
)
}
}
stage ('Build') {
agent {
dockerfile {
filename 'Dockerfile.ci'
}
steps {
[...]
}
}
}
[...]
}
If you're using a multi-branch pipeline it automatically checks out your SCM before evaluating the agent. So in that case you can specify the agent from a file in the SCM.
The answer is in the Jenkins documentation on the Dockerfile parameter:
In order to use this option, the Jenkinsfile must be loaded from
either a Multibranch Pipeline or a Pipeline from SCM.
Just scroll down to the Dockerfile section, and it's documented there.
The obvious problem with this approach is that it impairs pipeline development. Now instead of testing code in a pipeline field on the server, it must be committed to the source repository for each testable change. NOTE also that the Jenkinsfile checkout cannot be sparse or lightweight as that will only pick up the script -- and not any accompanying Dockerfile to be built.
I can think of a couple ways to work around this.
Develop against agents in nodes with the reuseNode true directive. Then when code is stable, the separate agent blocks can be combined together at the top of the Jenkinsfile which must then be loaded from the SCM.
Develop using the dir() solution that specs the exact workspace directory, or alternately use one of the other examples in this solution.
We use the jenkins pipeline dsl for our job descriptions. Now we have something like that:
pipeline {
agent none
options {
timestamps()
}
environment {
//SOME ENV VARS
}
stages {
// more stages
stage('stage1'){
when {
expression { env.BRANCH_NAME == 'master' }
}
agent { label 'slave1' }
steps{
//doing stuff
}
}
}
A stage in the build process that should only run when the master branch is build, you can think of a deploy job or something in that direction. The problem is, our resources of agents with that particular label are limited. When we build other branches the job still invoke a slave1 agent and than skips the stage after checking the that the branch is not master. This is bad, because when all slave1 agents are working on master branch jobs, the other jobs will have to wait for a slave1 agent becoming available just to check that it does need to run that stage.
Is there any way with the jenkins pipeline DSL to skip that stage without waiting for the slave1 agent to determine the branch?
The approach provided in the accepted answer works because you are not declaring an agent in the stage('stage1'). Instead you explicitly create a node within the steps and so the agent is not created when you check the condition.
This approach will work but it adds unnecessary complexity to your Jenkinsfile. As of Jenkins pipeline plugin version 1.2.6, the correct way to achieve this would be:
pipeline {
agent none
stages {
// more stages
stage('stage1'){
when {
beforeAgent true
branch 'master'
}
agent { label 'slave1' }
steps {
// do stuff
}
}
}
}
Check the syntax and available options available in the when tag documentation.
I found a solution that worked for me. Although I'm not quite sure why.
Introducing a parallel section and using nodes however fixed the issue and the stages are skipped without invoking the agent first. See the modified pipeline:
pipeline {
agent none
options {
timestamps()
}
environment {
//SOME ENV VARS
}
stages {
// more stages
stage('stage1'){
when { branch 'master' }
steps{
parallel(
'Job1': {
node( 'slave1' ){
//doing stuff
}
}
)
}
}
}