Jenkins file diffs with declarative pipeline and multibranch - jenkins

I have a scenario where I manage two pipelines within the same repo, with two different Jenkinsfiles.
I have set up two Jenkins multibranch pipelines to handle the two different Jenkinsfiles, by path discovery and set up github webhooks to trigger a build when a PR on particular branches is created.
I have not found a way to get changes in files for a particular PR so I thought of leveraging git by doing git diff --name-status origin/master...HEAD but it fails, since Jenkins only checks out the target branch.
The logstash:
using credential github-user-token-uname-pwd
Fetching changes from the remote Git repository
Fetching without tags
> git rev-parse --is-inside-work-tree # timeout=10
> git config remote.origin.url https://github.com/myreponame # timeout=10
Fetching upstream changes from https://github.com/myreponame
> git --version # timeout=10
using GIT_ASKPASS to set credentials Github token in uname-pwd form used by jenkins to register and manage webhooks
> git fetch --no-tags --force --progress https://github.com/myreponame +refs/heads/BRANCH-X:refs/remotes/origin/BRANCH-X
Checking out Revision 440df9b08667458fa4ade4be57ecbf59a4 (BRANCH-X)
Commit message: "move post build where it belongs"
> git config core.sparsecheckout # timeout=10
> git checkout -f 440df9b08667458fa4ade4be57ecbf59a4
> git rev-list --no-walk ab28e843c0fc7807c4cbd2d6f612e5d76b # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withCredentials
Masking supported pattern matches of $SECRET_ACCESS_KEY or $ACCESS_KEY_ID
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] sh
+ git diff --name-status origin/master...HEAD
and the error I get:
fatal: ambiguous argument 'origin/master...HEAD': unknown revision or path not in the working tree.
Is there a way to retrieve the files' changes of a PR within multibranch pipelines or get Jenkins to be able to discover the source branch of said PR?

You could try to fetch origin and then diff with FETCH_HEAD
{ git fetch origin master
git diff —name-status FETCH_HEAD...HEAD
}

So here is what I had to do.
The error I was facing was due to Jenkins default checkout behaviour with multibranch pipelines: it checks out and track a branch and that branch only, it does not fetch other branches.
I eventually added
options {
skipDefaultCheckout true
}
and a stage that checks out all the origin remote branches
stage('Checking out repo'){
steps {
script {
checkout(
[
$class: 'GitSCM',
branches: [[name: 'origin/FEATUREBRANCH*']],
doGenerateSubmoduleConfigurations: false,
extensions: [[$class: 'LocalBranch'],
[$class: 'CleanBeforeCheckout']],
submoduleCfg: [],
userRemoteConfigs: [[credentialsId: 'github-deploy-key',
url: 'git#github.com:myawesomerepo.git']]
]
)
}
}
}

Related

Jenkins git checkout on agent not working

The Jenkins file in my github repository is used in a Jenkins Master/Slave environment.
I need to execute a testing command on a remote Jenkins Slave Server.
In my declarative pipeline the agent is called like this:
stage("Testautomation") {
agent { label 'test-device' }
steps {
bat '''
#ECHO ON
ECHO %WORKSPACE%
... '''
}
}
Before Jenkins can even execute a remote command, it starts checking out from version control. The checkout on Jenkins Master is no problem and working fine. But on this Jenkins Slave I always receive this error message.
using credential github-enterprise:...
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://...git # timeout=10
Fetching upstream changes from https://...git
> git --version # timeout=10
using GIT_ASKPASS to set credentials GitHub Enterprise Access Token
> git fetch --tags --force --progress --depth=1 -- https://...git +refs/heads/development:refs/remotes/origin/development # timeout=120
Checking out Revision ... (development)
> git config core.sparsecheckout # timeout=10
> git checkout -f ...
Could not checkout ...
The Declarative pipeline performs a SCM checkout on every agent by default. Check if Git is installed on the Jenkins slave.
Conversely, if you want the code to be checked out on master but not on the agent, disable the default checkout in the options directive and use the scm checkout step inside a stage.
pipeline {
agent { label 'master' }
options {
skipDefaultCheckout(true)
}
stages {
stage('Build') {
steps {
checkout scm
// do other stuff on master
}
}
stage("Testautomation") {
agent { label 'test-device' }
steps {
bat '''
#ECHO ON
ECHO %WORKSPACE%
'''
}
}
}
}
You can further customize the checkout behavior as described in this answer https://stackoverflow.com/a/42293620/8895640.

Jenkins multibranch pipeline triggers pipeline on unrelated branches

I have a problem with Jenkins multibranch pipleline using JenkinsFile and the GIT plugin.
The problem is that every push to staging branch triggers the pipeline of master as well.
The desired behavior is that push to staging branch only triggers the pipleine for staging, and push to master branch only triggers the pipeline for master
This is my JenkinsFile
#!/usr/bin/env bash
pipeline {
agent any
triggers {
pollSCM('*/1 * * * *')
}
environment {
GCLOUD_PATH="/var/jenkins_home/GoogleCloudSDK/google-cloud-sdk/bin"
}
stages {
stage('Git Checkout'){
steps{
// Clean Workspace
cleanWs()
// Get source from Git
git branch: 'staging',
credentialsId: ****',
url: 'git#github.com:***/****.git'
}
}
stage('Update Staging') {
when {
branch 'staging'
}
environment{
INSTANCE="***"
}
steps {
sshagent(credentials : ['****']) {
sh 'ssh -tt -o StrictHostKeyChecking=no jenkins#"${INSTANCE}" sudo /opt/webapps/****/deploy.sh firstinstance'
}
}
}
stage('Update Production') {
when {
branch 'master'
}
environment{
gzone="us-central1-a"
}
steps {
sh '''
#!/bin/bash
echo "${BRANCH_NAME}"
export instances=$("${GCLOUD_PATH}"/gcloud compute instances list --filter="status:(running) AND tags.items=web" --format="value(name)")
FIRST=1
for instance in ${instances}
do
echo "### Running Instance: ${instance} ###"
if [[ $FIRST == 1 ]]; then
echo "first instance"
${GCLOUD_PATH}/gcloud compute ssh jenkins#${instance} --zone ${gzone} '--ssh-flag=-tt -i /root/.ssh/id_rsa -o StrictHostKeyChecking=no' --command="echo first"
else
${GCLOUD_PATH}/gcloud compute ssh jenkins#${instance} --zone ${gzone} '--ssh-flag=-tt -i /root/.ssh/id_rsa -o StrictHostKeyChecking=no' --command="sudo uptime"
fi
FIRST=0
done
'''
}
}
}
post {
success {
cleanWs()
}
}
}
I'll share some logs:
The is a log for master branch
http://34.69.57.212:8080/job/tinytap-server/job/master/2/pollingLog/ returns
Started on Dec 10, 2019 1:42:00 PM
Using strategy: Specific revision
[poll] Last Built Revision: Revision 12ecdbc8d2f7e7ff1f578b135ea0b23a28d7672d (master)
using credential ccb9a735-04d9-4aab-8bab-5c86fe0f363c
> git --version # timeout=10
using GIT_ASKPASS to set credentials
> git ls-remote -h -- https://github.com/tinytap/tinytap-web.git # timeout=10
Found 222 remote heads on https://github.com/tinytap/tinytap-web.git
[poll] Latest remote head revision on refs/heads/master is: 12ecdbc8d2f7e7ff1f578b135ea0b23a28d7672d - already built by 1
Using strategy: Default
[poll] Last Built Revision: Revision f693e358ce14bc5dfc6111e62ed88e6dd1d0dfc9 (refs/remotes/origin/staging)
using credential 17f45a89-da78-4969-b18f-cb270a526347
> git --version # timeout=10
using GIT_SSH to set credentials jenkins key
> git ls-remote -h -- git#github.com:tinytap/tinytap-web.git # timeout=10
Found 222 remote heads on git#github.com:tinytap/tinytap-web.git
[poll] Latest remote head revision on refs/heads/staging is: 907899a0e7e131e9416ee65aad041c8da111e2fe
Done. Took 1 sec
Changes found
The is a log for master branch, but only staging had a new commit :
http://34.69.57.212:8080/job/tt-server/job/master/3/pollingLog/ returns
Started on Dec 10, 2019 1:55:00 PM
Using strategy: Specific revision
[poll] Last Built Revision: Revision 12ecdbc8d2f7e7ff1f578b135ea0b23a28d7672d (master)
using credential ****-****-****-****-5c86fe0f363c
> git --version # timeout=10
using GIT_ASKPASS to set credentials
> git ls-remote -h -- https://github.com/tt/tt-web.git # timeout=10
Found 222 remote heads on https://github.com/tt/tt-web.git
[poll] Latest remote head revision on refs/heads/master is: 12ecdbc8d2f7e7ff1f578b135ea0b23a28d7672d - already built by 2
Using strategy: Default
[poll] Last Built Revision: Revision 907899a0e7e131e9416ee65aad041c8da111e2fe (refs/remotes/origin/staging)
using credential ****-****-****-****-cb270a526347
> git --version # timeout=10
using GIT_SSH to set credentials jenkins key
> git ls-remote -h -- git#github.com:tt/tt-web.git # timeout=10
Found 222 remote heads on git#github.com:tt/tt-web.git
[poll] Latest remote head revision on refs/heads/staging is: eab6e8bc6d8586084e9fe9856dec7fd8b31dd098
Done. Took 0.98 sec
Changes found
Notice "changes found" even though head did not change on master branch
Jenkins ver. 2.190.1
Git plugin ver 4.0.0
Git client plugin ver 2.9.0
I use this plugin - https://github.com/lachie83/jenkins-pipeline and it works fine for me. You need to have separate if blocks for each branch and then the stage block inside it. Example below:
#!/usr/bin/groovy
#Library('https://github.com/lachie83/jenkins-pipeline#master')
def pipeline = new io.estrado.Pipeline()
def cloud = pipeline.getCloud(env.BRANCH_NAME)
def label = pipeline.getPodLabel(cloud)
// deploy only the staging branch
if (env.BRANCH_NAME == 'staging') {
stage ('deploy to k8s staging') {
//Deploy to staging
}
}
// deploy only the master branch
if (env.BRANCH_NAME == 'master') {
stage ('deploy to k8s production') {
//Deploy to production
}
}
I think you have some logical omissions in your Jenkinsfile. As it currently stands, you poll SCM for changes. If any change is detected, first stage 'Git Checkout' will checkout staging branch (always). Then you have another stage which does something if the branch is 'staging' (which it is, because it's hardcoded to checkout that branch above) etc. This will be the first thing to fix - if SCM changes are detected, checkout the right branch. How - there are a few options. I usually use 'skipDefaultCheckout()' in 'options' together with explicit checkout in my first pipeline stage:
steps {
sshagent(['github-creds']) {
git branch: "${env.BRANCH_NAME}", credentialsId: 'github-creds', url: 'git#github.com:x/y.git'
}
}
The second thing is that you try to squeeze handling two different branches into a single Jenkinsfile. This is not how it should be done. Jenkins wil use Jenkinsfile from a given branch - just make sure Jenkinsfile on staging contains what you want it to contain, same with Jenkinsfile on master.
Hope it helps.

Jenkins Pipeline - Jenkins SCM step. Checkout returns invalid data not referring to parameters provided

I am setting up CI/CD for my project and decided to use Jenkins Pipeline. What I need basically:
Build from chosen branch(chosen from dropdown)
Execute some stages conditionally(depending on branch)
I configured pipeline to fetch Jenkinsfile from scm:
Then I decided to write cutom scm checkout as following:
stage('Checkout specific branch') {
steps {
echo "Checking out ${params.branchToBuild}..."
script {
def actualBranchCheckout = checkout([$class: 'GitSCM',
branches: [
[name: "${params.branchToBuild}"]
],
doGenerateSubmoduleConfigurations: false,
extensions: [[$class: 'LocalBranch', localBranch: "**"]],
submoduleCfg: [],
userRemoteConfigs: [
[credentialsId: "${env.REPO_SSH_CREDS_ID}", url: "${env.REPO_SSH_URL}"]
]
])
}
}
}
And I have sveral concerns. GIT_ environment variables are set when checking out branch for Jenkinsfile.
GIT_BRANCH=origin/master
GIT_COMMIT=11a5800b6ca31bc81545fa4874d73fa275c820c2, GIT_LOCAL_BRANCH=master, GIT_PREVIOUS_COMMIT=11a5800b6ca31bc81545fa4874d73fa275c820c2
That is fine, but when my checkout is performed the are not getting updated with actual values and remain the same.
Okay, the method itself returns map which may contain what I need so I can manually reset them, right?
But they turned out to be the same when checking out develop branch:
Here are the logs:
Obtained Jenkinsfile from git git#bitbucket.org:team/repo.git
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/Repo/repo_pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
using credential jenkins-bitbucket
Cloning the remote Git repository
Cloning repository git#bitbucket.org:team/repo.git
> git init /var/lib/jenkins/workspace/Repo/repo_pipeline # timeout=10
Fetching upstream changes from git#bitbucket.org:team/repo.git
> git --version # timeout=10
using GIT_SSH to set credentials
> git fetch --tags --progress git#bitbucket.org:team/repo.git +refs/heads/*:refs/remotes/origin/*
> git config remote.origin.url git#bitbucket.org:team/repo.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url git#bitbucket.org:team/repo.git # timeout=10
Fetching upstream changes from git#bitbucket.org:team/repo.git
using GIT_SSH to set credentials
> git fetch --tags --progress git#bitbucket.org:team/repo.git +refs/heads/*:refs/remotes/origin/*
> git rev-parse refs/remotes/origin/master^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 31e727bf81b3379c9aa80224b8f941bc867acd77 (refs/remotes/origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 31e727bf81b3379c9aa80224b8f941bc867acd77
Commit message: "print out variables"
> git rev-list --no-walk 11a5800b6ca31bc81545fa4874d73fa275c820c2 # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Checkout specific branch)
[Pipeline] echo
Checking out develop...
[Pipeline] script
[Pipeline] {
[Pipeline] checkout
using credential jenkins-bitbucket
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url git#bitbucket.org:team/repo.git # timeout=10
Fetching upstream changes from git#bitbucket.org:team/repo.git
> git --version # timeout=10
using GIT_SSH to set credentials
> git fetch --tags --progress git#bitbucket.org:team/repo.git +refs/heads/*:refs/remotes/origin/*
> git rev-parse origin/develop^{commit} # timeout=10
Checking out Revision 11a5800b6ca31bc81545fa4874d73fa275c820c2 (origin/develop)
> git config core.sparsecheckout # timeout=10
> git checkout -f 11a5800b6ca31bc81545fa4874d73fa275c820c2
> git branch -a -v --no-abbrev # timeout=10
> git checkout -b develop 11a5800b6ca31bc81545fa4874d73fa275c820c2
Commit message: "another try"
First time build. Skipping changelog.
[Pipeline] echo
[GIT_BRANCH:origin/master, GIT_COMMIT:31e727bf81b3379c9aa80224b8f941bc867acd77, GIT_LOCAL_BRANCH:master, GIT_PREVIOUS_COMMIT:11a5800b6ca31bc81545fa4874d73fa275c820c2, GIT_PREVIOUS_SUCCESSFUL_COMMIT:11a5800b6ca31bc81545fa4874d73fa275c820c2, GIT_URL:git#bitbucket.org:team/repo.git]
I have another option to do everything manually, like git rev-parse --abbrev-ref HEAD for branch and so on. Do anyone know how to solve this/ is it a known issue or I am doing something wrong way?

How do you print out the value of a class property in a Jenkinsfile?

I'm creating a Jenkinsfile for use with GitHub Enterprise. I used the GUI settings in the pipeline job to specify a Jenkinsfile from a GitHub repo.
I'm using the scripted syntax instead of the declarative syntax.
I am able to checkout the repo in my Jenkinsfile using checkout scm. I want to use some information about the checkout in my script, such as the branch name and commit hash. However, I can't figure out how to access variables of the scm class.
When I run the job, it fails in the Checkout stage. The checkout from git seems to work properly, but it fails without printing any errors. If I delete the echo scm.GIT_BRANCH line it works fine.
node {
stage('Checkout') {
checkout scm
echo scm.GIT_BRANCH
}
}
Here's the output:
Started by user spark
Obtained nightly/Jenkinsfile from git https://github.enterprise.instance.com/spark/ci_flow_test
[Pipeline] node
Running on jenkins-server in /home/spark/ci_flow_test/pipeline_test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Checkout)
[Pipeline] checkout
> /apps/git/git18/bin/git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> /apps/git/git18/bin/git config remote.origin.url https://github.enterprise.instance.com/spark/ci_flow_test # timeout=10
Fetching upstream changes from https://github.enterprise.instance.com/spark/ci_flow_test
> /apps/git/git18/bin/git --version # timeout=10
using GIT_ASKPASS to set credentials
> /apps/git/git18/bin/git fetch --tags --progress https://github.enterprise.instance.com/spark/ci_flow_test +refs/heads/*:refs/remotes/origin/*
> /apps/git/git18/bin/git rev-parse refs/remotes/origin/working^{commit} # timeout=10
> /apps/git/git18/bin/git rev-parse refs/remotes/origin/origin/working^{commit} # timeout=10
Checking out Revision 396f172c6061ba2760a71cba817df24836ec7e3b (refs/remotes/origin/working)
Commit message: "try echo"
> /apps/git/git18/bin/git config core.sparsecheckout # timeout=10
> /apps/git/git18/bin/git checkout -f 396f172c6061ba2760a71cba817df24836ec7e3b
> /apps/git/git18/bin/git rev-list 778c36171927020bd1afbd7206d86bf94abd1ed8 # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] mail
[Pipeline] echo
Post script
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: FAILURE
You can use the checkout scm return value to get git infomation
node {
stage('Checkout') {
def d = checkout scm
echo "branch: " + d.GIT_BRANCH
echo "commit: " + d.GIT_COMMIT
}
}
// supported fields
GIT_AUTHOR_EMAIL
GIT_AUTHOR_NAME
GIT_BRANCH
GIT_COMMIT
GIT_COMMITTER_EMAIL
GIT_COMMITTER_NAME
GIT_LOCAL_BRANCH
GIT_PREVIOUS_COMMIT
GIT_PREVIOUS_SUCCESSFUL_COMMIT
GIT_URL

access workspace in jenkins pipeline

I have a jenkins pipeline job with the job config pulling Jenkinsfile from a repo . Once the job runs and pulls the jenkinsfile, it clones the repo and I can see it in workspace icon for the job.
Now when in Jenkinsfile I do cd ${workspace} and ls , it doesnt display anything. How do I access the workspace of the repo of the Jenkinsfile ? Or does it just store the Jenkinsfile itself ?
This is my Jenkinsfile :
node ("master"){
// Mark the code checkout 'Checkout'....
stage 'Checkout'
sh "pwd ; ls"
}
As I run it, I get the following log:
> GitHub pull request #282 of commit
> 0045a729838aae0738966423ff19c250151ed636, no merge conflicts. Setting
> status of 0045a729838aae0738966423ff19c250151ed636 to PENDING with url
> https://10.146.84.103/job/test1/ and message: 'Pull request builder
> for this security group pr started' Using context: SG Terraform
> Validate/Plan
> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository
> > git config remote.origin.url https://github.xxx.net/Terraform/djin-sg/ # timeout=10 Fetching
> upstream changes from https://github.xxx.net/Terraform/djin-sg/
> > git --version # timeout=10 using GIT_ASKPASS to set credentials wsjbuild
> > git fetch --tags --progress https://github.xxx.net/Terraform/djin-sg/
> +refs/heads/*:refs/remotes/origin/*
> > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 9dd8491b7b18c47eac09cec5a4bff4f16df979bf (origin/master)
> > git config core.sparsecheckout # timeout=10
> > git checkout -f 9dd8491b7b18c47eac09cec5a4bff4f16df979bf First time build. Skipping changelog. [Pipeline] node Running on master in
> /var/lib/jenkins/workspace/test1 [Pipeline] { [Pipeline] stage
> (Checkout) Using the ‘stage’ step without a block argument is
> deprecated Entering stage Checkout Proceeding [Pipeline] wrap
> [Pipeline] { [Pipeline] sh [test1] Running shell script
> + cd /var/lib/jenkins/workspace/test1
> + ls
My question specifically is that to get the Jenkinsfile it clones the djin-sg repo. Its in the workspace as well. So When I do ls why does it show no files ?
As I go to the Jenkins job pipeline steps and open workspace in console I can see the full repo in workspace but I cant seem to access it in the job.
Try instead a Jenkins pipeline syntax, like:
pipeline {
agent { node { label 'master' } }
stages {
stage('After Checkout') {
steps {
sh 'pwd; ls'
}
}
}
}
You can do checkout scm to actually checkout a repository to the workspace or you can find it under ../${env.JOB_NAME}#script only on master.
It's better to always checkout checkout scm manually because slaves do not have ../$env.JOB_NAME#script folder.

Resources