There is an open bug on jenkins 2.0 pipeline scripts relating to included regions in git, so it means for a large mono-repo as in my case each checkin to master will cause multiple pipelines to be kicked off which is not the desired behavior.
So to visualize:
top-level:
->application folder 1
->application folder 2
What I want to do is to do a git fetch first so then I can do a git diff to see if anything in the particular folder has changed and if it has then run the pipeline for that particular folder and not do anything if nothing changed
Code I have is below:
node{
git credentialsId: 'cred id', url: 'ssh://git#git-repo:1234/app/mono-repo.git'
ret = sh(script: 'git fetch; git diff origin/master remotes/origin/master | grep "folder-name"', returnStatus: true)
if(ret == 0){
doSomething()
}else{
doNothing()
}
}
The issue I have that the git fetch fails due a permissions error, I can use a the checkout but then I cannot get the diff before hand which is not what. Is there a way of using the u tiling the git fetch using the credentias?
It might help to simply get the references to tags. Note, I believe this is not equivalent to git fetch --tags. See Does "git fetch --tags" include "git fetch"? for example.
git([branches: [
[name: '*/master'],
[name: 'refs/tags/*:refs/tags/*']],
credentialsId: CREDENTIALS_ID_GIT,
url: REPO])
I noticed that on the Jenkin's console the Git plugin is performing a git fetch --tags, so by default the Git Plugin may already provide this functionality. Please check on this.
I'd like to also add this solution:
withCredentials(
[usernamePassword(
credentialsId: CREDENTIALS_ID_GIT,
passwordVariable: 'GIT_PASSWORD',
usernameVariable: 'GIT_USERNAME')]) {
sh("git fetch --tags https://${GIT_USERNAME}:${GIT_PASSWORD}#${REPO}")
}
Make sure you are using SSH credentials with the correct user. You can check this answer, which is summarized by the capture below :
In particular, make sure that in the url ssh://git#git-repo:1234/app/mono-repo.git the git# part is your actual SSH user. In my case it is the jenkins user, so I would use ssh://jenkins#git-repo:1234/app/mono-repo.git instead.
Related
Context
This question relates to multibranch pipelines where the behaviour merges the PR with the target branch revision (see screenshot of settings)
In this case, the merge may cause a new merge commit. So for a trigger with a given commit from a repository:
We actually get a different value of the GIT_COMMIT envvar:
If a tool (such as a build reporting tool) needs to use the GIT_COMMIT envvar to pass information onto a service, it cannot then be linked back to the actual commit from the project (this is a screenshot from Bitbucket but this would be the same for any repo hosting service):
Question
How, in a pipeline step, can I find the commit 709502c is the actual genesis of this build, when the GIT_COMMIT is set to 6781a3d1 (which is not an actual commit in the project)?
Maybe looking at git history can help?
REAL_GIT_COMMIT = sh (
script: "git rev-parse HEAD",
returnStdout: true,
).trim()
Background
My team wants to update several linting rules in our project, however, doing so will cause our Jenkins build pipeline which lints, tests and builds each feature branch to break. We don't want to lose the value of linting each feature branch before merging, so we agree that linting only the files that the feature branch changes is a reasonable way to introduce these new lint rules without forcing us to re-lint the whole project up-front. Given that our entire project is already linted, this seems like a reasonable move.
A while ago I wrote a git tool to do exactly this. It determines which files have changed since the feature branch diverged from master and outputs those files so they can be consumed by eslint, pycodestyle and other linters. Here's the source if you're interested in how this is done.
Problem
Jenkins declarative build process and it's GitHub Branch Source Plugin seem to have a brittle checkout behavior that can't be modified to checkout more than just the feature branch that it's called on to build.
If I call git fetch origin stage within a build step, Jenkins complains about missing credentials. I don't feel comfortable sticking in credentials into my pipeline file, I'd MUCH prefer to continue using the Git plugin to manage credentials to our private github repo and pull branches, however, I'm at a loss as to how to specify for it to fetch more than just the feature branch.
For reference, here's the relevant portions of my Jenkinsfile
As you can see, I've tried adding the GitSCM code block to no avail. I've read this medium article which solves a similar problem, but I'm not using SSH credentials and I'd prefer not to given than we're already managing credentials using the Git plugin.
pipeline {
agent any
tools {
nodejs 'node12.7.0'
}
stages {
stage('checkout') {
steps {
checkout([
$class: 'GitSCM',
branches: [[name: '*']],
extensions: scm.extensions,
userRemoteConfigs: [],
doGenerateSubmoduleConfigurations: true
])
}
}
stage('install') {
steps {
script {
sh 'git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master'
sh 'yarn install'
}
}
}
stage('lint & test') {
failFast true
parallel {
stage('lint') {
when {
not {
anyOf {
branch 'stage'; branch 'int'; branch 'prod'
}
}
}
steps {
script {
sh """
git submodule update --init
yarn run lint
"""
}
}
}
...
}
}
stage('deploy') {...}
}
}
post {
failure {
notifySlack()
}
}
}
Create credential in your Jenkins with ssh key and private key, which can be added to the checkout userRemoteConfigs which will be used while checking out (Value given down just an example of one the credential id in my Jenkins environment)
userRemoteConfigs: [[credentialsId: '7969s7612-adruj-au2cd-492msa802f']]
One frequent root cause - mentioned on the referenced medium article, too - is that Jenkins only checks out the current branch that needs to be build.
An easy option I just found to have other project branches available is to
Configure your pipeline job
Under Behaviors->General, add Specify ref specs
Optionally adjust the parameter to the refs you need, e.g. the branches to compare to. Or you can get all branches by maintaining the default +refs/heads/*:refs/remotes/#{remote}/* as shown in the screenshot:
Jenkins Pipeline Job - Ref Config
P.S.: This seems to be part of the GIT Jenkins plugin, but I couldn't find it in the docs...
I've created a Jenkins Multibranch Pipeline with the GitHub Branch Source plugin. The Jenkinsfile essentially just calls a Cake Build script (build.ps1, build.cake) that contains all the build/deploy logic. This allows me to move to another CI service easily.
Unfortunately, I cannot seem to figure out how to add my Cake Build scripts as a trusted file so that PR's from forks will pull the files from the source repo instead. The Trust setting of the Discover pull requests from forks behavior seems to indicate that there can be other trusted files besides Jenkinsfile:
Nobody
Pull requests from forks will all be treated as untrusted. This means that where Jenkins requires a trusted file (e.g. Jenkinsfile) the contents of that file will be retrieved from the target branch on the origin repository and not from the pull request branch on the fork repository.
However, I cannot seem to find any documentation on adding other trusted files. The primary reason for this is to prevent a PR from a fork from accessing credentials from the Cake script. They wouldn't be able to change Jenkinsfile, but they could still change the Cake script to expose the credentials.
Is it actually possible to add other trusted files?
It seems like Jenkins does not support this. My solution is checking out the untrusted files manually from the base version instead. First getting the commit 's hash of the base version with:
def commit = sh(
script: 'git rev-parse HEAD',
returnStdout: true
).trim()
def base = sh(
script: "git rev-list --parents -n 1 ${commit}",
returnStdout: true
).trim().split('\\s+')[2]
git rev-list --parents -n 1 ${commit} will return the hash of current commit, which is a merge commit that was created by Jenkins; the latest commit of the PR and the latest commit of the target branch, separated by a space (e.g. 05e9322574ea03003f87dcbb44f172e6fa62581f b3f6ef892af9c645f490106757d7d05df3a26060 069ffd55ae36414a51b4de166aef86966f9447a8). Hence, we grab the hash of the latest commit of the target branch by trim().split('\\s+')[2].
Now we can do sh "git checkout ${base} FILE" on any file that we don't trust from the PR.
This does not works if the PR is already merged with the latest version of target branch. So what I did is something like this:
// revert untrusted files to the base version and backup it before we execute any untrusted code so the attacker
// don't have a chance to put a malicious content
def latest = sh(script: 'git rev-parse HEAD', returnStdout: true).trim()
sh "git checkout origin/${env.CHANGE_TARGET}"
def baseCompose = readFile('docker-compose.yml')
// switch back to latest commit
sh "git checkout ${latest}"
sh 'git clean -d -f -f -q -x'
I need to clear workspace before build starts. I tried using cleanDir() in stages, but in the declarative pipeline, check out happens first and when stage with cleadDir runs, checked out code also gets cleared which is not desired. How can we clear the workspace before check out in the declarative pipeline?
Actually, I have to revise my answer based on recent changes to the pipeline plugins, e.g. GitHub Branch Source Plugin 2.2.0 with JENKINS-43507.
Besides the different branch discovery behaviours, which can be configured, one can now define additional steps to take, including Clean before checkout (and Clean after checkout):
The resulting output in the pipeline execution will then be
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
so, pretty close to the calling git clean yourself.
stage('Git') {
steps {
step([$class: 'WsCleanup'])
checkout scm
}
}
the WsCleanup does the trick
Use the means of your VCS, with Git run
git clean -fdx
Add Wipe out repository & force clone in Additional Behaviours of SCM configuration as on picture below:
Something like this should work:
env.WORKSPACE = pwd()
sh "rm ${env.WORKSPACE}/* -fr"
A slightly different approach would be to use the stash step to stash your code first:
stash includes: 'src/**', name: 'source-code'
After that you can delete everything in your current Workspace. In a later stage you can simply unstash the source code again:
unstash 'source-code'
Another advantage of stash/unstash is that you can use it to share files between multiple jenkins nodes.
I need to know which branch is being built in my Jenkins multibranch pipeline in order for it to run steps correctly.
We are using a gitflow pattern with dev, release, and master branches that all are used to create artifacts. The dev branch auto deploys, the other two do not. Also there are feature, bugfix and hotfix branches. These branches should be built, but not produce an artifact. They should just be used to inform the developer if there is a problem with their code.
In a standard build, I have access to the $GIT_BRANCH variable to know which branch is being built, but that variable isn't set in my multibranch pipeline. I have tried env.GIT_BRANCH too, and I tried to pass $GIT_BRANCH as a parameter to the build. Nothing seems to work. I assumed that since the build knows about the branch being built (I can see the branch name at the top of the console output) that there is something that I can use - I just can't find any reference to it.
The env.BRANCH_NAME variable contains the branch name.
As of Pipeline Groovy Plugin 2.18, you can also just use BRANCH_NAME
(env isn't required but still accepted.)
There is not a dedicated variable for this purpose yet (JENKINS-30252). In the meantime you can take advantage of the fact that the subproject name is taken from the branch name, and use
env.JOB_NAME.replaceFirst('.+/', '')
This has now been resolved, see Krzysztof KrasoĊ's answer.
There are 2 branches to consider in a Jenkins multibranch pipeline job:
The Jenkins job branch - env.BRANCH_NAME. This may have the same name as a git branch, but might also be called PR-123 or similar
The git branch - env.GIT_BRANCH. This is the actual branch name in git.
So a job might have BRANCH_NAME=PR-123 and GIT_BRANCH=my-scm-branch-name
Jenkins documentation has a list of all the env variable for your perusal here
Another way is using the git command to obtain the branch name on the current jenkins pipeline. For example, you can add the following snippet to print the branch name in your Jenkinsfile.
...
script {
def BRANCH = sh(returnStdout: true, script: 'git rev-parse --abbrev-ref HEAD').trim()
echo ${BRANCH}
}
...
I found this stackoverflow post example useful: Git Variables in Jenkins Workflow plugin
sh '//...
git rev-parse --abbrev-ref HEAD > GIT_BRANCH'
git_branch = readFile('GIT_BRANCH').trim()
echo git_branch
//...
'