Scenario:
A developer creates a PR against the master branch. Jenkins (Cloudbees) goes to work building and validating that PR as well as generating a text file containing a build_info.txt file as an artifact.
When the PR is merged into master I need to be able to access the artifact that was created in the PR validation step, extract the version information it contains and commit that version information into master along with PR code changes.
Problem:
I've printed out the env vars during the merge to master script (run_pr_merge), but I haven't seen information (It might be there, I just don't recognize it) that would allow me to link back to the PR job that is being merged or a way to say "give me the artifacts that were created during this PR's build and validation job"
My script looks something like this:
if (isMasterBranch()) {
// run this when code is pushed to master
sh "bash ./run_pr_merge.sh" // which requires build_info.txt from PR build
} else {
// run this for each PR build
sh "bash ./build_and_validate.sh"
archiveArtifacts 'build_info.txt'
}
Am not super familiar with Jenkins/Cloudbees, so there might be a better way of structuring the pipeline to achieve what I need but am hoping there's a relatively easy way to get hold of the PR info being merged int master.
Have looked at copyArtifacts but again I'm not sure how to to reference the PR being merged. Any help greatly appreciated.
There's a CopyArtifact plugin for Jenkins that can be used to copy artifacts from another job. It supports both "Freestyle" jobs and pipeline. You need to provide job URL, and, optionally, build number (by default it will use last successful build). You can craft the URL of the job provided you have the PR number.
In our environment, the URL for a specific job looks something like:
https://jenkins.mycompany.com/job/mycompany/job/myproj/job/PR-<number>/<build_no>/
Provided I have the PR number, I can build the needed URL with e.g.
def pr_number = 12345
def job_url = "https://jenkins.mycompany.com/job/mycompany/job/myproj/job/PR-${pr_number}/"
step([
$class: 'CopyArtifact',
filter: 'build_info.txt',
fingerprintArtifacts: true,
optional: true,
projectName: job_url
// default selector is "last successful build"
// selector: [$class: 'SpecificBuildSelector',
// buildNumber: build_number]
])
// check the file is there
sh "cat ${WORKSPACE}/build_info.txt"
Related
In a Jenkins / git / Gerrit setup I currently maintain a bunch of "pipeline-scm" jobs: e.g. one of them gets triggered by Gerrit on an incoming change request in order to validate this change.
There is a "Gerrit Trigger" plugin for Jenkins that provides that provides job with a git refspec and the branch that change relates to - among lots of other details.
When you setup a "Pipeline script from SCM" a job instance (build) will be provided with an scm object which can be used to checkout the change and also gets used by Jenkins itself to show some meta information on the build's overview, e.g. the commits that define this change:
Now here comes my problem:
For [..] reasons I need to turn this "Pipeline script from SCM" job into a "Pipeline script" job. I.e. instead of just using the auto-generated scm object (which is being used on the master node already to fetch the pipeline script (and most likely to utilize the meta information about this change)) to run checkout scm in my pipeline script I now have to create this object manually:
scm = [
$class: "GitSCM",
userRemoteConfigs: [[
credentialsId: <SOME ID>,
url: <REPO-URL>,
]],
branches: [[
name: <BRANCH-NAME>,
]],
]
checkout scm
This raises two questions:
I know the Gerrit plugin provides details about a change in environment variables like GERRIT_PATCHSET_REVISION, GERRIT_REFSPEC and GERRIT_BRANCH - but how do I use them? I've seen examples which define branches using GERRIT_BRANCH, others use GERRIT_PATCHSET_REVISION others use a branch pattern like */master.
how does the meta information which can be extracted from a valid scm instance get to the build's overview page? I guess the pipeline-scm mechanism makes the master node populate this page automatically, but now the master node has no scm object in advance anymore. Is there a mechanism like "currentBuild.updateMetaInfo(scm)`?
Is there any documentation about how to properly set up and use a scm object (as Jenkins does) in a standalone pipeline job (i.e. without the scm mechanism)?
Update:
This scm configuration works for me in terms of checkout:
scm = [
$class: "GitSCM",
userRemoteConfigs: [[
credentialsId: <MY_ID>,
url: <MY_REPO_URL>,
refspec: env["GERRIT_REFSPEC],
]],
branches: [[name: "FETCH_HEAD"]],
];
Unfortunately even after checkout scm no "Changes" show up, instead the overview page only displays this:
The build matrix in the job overview also states there are no changes:
Regarding your first question on how to use the GERRIT_ variables with the Jenkins Git SCM, check the docs:
To get the Git Plugin to download your change; set Refspec to
$GERRIT_REFSPEC and the Choosing strategy to Gerrit Trigger. This may
be under ''Additional Behaviours/Strategy For Choosing What To Build'
rather than directly visible as depicted in the screenshot. You may
also need to set 'Branches to build' to $GERRIT_BRANCH. If this does
not work for you set Refspec to refs/changes/:refs/changes/* and
'Branches to build' to *$GERRIT_REFSPEC.
Regarding the second one:
The checkout step should add the information to the build status page.
Regarding your closing question:
The scm object is just a convenience object that represents the scm you would configure in Pipeline from SCM, so you dont need to provide the details specified there in the Pipeline script again.
To generate a checkout step for a Pipeline without SCM, use the Snippet Generator from the Pipeline Syntax page.
Context
This question relates to multibranch pipelines where the behaviour merges the PR with the target branch revision (see screenshot of settings)
In this case, the merge may cause a new merge commit. So for a trigger with a given commit from a repository:
We actually get a different value of the GIT_COMMIT envvar:
If a tool (such as a build reporting tool) needs to use the GIT_COMMIT envvar to pass information onto a service, it cannot then be linked back to the actual commit from the project (this is a screenshot from Bitbucket but this would be the same for any repo hosting service):
Question
How, in a pipeline step, can I find the commit 709502c is the actual genesis of this build, when the GIT_COMMIT is set to 6781a3d1 (which is not an actual commit in the project)?
Maybe looking at git history can help?
REAL_GIT_COMMIT = sh (
script: "git rev-parse HEAD",
returnStdout: true,
).trim()
We are currently using the Jenkins/Groovy pipeline method for CI.
I'm trying to create a single pipeline for building packages and running unit tests on a branch (and let me know if this is bad practice).
The problem is on a commit I don't want to execute my test steps (due to the large number of commits and time to execute the full pipeline), but I still want the packaging stage to run for our manual testers to be able to pull and install on instances.
Is there any way to distinguish between a run for a PR vs a commit in the pipeline steps or in the job configuration?
Using the (assuming github, but theres bitbucket equiv as well)
https://wiki.jenkins.io/display/JENKINS/GitHub+Branch+Source+Plugin to discover Repositories to build branches and PR's will allow you to fall back on Jenkins ENV variables.
This allows simple if statement to determine if the build is for a branch, or for a PR, as PR's are built on a 'branch' of PR-n
Once a PR is open however, all commits would be built.
https://go.cloudbees.com/docs/cloudbees-documentation/cje-user-guide/index.html#github-branch-source
you can use the changeRequest built-in condition from Jenkins.
Executes the stage if the current build is for a "change request" (a.k.a. Pull Request on GitHub and Bitbucket, Merge Request on GitLab, Change in Gerrit, etc.). When no parameters are passed the stage runs on every change request, for example: when { changeRequest() }.
stage('Run only for pull requests to master branch or at the master branch') {
when {
anyOf {
branch 'master'
changeRequest target: 'master'
}
}
steps {
// this is a very long step for integration test that we don't want to execute often, but we need to execute before to merge to master
sh "${mvn} " +
"clean " +
"test-compile " +
"failsafe:integration-test failsafe:verify"
}
}
I have Jenkins LTS 2.60.2 on Windows Server 2016 and using these plugins:
Folders plugin (6.1.0)
Copy Artifact plugin (1.38.1)
Pipeline plugin (2.5) + all dependent pipeline sub-plugins
Various other dependent plugins...
See Pipeline to use artifacts from 2 projects associated by the same git branch name for more details about my setup, but to sum it up I have these items:
playground (a folder created with the Folders plugin to group all these following items)
frontend (multibranch pipeline)
backend (multibranch pipeline)
configure (pipeline with a parameter called BRANCH_NAME)
The frontend and backend git repos, both have a branch called master and one called release/2017.2.
The idea is to call the configure pipeline automatically after each successful build, passing the git branch name. Automatically triggering the configure pipeline works.
What doesn't work and I need your help to fix, is the step inside the configure pipeline to copy the artifacts from a multibranchPipeline/specificBranch.
If for the BRANCH_NAME parameter (or the upstream pipeline) is master it works. If BRANCH_NAME is: release/2017.2 I get this error:
ERROR: Unable to find project for artifact copy:
playground/frontend/release%2f2017.2 This may be due to incorrect project
name or permission settings; see help for project name in job
configuration. Finished: FAILURE
The configure pipeline looks like this:
node {
stage('Prepare') {
def projectname = "playground/frontend/" + "${BRANCH_NAME}".replace("/", "%2f")
step([$class: 'CopyArtifact', projectName: "${projectname}", selector: [$class: 'StatusBuildSelector', stable: false]])
}
stage('Archive') {
archiveArtifacts '**'
}
}
As you can see I already replace / with %2f (it's needed).
If I don't use the "playground" folder (all my pipelines as is, not inside a folder item), it works. If I use the folder and use the master branch, it works. It doesn't work if I use the folder and a branch name like 2017.2. What am I doing wrong? Can you help making it work? Of well if it's a bug (I searched in https://issues.jenkins-ci.org and found some bugs where a similar setup with folder doesn't work, but they have been fixed... so I really wonder...) in the copy artifact plugin, please file the bug and share the link here, so we can all monitor its progress...
Thank you.
I finally found the issue. The configure pipeline was failing to find a branch with a slash because the encoding was incorrect.
So, in my question, in the configure pipeline:
this (replace / with %2f) is wrong and generates the error:
def projectname = "playground/frontend/" + "${BRANCH_NAME}".replace("/", "%2f")
this is the proper way to encode the slash, and it works:
def projectname = "playground/frontend/" + URLEncoder.encode("${BRANCH_NAME}", "UTF-8").replace("+", "%20")
Credits to: http://www.pipegrep.se/copy-artifacts-from-jenkins-pipeline-jobs.html
UPDATE: actually, I investigated a bit further and added echo "${projectname}" just before step, with the previous and fixed projectname, and I noticed that the difference was %2f lowercase.
Uppercase, like this: %2F works:
def projectname = "playground/frontend/" + "${BRANCH_NAME}".replace("/", "%2F")
So, the fixed configure pipeline looks like this (I kept my replace function, which was enough for my case):
node {
stage('Prepare') {
def projectname = "playground/frontend/" + "${BRANCH_NAME}".replace("/", "%2F")
step([$class: 'CopyArtifact', projectName: "${projectname}", selector: [$class: 'StatusBuildSelector', stable: false]])
}
stage('Archive') {
archiveArtifacts '**'
}
}
I created a sample project to try and recreate what you were seeing, and I was able to do so, after a fashion, except that the build that I was having trouble on was master instead of release/2017.2. Eventually, I realized that I was doing the build job incorrectly from the frontend project, and it was giving me the same error as you because I hadn't ever completed a successful build of the frontend/master branch (I had completed a successful build of the release/2017.2 branch because I didn't have it triggering the configure build initially, so it didn't give me the same error once I did configure it to trigger the configure build).
What worked was changing the build job in the frontend Jenkinsfile to this:
build job: 'playground/configure', parameters: [[$class: 'StringParameterValue', name: 'BRANCH_NAME', value: env.BRANCH_NAME]], quietPeriod: 2, wait: false
Adding in the quietPeriod gives a couple seconds of quiet time between completing the previous job (I'm not certain that this is critical, but it seems like it might be a nice fail-safe, to try and make sure there's enough time for the build to complete), but the important part is the wait: false, which instructs Jenkins that this build shouldn't wait for the triggered build to complete. Once I changed that, the frontend/master branch completed successfully, and the configure build that it triggered also completed successfully.
Hopefully this helps. I was able to get both my master and release/2017.2 branches to build properly, so I don't believe there's any intrinsic problem with the / in the project name. You can see my simple Jenkinsfiles in the referenced repo, and I used the same pipeline script as you posted in your question.
I'm using Jenkins pipeline to build Pull Requests branches using the GitHub Organization plugin.
I can build them fine, but I want to avoid some steps (such as publishing the artifacts). Checking git's current branch gives me the PR's target since the PR branch is being merged into the target before the build is attempted.
How can I check if the build is being initiated from a PR vs than a regular branch build?
At least on Jenkins 2.16 env.BRANCH_NAME gives the source branch not the target branch. You could do something like:
if (env.BRANCH_NAME == "master") {
sh "./publish.sh"
}
Other env vars that could be useful are the CHANGE_* variables. E.g.,
CHANGE_AUTHOR='me'
CHANGE_ID='6'
CHANGE_TARGET='master'
CHANGE_TITLE='Update README.md'
CHANGE_URL='https://github.com/test-org/test-repo/pull/6'
For documentation on these and more: https://ci.eclipse.org/webtools/env-vars.html/
The env variable CHANGE_ID only exists when the build is triggered from a Pull Request check.
For a multibranch project corresponding to some kind of change request, this will be set to the change ID, such as a pull request number, if supported; else unset.
To specifically detect GitHub pull requests, this can be used:
script {
if (env.BRANCH_NAME == 'master') {
sh 'make'
} else if (env.BRANCH_NAME.startsWith('PR')) {
// do actions for pull request
} else {
// some other branch
}
}
Of course, if you expect to have branches starting with PR on your main repository this wouldn't be reliable. The nice thing about it is that script can be used also in post not just stages, which is useful since when is not allowed in post. If you don't care about that it's worth looking into the when directive. There is some documentation from Cloudbees and Jenkins with some examples.