How do I load CPS Global Lib from the specified branch instead of master? - jenkins

I am studying Jenkins Pipeline Global Lib functionality. It seems pretty handy however due to its global nature any harmful change will affect all the jobs. Thus I want to be able to test it before pushing to master on a different branch.
Is there a way to specify a branch from which I want to to include the global lib sources for a particular job?
UPDATE. I tried a workaround with direct git clone from the test branch then load my library file explicitly replacing the automatically loaded one.
The problem comes when this lib uses some other class from the src/. Because in this case its pre-loaded version from master is always used.
So in the conditions below it runs common.groovy from feature-test but prints Hello from master!!!! when b.dummy() is called.
Pipeline Script in Jenkins:
node('myhost'){
git url: 'ssh://10.0.0.1:12345/workflowLibs.git',
branch: 'feature-test'
dir ('src'){
load 'com/foo/Base.groovy'
}
dir ('vars'){
common = load 'common.groovy'
}
}
println common.dummy()
vars/common.groovy (feature-test):
package com.foo
def dummy(){
def b = new com.foo.Base()
b.dummy()
}
src/com/foo/Base.groovy (master):
package com.foo
def dummy(){
return 'Hello from master!!!!'
}
src/com/foo/Base.groovy (feature-test):
package com.foo
def dummy(){
return 'Hello from feature-test!!!!'
}

As far as I'm aware, this isn't possible — scripts pushed to this repo can't be versioned by having different branches.
There are a few alternative approaches you could take, which involve hosting your scripts in an external repo (i.e. not with the Global Lib repo):
You could use the Pipeline Remote Loader plugin, which allows you to pull Pipeline scripts from a remote repo, e.g.
def p = fileLoader.fromGit('bar/common.groovy',
'https://example.com/foo/pipelines.git',
'test-branch', null, '')
p.doSomething()
You can also use this plugin to easily load multiple Pipeline scripts from the same repo.
Alternatively, you could check out your script Git repo within a Pipeline execution and load a script directly:
stage 'Load scripts'
def p
dir('tmp') {
git url: 'https://example.com/foo/pipelines.git',
branch: 'test-branch'
p = load 'bar/common.groovy'
}
stage 'Do something'
p.doSomething()
If you want to keep on using the Global Lib repo, you could use the above techniques for testing the scripts, and then set up a Jenkins job to push your script changes into the master Global Lib repo.

Related

Jenkins: Access job/plugin configuration values inside pipeline

I am trying the access the values set on a job's configuration page from within my pipeline. These values are not made available as params, nor are they injected as envvars.
Setup
Jenkins, v2.263.1
GitLab Branch Source plugin, v1.5.3 (link)
Multibranch pipeline job which is pointed to a Gitlab repo
Remote Jenkinsfile Provider, v1.13 (link)
Problem
Ordinarily, one would have a Jenkinsfile in the root of the repo and therefore the scm would be associated with the repo we want to checkout and build. However, in my case the code I want to build is in a different repo to the Jenkinsfile (hence the Remote Jenkinsfile Provider plugin).
This means that I need to checkout the code I wish to build as an explicit step in the pipeline, and to do that I need to know the repo. This repo is, however, already defined in the job config.
The Branch Source plugin does export things like the branch name or merge request number/branch/target into appropriate envvars, but NOT the actual repo.
As this is a multibranch pipeline, I cannot use something like envInject either (multibranch jobs do not provide the option to 'Prepare an environment for the run' as with other jobs)
Goal
I would like to be able to access the server, owner and project fields set in the job config page. Ultimately I could manage with just the project's ssh/http address even.
Is there some clever way of accessing a job's config from within the pipeline?
Thanks for any suggestions!
Reference images
Within the gitlab branch source plugin (and the documentation) you have a lot more information, than just with the normal branch source plugin. there are environment variables for the project like GITLAB_PROJECT_GIT_SSH_URL/GITLAB_PROJECT_GIT_HTTPS_URL for the git source and many more. So far i did not see one for the server, but that would be parse-able our of the URLs.
Within this information, it should be fairly easy to checkout the repository and build it.
As through the process it came clear, that it is needed to also trigger the pipeline manually, and this is normally also possible with variables (not sure about the Remote File plugin). I assume your Jenkinsfile is a groovy script, which opens up a lot of possibilities. You can define variables and use some logic to determine if the env variable or the parameter is used.
pipeline {
parameters {
string(name: 'projectUrl', defaultValue: "")
}
stages {
stage('Prepare') {
steps {
def projectUrl = env.GITLAB_PROJECT_GIT_SSH_URL ?: params.projectUrl
// DO Checkout with projectUrl
}
}
}
}
The only critical thing you have to take into account, is that the multibranch pipeline, has to run once, for each branch or mr - so they detect the variables. Afterwards you can easily trigger it, manually by providing your values.
This allows you, to utilize webhooks for automatic actions, and also allows you to trigger the build manually when ever you like.
Sidenote: if you use the centralized jenkinsfile, for reducing duplication, you might also want to checkout Shared libraries for jenkins.
For completeness, here is a list of all current environment variables added by the jenkins gitlab branch source plugin version 1.5.3 (and only for Push Events - but they are pretty similar in the other event types too)
GITLAB_OBJECT_KIND
GITLAB_AFTER
GITLAB_BEFORE
GITLAB_REF
GITLAB_CHECKOUT_SHA
GITLAB_USER_ID
GITLAB_USER_NAME
GITLAB_USER_EMAIL
GITLAB_PROJECT_ID
GITLAB_PROJECT_ID_2
GITLAB_PROJECT_NAME
GITLAB_PROJECT_DESCRIPTION
GITLAB_PROJECT_WEB_URL
GITLAB_PROJECT_AVATAR_URL
GITLAB_PROJECT_GIT_SSH_URL
GITLAB_PROJECT_GIT_HTTP_URL
GITLAB_PROJECT_NAMESPACE
GITLAB_PROJECT_VISIBILITY_LEVEL
GITLAB_PROJECT_PATH_NAMESPACE
GITLAB_PROJECT_CI_CONFIG_PATH
GITLAB_PROJECT_DEFAULT_BRANCH
GITLAB_PROJECT_HOMEPAGE
GITLAB_PROJECT_URL
GITLAB_PROJECT_SSH_URL
GITLAB_PROJECT_HTTP_URL
GITLAB_REPO_NAME
GITLAB_REPO_URL
GITLAB_REPO_DESCRIPTION
GITLAB_REPO_HOMEPAGE
GITLAB_REPO_GIT_SSH_URL
GITLAB_REPO_GIT_HTTP_URL
GITLAB_REPO_VISIBILITY_LEVEL
GITLAB_COMMIT_COUNT
GITLAB_COMMIT_ID_#
GITLAB_COMMIT_MESSAGE_#
GITLAB_COMMIT_TIMESTAMP_#
GITLAB_COMMIT_URL_#
GITLAB_COMMIT_AUTHOR_AVATAR_URL_#
GITLAB_COMMIT_AUTHOR_CREATED_AT_#
GITLAB_COMMIT_AUTHOR_EMAIL_#
GITLAB_COMMIT_AUTHOR_ID_#
GITLAB_COMMIT_AUTHOR_NAME_#
GITLAB_COMMIT_AUTHOR_STATE_#
GITLAB_COMMIT_AUTHOR_USERNAME_#
GITLAB_COMMIT_AUTHOR_WEB_URL_#
GITLAB_COMMIT_ADDED_#
GITLAB_COMMIT_MODIFIED_#
GITLAB_COMMIT_REMOVED_#
GITLAB_REQUEST_URL
GITLAB_REQUEST_STRING
GITLAB_REQUEST_TOKEN
GITLAB_REFS_HEAD

groovy script loaded from jenkinsfile not found

currently I have an "all inclusive" jenkinsfile which contains various functions.
In order to re-use those functions in other jenkinsfiles I want to put them into separate groovy scripts and load them from the jenkinsfile(s).
scmHandler.groovy:
#!groovy
def handleCheckout() {
if (env.gitlabMergeRequestId) {
echo 'Merge request detected. Merging...'
}
...
}
return this;
in jenkinsfile I do:
...
def scmHandler = load ("test/scmHandler.groovy")
scmHandler.handleCheckout()
I tried to follow the instructions from here but jenkins is constantly complaining that there is no such file scmHandler.groovy an I get:
java.io.FileNotFoundException: d:\jenkins\workspace\myJenkinsJob\test\scmHandler.groovy
Both jenkinsfile and scmHandler.groovy reside in a test/ subdir of the workspace in the git repo of the project to boild and are checked out correctly on master:
/var/lib/jenkins/jobs/myJenkinsJob/workspace#script/test/scmHandler.groovy
However I cannot find them on the slave node where the jenkinsfile executes the build steps inside a node {}. There I only see old versions of the jenkinsfile since the (separated) checkout step is not executed yet.
How do I correctly access the handleCheckout.groovy? What am I miss here?
Actually I find this a neat way to "include" external groovy files without using a separate library.
Use checkout scm before loading scmHandler.groovy
checkout scm
def scmHandler = load ("test/scmHandler.groovy")
scmHandler.handleCheckout()

How to manage multiple Jenkins pipelines from a single repository?

At this moment we use JJB to compile Jenkins jobs (mostly pipelines already) in order to configure about 700 jobs but JJB2 seems not to scale well to build pipelines and I am looking for a way to drop it from the equation.
Mainly i would like to be able to have all these pipelines stored in a single centralized repository.
Please note that keeping the CI config (Jenkinsfile) inside each repository and branch is not possible in our use case, we need to keep all pipelines in a single "jenkins-jobs.git" repo.
As far as I know this is not possible yet, but in progress. See: https://issues.jenkins-ci.org/browse/JENKINS-43749
I think this is the purpose of jenkins shared libraries
I didn't dev such library my-self but I am using some. Basically:
Develop the "shared code" of the jenkins pipeline in a shared library
it can contains the whole pipeline (seq of steps)
Add this library to the jenkins server
In each project, add a jenkinsfile that "import" those using #Library
as #Juh_ said, you can use jenkins shared libraries, here is a complete steps, Suppose that we have three branches:
master
develop
stage
and we want to create a single Jenkins file so that we can change in only one place. All you need is creating a new branch ex: common. This branch MUST have this structure. What we are interested for now is adding a new groovy file in vars directory, ex: common.groovy. Here we can put the common Jenkins file that you wish to be used across all branches.
Here is a sample:
def call() {
node {
stage("Install Stage from common file") {
if (env.BRANCH_NAME.equals('master')){
echo "npm install from common files master branch"
}
else if(env.BRANCH_NAME.equals('develop')){
echo "npm install from common files develop branch"
}
}
stage("Test") {
echo "npm test from common files"
}
}
}
You must wrap your code call function in order to be used in other branches. now we have finished work in common branch we need to use it in our branches. go to any branch you wish to use this pipline ex: master and create Jenkinsfile and put this one line of code:
common()
This will call the common function that you have created before in common branch and will execute the pipeline.

How to configure a Jenkins 2 Pipeline so that Jenkinsfile uses a predefined variable

I have several projects that use a Jenkinsfile which is practically the same. The only difference is the git project that it has to checkout. This forces me to have one Jenkinsfile per project although they could share the same one:
node{
def mvnHome = tool 'M3'
def artifactId
def pomVersion
stage('Commit Stage'){
echo 'Downloading from Git...'
git branch: 'develop', credentialsId: 'xxx', url: 'https://bitbucket.org/xxx/yyy.git'
echo 'Building project and generating Docker image...'
sh "${mvnHome}/bin/mvn clean install docker:build -DskipTests"
...
Is there a way to preconfigure the git location as a variable during the job creation so I can reuse the same Jenkinsfile?
...
stage('Commit Stage'){
echo 'Downloading from Git...'
git branch: 'develop', credentialsId: 'xxx', url: env.GIT_REPO_LOCATION
...
I know I can set it up this way:
This project is parameterized -> String Parameter -> GIT_REPO_LOCATION, default= http://xxxx, and access it with env.GIT_REPO_LOCATION.
The downside is that the user is promted to start the build with the default value or change it. I would need that it were transparent to he user. Is there a way to do it?
You can use the Pipeline Shared Groovy Library plugin to have a library that all your projects share in a git repository. In the documentation you can read about it in detail.
If you have a lot of Pipelines that are mostly similar, the global variable mechanism provides a handy tool to build a higher-level DSL that captures the similarity. For example, all Jenkins plugins are built and tested in the same way, so we might write a step named buildPlugin:
// vars/buildPlugin.groovy
def call(body) {
// evaluate the body block, and collect configuration into the object
def config = [:]
body.resolveStrategy = Closure.DELEGATE_FIRST
body.delegate = config
body()
// now build, based on the configuration provided
node {
git url: "https://github.com/jenkinsci/${config.name}-plugin.git"
sh "mvn install"
mail to: "...", subject: "${config.name} plugin build", body: "..."
}
}
Assuming the script has either been loaded as a Global Shared Library
or as a Folder-level Shared Library the resulting Jenkinsfile will be
dramatically simpler:
Jenkinsfile (Scripted Pipeline)
buildPlugin {
name = 'git'
}
The example shows how a jenkinsfile passes name = git to the library.
I currently use a similar setup and am very happy with it.
Instead of having a Jenkinsfile in each Git repository, you can have an additional git repository from where you get the common Jenkinsfile - this works when using Pipeline type Job and selecting the option Pipeline script from SCM. This way Jenkins checks out the repo where you have the common Jenkinsfile before checking out the user repo.
In case the job can be triggered automatically, you can create a post-receive hook in each git repo that calls the Jenkins Pipeline with the repo as a parameter, so that the user does not have to manually run the job entering the repo as a parameter (GIT_REPO_LOCATION).
In case the job cannot be triggered automatically, the least annoying method I can think of is having a Choice parameter with a list of repositories instead of a String parameter.

Multi-branch configuration with externally-defined Jenkinsfile

I have an open-source project, that resides in GitHub and is built using a build farm, controlled by Jenkins.
I want to build it branch-wise using a pipeline, but I don't want to store Jenkinsfile inside the code. Is there a way to accomplish this?
I have encountered the same issue as you. While the idea of having the build process as part of the code is good, there is information that the Jenkinsfile would include that are not intrinsic to the project build itself, but rather are specific to the build environment instance, which may change.
The way I accomplished this is:
Encapsulate the core build process in a single script (build.py or build.sh). This may call specific build tools like Make, CMake, Ant, etc.
Tell Jenkins via the Jenkinsfile to call a function defined in a single global library
Define the global Jenkins build function to call the build script (e.g. build.py) with appropriate environment settings. For example, using custom tools and setting up the PATH.
So for step 2, create a Jenkinsfile in your project containing just the line
build_PROJECTNAME()
where PROJECTNAME is based on the name of your project.
Then use the Pipeline Shared Groovy Libraries Plugin and create a Groovy script in the shared library repository called vars/build_PROJECTNAME.groovy containing the code that sets up the environment and calls the project build script (e.g. build.py):
def call() {
node('linux') {
stage("checkout") {
checkout scm
}
stage("build") {
withEnv([
"PATH+CMAKE=${tool 'CMake'}/bin",
"PATH+PYTHON=${tool 'Python-3'}",
"PATH+NINJA=${tool 'Ninja'}",
]) {
execute 'python build.py'
}
}
}
}
First of all, why do you not want a Jenkinsfile in your code? The pipeline is just as much part of the code as would be your build file.
Other then that, you can load groovy files to be evaluated as a pipeline script. You can do this either from a different location with the from SCM option and then checkout the actual code. But this will force you to manually take care of the branch builds.
Another option would be to have a very basic Jenkinsfile that merely checkouts an external pipeline.
You would get something like this:
node{
deleteDir()
git env.flowScm
def flow = load 'pipeline.groovy'
stash includes: '**', name: 'flowFiles'
stage 'Checkout'
checkout scm // short hand for checking out the "from scm repository"
flow.runFlow()
}
Where the pipeline.groovy file would contain the actual pipeline would look like this:
def runFlow() {
// your pipeline code
}
// Has to exit with 'return this;' in order to be used as library
return this;

Resources