We are using the pipeline plugin with multibranch configuration for our CD.
We have checked in the Jenkinsfile which works off git.
git url: "$url",credentialsId:'$credentials'
The job works fine, but does not auto trigger when a change is pushed to github.
I have set up the GIT web hooks correctly.
Interestingly, when I go into a branch of the multibranch job and I click "View Configuration", I see that the "Build when a change is pushed to Github" is unchecked. There is no way to check it since I can not modify the configuration of the job (since it takes from parent) and the same option is not there in parent.
Any ideas how to fix this?
For declarative pipelines try:
pipeline {
...
triggers {
githubPush()
}
...
}
For me this enables the checkbox "GitHub hook trigger for GITScm polling", but polling is not actually required. This requires the GitHub plugin.
I found a way to check the checkbox "Build when a change is pushed to Github".
This line did the trick:
properties([pipelineTriggers([[$class: 'GitHubPushTrigger'], pollSCM('H/15 * * * *')])])
I think the polling is needed to make it work. Would be nice if no polling is needed.
Here's a Jenkinsfile example with this implemented:
#!/usr/bin/env groovy
node ('master'){
stage('Build and Test') {
properties([pipelineTriggers([[$class: 'GitHubPushTrigger'], pollSCM('H/15 * * * *')])])
checkout scm
env.PATH = "${tool 'Maven 3'}/bin:${env.PATH}"
sh 'mvn clean package'
}
}
For declarative pipelines, try this:
pipeline {
agent any
triggers {
pollSCM('') //Empty quotes tells it to build on a push
}
}
If you use Stash for example you can register a Post-Receive WebHook where you have to insert your URL form Jenkins like : http://jenkinsHost:9090/git/notifyCommit?url=ssh://git#gitHost:1234/test.git
In your jenkins Job you have to set at least the Build trigger "Poll SCM".
And set a polling time of e.g 5 mins.
This enables also the automatic branch indexing for your multibranch project configuration.
Resorting to polling adds latency - time that it takes for a build to start and hence finish giving back a result.
It seemed to me that the basic plugins have a low level of abstraction, so I switched to the Github Organization Folder plugin, which depends on all of them and sets up an organization hook for triggering builds branches and/or pull requests.
Before I start, I would like to emphasize that I had no previous experience with Jenkins so far, so there might be a bunch of better solutions out there.
What I wanted to achieve in a nutshell:
After every push made to a Bitbucket repo(test2), on every branch,
pull and build another Bitbucket repo(test1), from an identical
branch name and right after that, build test2 using test1 as a
dependency.
How I managed to achieve that?
I started a new job with type 'Multibranch Pipeline'
I added the following Jenkinsfile to test2:
pipeline {
agent any
stages {
stage('build') {
steps {
dir('test1') {
git branch: BRANCH_NAME, url: 'git#bitbucket.org:user/test1.git', credentialsId: 'credentials_id'
}
sh('build_process')
}
}
}
}
I come across the issue that you can't set up a Bitbucket hook for pipelines
I added Bitbucket Branch Source Plugin to Jenkins
I selected Bitbucket at 'Branch Sources' when setting up the job
I added credentials and put a checkmark to Auto-register webhook
Under 'Scan Multibranch Pipeline Triggers' I put a checkmark to Periodically if not otherwise run, with an interval of 1 min
I added a webhook to my Bitbucket repo
I updated all my plugins, restarted Jenkins and it's ready to go
Other plugins I have installed: Bitbucket Plugin, Pipeline plugin. Hope this helps for somebody, I did manage to solve it this way after hours of struggling without the Bitbucket Branch Source Plugin.
node{
stage('Build and Test') {
properties([pipelineTriggers([[$class: 'GitHubPushTrigger'], pollSCM('* * * * *')])])
checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'xxx-xxx-xxxx[your credentails Id]', url: 'https://github.com/git']]])
echo 'Build and Test has been done'
}
}
Related
My project has 3 submodules in GitLab, which are all needed to build my project. I want to create independent pipelines in Jenkins to monitor and pull when a merge request is open.
If I create individual pipelines, Jenkins will create a new folder with the name of the pipeline project like so: "jenkins_home/workspace/submodule1", "jenkins_home/workspace/submodule2", "jenkins_home/workspace/submodule3".
Is it possible to specify the directory where I want to checkout each submodule? As in, checkout all into "jenkins_home/workspace/common_folder", where common_folder will contain submodule1, submodule2 and submodule3.
P.S. I tried bat 'cd common_folder', but the cd command just hangs and never executes.
Also tried dir (**subdir**){} which just creats a new directory inside the submodule pipeline directory: "jenkins_home/workspace/submodule1/subdir/code_from_git".
#!/usr/bin/env groovy
pipeline {
agent { label 'master' }
environment {
gbuild = 'true'
DB_ENGINE = 'sqlite'
}
options{
skipDefaultCheckout()
}
stages {
stage('Checkout') {
steps {
script {
checkout([
HERE, need to checkout into a custom folder and not the workspace
$class: 'GitSCM',
branches: scm.branches,
extensions: scm.extensions + [
[$class: 'GitLFSPull'],
[$class: 'CleanCheckout']
],
userRemoteConfigs: scm.userRemoteConfigs
])
}
}
}
I believe what you are doing dir is the correct approach or you can create separate pipelines.
Jenkins works on master slave configuration and the pipeline you create creates the same name folder in workspace on master server which is then created to slave server when you run the pipelines for the first time, once the pipeline runs and checkout the code on slave server it is then pushed to your master server.
I hope the answer explained you the working principle.
A possible workaround for projects with subprojects, where you want to track each subproject for any merge requests and need all the subprojects to build is: use an independent pipeline.
Additional comment: as there is no admin access on the server pc this might be limiting my capabilities to execute some simple commands, this solution might not be correct for you.
As my cmd commands in the pipeline were not executing and keeping the whole pipeline from running, and I was not able to change the location of the project from workspace to a desired location, I created 2 extra pipelines.
First pipeline is there to listen to webhooks from GitLab and pull the branch in the merge request (also verifying if its a merge request, if so its going to take the branch being merged, if not it will take the master branch):
stage('Checkout'){
steps{
script{
if(env.gitlabActionType == 'Merge')
{
checkout([
$class: 'GitSCM'
branches: [[name: "${env.gitlabSourceBranch}"]]
])
}
else
{
checkout([
$class: 'GitSCM'
branches: master
])
}
}
}
}
Second pipeline to copy the checkouted files into the desired location. For this step I made a freestyle project, where i execute a windows batch command to xcopy CheckedoutDir DesiredDestination.
The second pipeline has Build Trigger to build after first pipeline is built stable. It also has a Trigger/call builds on other projects to trigger the main pipeline that does the building and unit testing.
Background
My team wants to update several linting rules in our project, however, doing so will cause our Jenkins build pipeline which lints, tests and builds each feature branch to break. We don't want to lose the value of linting each feature branch before merging, so we agree that linting only the files that the feature branch changes is a reasonable way to introduce these new lint rules without forcing us to re-lint the whole project up-front. Given that our entire project is already linted, this seems like a reasonable move.
A while ago I wrote a git tool to do exactly this. It determines which files have changed since the feature branch diverged from master and outputs those files so they can be consumed by eslint, pycodestyle and other linters. Here's the source if you're interested in how this is done.
Problem
Jenkins declarative build process and it's GitHub Branch Source Plugin seem to have a brittle checkout behavior that can't be modified to checkout more than just the feature branch that it's called on to build.
If I call git fetch origin stage within a build step, Jenkins complains about missing credentials. I don't feel comfortable sticking in credentials into my pipeline file, I'd MUCH prefer to continue using the Git plugin to manage credentials to our private github repo and pull branches, however, I'm at a loss as to how to specify for it to fetch more than just the feature branch.
For reference, here's the relevant portions of my Jenkinsfile
As you can see, I've tried adding the GitSCM code block to no avail. I've read this medium article which solves a similar problem, but I'm not using SSH credentials and I'd prefer not to given than we're already managing credentials using the Git plugin.
pipeline {
agent any
tools {
nodejs 'node12.7.0'
}
stages {
stage('checkout') {
steps {
checkout([
$class: 'GitSCM',
branches: [[name: '*']],
extensions: scm.extensions,
userRemoteConfigs: [],
doGenerateSubmoduleConfigurations: true
])
}
}
stage('install') {
steps {
script {
sh 'git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master'
sh 'yarn install'
}
}
}
stage('lint & test') {
failFast true
parallel {
stage('lint') {
when {
not {
anyOf {
branch 'stage'; branch 'int'; branch 'prod'
}
}
}
steps {
script {
sh """
git submodule update --init
yarn run lint
"""
}
}
}
...
}
}
stage('deploy') {...}
}
}
post {
failure {
notifySlack()
}
}
}
Create credential in your Jenkins with ssh key and private key, which can be added to the checkout userRemoteConfigs which will be used while checking out (Value given down just an example of one the credential id in my Jenkins environment)
userRemoteConfigs: [[credentialsId: '7969s7612-adruj-au2cd-492msa802f']]
One frequent root cause - mentioned on the referenced medium article, too - is that Jenkins only checks out the current branch that needs to be build.
An easy option I just found to have other project branches available is to
Configure your pipeline job
Under Behaviors->General, add Specify ref specs
Optionally adjust the parameter to the refs you need, e.g. the branches to compare to. Or you can get all branches by maintaining the default +refs/heads/*:refs/remotes/#{remote}/* as shown in the screenshot:
Jenkins Pipeline Job - Ref Config
P.S.: This seems to be part of the GIT Jenkins plugin, but I couldn't find it in the docs...
Consider the following setup using Jenkins 2.176.1:
A new pipeline project named Foobar
Poll SCM as (only) build trigger, with: H/5 * * * * ... under the assumption that this refers to the SCM configured in the next step
Pipeline script from SCM with SCM Git and a working Git repository URL
Uncheck Lightweight checkout because of JENKINS-42971 and JENKINS-48431 (I am using build variables in the real project and Jenkinsfile; also this may affect how pollSCM works, so I include this step here)
Said repository contains a simple Jenkinsfile
The Jenkinsfile looks approximately like this:
#!groovy
pipeline {
agent any
triggers { pollSCM 'H/5 * * * *' }
stages {
stage('Source checkout') {
steps {
checkout(
[
$class: 'GitSCM',
branches: [],
browser: [],
doGenerateSubmoduleConfigurations: false,
extensions: [],
submoduleCfg: [],
userRemoteConfigs: [
[
url: 'git://server/project.git'
]
]
]
)
stash 'source'
}
}
stage('OS-specific binaries') {
parallel {
stage('Linux') {
agent { label 'gcc && linux' }
steps {
unstash 'source'
echo 'Pretending to do a build here'
}
}
stage('Windows') {
agent { label 'windows' }
steps {
unstash 'source'
echo 'Pretending to do a build here'
}
}
}
}
}
}
My understanding so far was that:
a change to the Jenkinsfile (not the whole repo) triggers the pipeline on any registered agent (or as configured in the pipeline project).
said agent (which is random) uses the pollSCM trigger in the Jenkinsfile to trigger the pipeline stages.
But where does the pollSCM trigger poll (what SCM repo)? And if it's a random agent then how can it reasonably detect changes across poll runs?
then the stages are being executed on the agents as allocated ...
Now I am confused what refers to what. So here my questions (all interrelated which is why I keep it together in one question):
The pipeline project polls the SCM just for the Jenkinsfile or for any changes? The repository in my case is the same (for Jenkinsfile and source files to build binaries from).
If the (project-level) polling triggers at any change rather than changes to the Jenkinsfile
Does the pollSCM trigger in the Jenkinsfile somehow automagically refer to the checkout step?
Then ... what would happen, would I have multiple checkout steps with differing settings?
What determines what repository (and what contents inside of that) gets polled?
... or is this akin to the checkout scm shorthand and pollSCM actually refers to the SCM configured in the pipeline project and so I can shorten the checkout() to checkout scm in the steps?
Unfortunately the user handbook didn't answer any of those questions and pollSCM has a total of four occurrences on a single page within the entire handbook.
I'll take a crack at this one:
The pipeline project polls the SCM just for the Jenkinsfile or for any
changes? The repository in my case is the same (for Jenkinsfile and
source files to build binaries from).
The pipeline project will poll the repo for ANY file changes, not just the Jenkinsfile. A Jenkinsfile in the source repo is common practice.
If the (project-level) polling triggers at any change rather than
changes to the Jenkinsfile Does the pollSCM trigger in the Jenkinsfile
somehow automagically refer to the checkout step?
Your pipeline will be executed when a change to the repo is seen, and the steps are run in the order that they appear in your Jenkinsfile.
Then ... what would happen, would I have multiple checkout steps with
differing settings?
If you defined multiple repos with the checkout step (using multiple checkout SCM calls) then the main pipeline project repo would be polled for any changes and the repos you define in the pipeline would be checked out regardless of whether they changed or not.
What determines what repository (and what contents inside of that)
gets polled? ... or is this akin to the checkout scm shorthand and
pollSCM actually refers to the SCM configured in the pipeline project
and so I can shorten the checkout() to checkout scm in the steps?
pollSCM refers to the pipeline project's repo. The entire repo is cloned unless the project is otherwise configured (shallow clone, lightweight checkout, etc.).
The trigger defined as pollSCM polls the source-control-management (SCM), at the repository and branch in which this jenkinsfile itself (and other code) is located.
For Pipelines which are integrated with a source such as GitHub or BitBucket, triggers may not be necessary as webhooks-based integration will likely already be present. The triggers currently available are cron, pollSCM and upstream.
It works for a multibranch-pipeline as trigger to execute the pipeline.
When Jenkins polls the SCM, exactly this repository and branch, and detects a change (i.e. new commit), then this Pipeline (defined in jenkinsfile) is executed.
Usually then the following SCM Step checkout will be executed, so that the specified project(s) can be built, tested and deployed.
See also:
SCM Poll in jenkins multibranch pipeline
SehllHacks(2020): Jenkins: Scan Multibranch Pipeline Without Build
I'm setting up Jenkins pipeline for my .Net Core application.
Jenkins multibranch pipeline build gets trigger on Git commit if I am configuring the checkout SCM in multibranch Pipeline configuration. But multibranch Pipeline build is not getting trigger on git commit if I checkout SCM explicitly inside Jenkins Declarative Pipeline script.
Is there any way to resolve this issue?
Below is the checkout command which I am using inside the script:
checkout([$class: 'GitSCM', branches: [], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CleanCheckout'], [$class: 'PruneStaleBranch']], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'credential-id', url: 'my/git/ssh/url']]])
To get the build triggered on new git commit you should first enable SCM polling in your pipeline script by adding proper triggers directive to your Jenkinsfile:
triggers {
pollSCM 'H/2 * * * *'
}
This will poll your SCM for any changes every two minutes. If a change is detected since the last build your job will be triggered to build the changes.
Polling is the easiest way to get what you want. However, you should consider using post-commit hook instead of polling. Using polling Jenkins has to periodically check SCM for the changes. In case of post-commit hook, Jenkins will be notified about the changes by SCM if necessary. It's prefered over polling because it will relieve the number of required requests and the traffic from Jenkins towards SCM repository.
After configuring post-commit hook, the triggers directive should be modified by providing an empty string as cron parameter to pollSCM trigger.
triggers {
pollSCM ''
}
It can be confusing but this empty string is required to enable post-commit hooks requests to be handled by the job. It's also not very well documented by Jenkins docs.
I am using Jenkins 2.89.2 and my project has Jenkinsfile which defines all build pipeline and steps. I have searched online for triggering build when push to github repo and all of them mentioned an options Build when a change was pushed to Github in jenkins configuration page. But in my jenkins configuration page I couldn't find this options. Below is an screenshot. It only has one options Periodically if not otherwise run. I have installed github related plugins but still I couldn't find that option. Is there any other configuration I can change?
Below is my Jenkinsfile:
pipeline {
agent {
label 'master'
}
tools {
maven 'maven-3.5.2'
jdk 'jdk9'
}
stages {
stage ('Checkout SCM') {
steps {
echo 'Checkout from Git...'
checkout scm
}
}
stage ('Build') {
steps {
echo 'Building '
}
}
}
}
After some searching I think I find the solution. The trigger option can be defined in jenkinsfile as below:
pipelineTriggers([
[$class: "GitHubPushTrigger"]
])
You basically have to create a HOOK from jenkins to gitHub, which is located under the Build Triggers option
This option should be available to you since you have the Git Plugin installed.
P.S. the solution you have mentioned will work only for Declarative Pipeline and will not work if you plan to switch to scripted pipeline