Jenkins pipeline polling multiple scm issue - jenkins

I have a Jenkins pipeline script, which is supposed get 2 repositories from GitLab (triggered by Gitlab webhook), if there are changes on "master" branch in any of the repositories (and so more stuff afterwards).
My job DSL config:
pipelineJob('pipline') {
// Enable but do not use SCM polling. This needs to be enabled for
// notifyCommit to work but we don't want to do any actual polling.
configure { project ->
project / 'triggers' << 'hudson.triggers.SCMTrigger' {
spec('')
}
}
definition {
cps {
sandbox(true)
script ('''node('Foo') {
//repo abc
checkout changelog: false,
poll: true,
scm: [$class: 'GitSCM',
branches: [[name: 'refs/heads/master']],
doGenerateSubmoduleConfigurations: false,
extensions: [
[$class: 'SubmoduleOption',
disableSubmodules: true,
parentCredentials: false,
recursiveSubmodules: false,
reference: '',
trackingSubmodules: false],
[$class: 'RelativeTargetDirectory',
relativeTargetDir: "abc"]
],
submoduleCfg: [],
userRemoteConfigs: [
[url: '<repository abc>'],
[refspec: '+refs/heads/master:refs/remotes/origin/master'],
[name: 'origin']
]]
//repo def
checkout changelog: false,
poll: true,
scm: [$class: 'GitSCM',
branches: [[name: 'refs/heads/master']],
doGenerateSubmoduleConfigurations: false,
extensions: [
[$class: 'CheckoutOption', timeout: 20],
[$class: 'SubmoduleOption',
disableSubmodules: true,
parentCredentials: false,
recursiveSubmodules: false,
reference: '',
trackingSubmodules: false],
[$class: 'RelativeTargetDirectory',
relativeTargetDir: "def"],
[$class: 'PruneStaleBranch']],
submoduleCfg: [],
userRemoteConfigs: [
[url: '<repository def>'],
[refspec: '+refs/heads/master:refs/remotes/origin/master'],
[name: 'origin']
]]
...
do more things
''')
}
}
}
The polling working fine for the first repository, but when I have changes in a seconds repository not in a master branch, I still get the whole thing started. It looks that it takes last build revision SHA only from the first repository and then uses it to compare in all other repositories.
here's how the git polling log looks like:
Started on Feb 15, 2017 12:10:09 PM
Using strategy: Default
[poll] Last Built Revision: Revision <SHA from last commit in repository abc on master branch > (refs/remotes/origin1/master)
> git ls-remote -h <repository abc> # timeout=10
Found 4 remote heads on <repository abc>
[poll] Latest remote head revision on refs/heads/master is: <SHA from last commit in repository abc on master branch > - already built by 76
Using strategy: Default
[poll] Last Built Revision: Revision <SHA from last commit in repository abc on master branch > (refs/remotes/origin1/master)
> git ls-remote -h <repository def> # timeout=10
Found 2 remote heads on <repository def>
[poll] Latest remote head revision on refs/heads/master is: <SHA from last commit in repository def on master branch >
Done. Took 1.1 sec
Changes found
I was using buildFlowJob before with same functionality and it worked fine.

Related

Jenkins checkout scm step

I'm new to Jenkins and I'm trying to understand the following step in Jenkins pipeline line by line:
checkout scm
dir("some_directory") {
checkout(
changelog: false,
poll: false,
scm: [
$class : 'GitSCM',
branches : [[name: SOME_BRANCH_NAME]],
doGenerateSubmoduleConfigurations: false,
extensions : [[$class: 'CloneOption', depth: 0, honorRefspec: true, reference: '', shallow: false]],
submoduleCfg : [],
userRemoteConfigs : [[url: SOME_URL]]
]
)
sh 'pwd; ls'
}
From the research I've done I understood that
checkout scm dir("some_directory")
'dir' creates a folder in workspace if it doesn't exist, and git project gets checked out into this directory
checkout(
changelog:false,
poll:false,
scm: [...]
)
this block of code specifies git parameters of the git project that is being checked out into the directory specified above.
This is so far what I understood, can someone please lt me know if my understanding is correct? And possibly add more details to it.
Also, I am confused with the current code syntax. Would it make any difference if I rewrite the top few lines as:
checkout scm dir("some_directory")(
changelog: false,
poll: false,
etc.
)
instead of using 'checkout' two times.

How to write 'Git LFS pull after checkout' settings in the Jenkins pipeline

I created a configuration file with Jenkins file to configure the Jenkins pipeline.
An error occurred when doing git pull.
The cause was that the groovy file did not have the Git LFS pull after checkout setting.
I do not know how to write Git LFS pull after checkout setting to groovy.
git(
 url: git#...,
 branch: "master",
 credentialsId:"abcdefg"
)
// Git LFS pull after checkout setting??
Here's how I was able to use the Git plugin in the pipeline. Refer to the documentation here for more info:
checkout([ $class: "GitSCM",
branches: [[name: "refs/heads/${your branch name}"]],
extensions: [
[$class: "GitLFSPull"]
],
userRemoteConfigs: [
[credentialsId: "${your git credential ID}",
url: "${your git URL}"]
]
])
Considering you are trying to download a large file, you may want to increase the time out limit too (default is set to 10 minutes):
checkout([ $class: 'GitSCM',
branches: [[name: 'refs/heads/'+"${branch_or_tag}"]],
extensions: [[$class: 'GitLFSPull']]
+[[$class: 'CloneOption', timeout: 30]],
userRemoteConfigs: [
[credentialsId: "${your git credential ID}",
url: "${your git URL}"]
]
])

How to set up a pipeline for specific branch

I have a Java project in http://localhost:7990/scm/bout/boutique-a.git
I want to have 2 Jenkins pipeline jobs:
Job 1/ trigger on commit done on */develop
Job 2/ trigger on commit done on any */feature
p
each job will do a basic mvn install, mvn test, sonar ...
a simple script with
node {
checkout([$class: 'GitSCM',
branches: [[name: 'develop]],
doGenerateSubmoduleConfigurations: false,
extensions: [[$class: 'SubmoduleOption', disableSubmodules: false,
parentCredentials: false, recursiveSubmodules: true, reference: '',
trackingSubmodules: false]], submoduleCfg: [],
userRemoteConfigs: [[credentialsId: 'admin',
url: 'http://localhost:7990/scm/bout/boutique-a.git']]])
}
works if a commit is done in /develop or if I give explicitly the branch name like feature/test-a but how to configure a script for any feature/
It seems that what i'm asking is not possible using pipeline job.
I found a work arround for "feature/** ". I created a param BRANCH_NAME in the job, then the branch name is send by bitbucket when a push is made on "feature/** " through a basic POST request.
http://user:token#localhost:8081/jenkins/job/MY_JOB_NAME/buildWithParameters?token=U1C1yQo7x3&BRANCH_NAME=feature/branche-test

Checkout from Gitlab is getting fail in Jenkins pipeline

I have read lots of documentation but not able to fix this issue.
I used below code for checkout from gitlab in jenkinfile
checkout changelog: true, poll: true, scm: [
$class: 'GitSCM',
branches: [[name: "origin/${env.gitlabSourceBranch}"]],
doGenerateSubmoduleConfigurations: false,
extensions: [[
$class: 'PreBuildMerge',
options: [
fastForwardMode: 'FF',
mergeRemote: 'origin',
mergeStrategy: 'default',
mergeTarget: "${env.gitlabTargetBranch}"
]
]],
submoduleCfg: [],
userRemoteConfigs: [[
credentialsId: 'gitlab-jenkins-user-credentials',
name: 'origin',
url: "${env.gitlabSourceRepoHttpUrl}"
]]
]
I even tried
checkout changelog: false, poll: false, scm: [$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'gitlab-jenkins-user-credentials', url: 'http://jenkins-master/testproject/devops_artifacts/test_devops.git']]]
But I am keep getting this error:
Warning: CredentialId "gitlab-jenkins-user-credentials" could not be found.
Cloning the remote Git repository
Cloning repository http://gitlab-master.com/testproject/devops_artifacts/test_devops.git
> C:\Program Files\Git\cmd\git.exe init C:\jenkins-docker\workspace\wildfly_gitlab # timeout=10
Fetching upstream changes from http://gitlab-master.com/testproject/devops_artifacts/test_devops.git
> C:\Program Files\Git\cmd\git.exe --version # timeout=10
> C:\Program Files\Git\cmd\git.exe fetch --tags --force --progress http://gitlab-master.com/testproject/devops_artifacts/test_devops.git +refs/heads/*:refs/remotes/origin/*
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "C:\Program Files\Git\cmd\git.exe fetch --tags --force --progress http://gitlab-master.com/testproject/devops_artifacts/test_devops.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout:
stderr: Logon failed, use ctrl+c to cancel basic credential prompt.
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'http://gitlab-master.com/testproject/devops_artifacts/test_devops.git/'
I have set webhook in this form
http://username:c4cc893d00cfc8865fc3#10.50.9.XXX:8080/project/wildfly_gitlab
And I tested the connection it is working. Same way I configured Gitlab in Jenkins as well
And connection is working as well. I don't know why I am still getting the error.
I assume your "gitlab-jenkins-user-credentials" is a user credential (credential scoped to a specific user).
You can circumvent by getting the user credential with withCredentials and then using the https with credentials url in the userRemoteConfigs config:
parameters {
credentials(credentialType: 'com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl', defaultValue: '', description: 'Jenkins wants to use the "gitlab-jenkins-user-credentials" credentials. Select your credential to allow this.', name: 'gitlab-jenkins-user-credentials', required: true)
}
...
withCredentials([usernamePassword(credentialsId: "gitlab-jenkins-user-credentials", passwordVariable: 'GITLAB_PASSWORD', usernameVariable: 'GITLAB_USER')]) {
checkout([$class: 'GitSCM', branches: [[name: 'refs/tags/mytag']], userRemoteConfigs: [[url: "https://${GITLAB_USER}:${GITLAB_PASSWORD}#gitlab.com/myproject.git"]]])
}
gitlab-jenkins-user-credentials is a user credential of type " Username with password" credential created with:
id: gitlab-jenkins-user-credentials
username = gitlab token name
password = gitlab token
When running the job, the user will be prompted to select it's user credential.
However note that groovy interpolation of secrets is not a good practice on a security point of view (see https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#string-interpolation).

Running Jenkins job after merge request in Gitlab

I'm trying to use a Gitlab web hook to trigger a job in Jenknis after pushing a commit/opening a merge commit using a pipeline script.
For some reason, Jenkins always checks out the master branch and builds it. How
can I specify which branch to build using the Groovy script?
I tried to use the environment variable from the Gitlab POST request, but it still always uses the master branch:
checkout changelog: false, poll: false, scm: [$class: 'GitSCM' , branches: [[name:'origin/${env.gitlabSourceBranch}']], browser: [$class 'GitLab', repoUrl: 'some-git-repo.com', version: 9.0], doGenerateSubmoduleConfiguration: false, extensions: [[$class: 'SubmoduleOption' disableSubmodules: false, parentCredentials: true, recursiveCredentials: true, recursiveSubmodules: true, reference: '', trackingSubmodules: false], [$class: 'PrebuildMerge', options: [fastForwardMode: 'FF', mergeRemote: '', mergeTarget: 'origin/${env.gitlabTargetBranch}']]], submodulecfg: [], userRemoteConfigs: [[credentialsId: '12345', url: 'git#some-git-repo.com:A/repo.git']]]
(I generated this command using the snippet generator)
You can use the "Generic Webhook trigger" plugin in Jenkins.
It allows you to get the branch name from the POST request sent by Gitlab, map it as a variable and use it inside your pipeline.
Configuration:
Inside the pipeline:
stage('Clone sources') {
steps {
git credentialsId: '...', poll: false, branch: ${branch}, url: '...'
}
}
Full example at: https://pillsfromtheweb.blogspot.com/2021/10/trigger-jenkins-on-merge-request-from.html

Resources