Gitlab to Jenkins webhook with parameters - jenkins

I'm trying to build a job on my jenkins using Gitlab webhook when pushing into a specific branch.
The only issue I have is that the jenkins job is parameterized. The parameter is the branch name.
Is there an option to add the branch name (its a different branch every time) to the webhook so it will be inserted automatically into the parameter in the jenkins job parameter?
Thanks

GitLab invokes the webhook URL with a JSON payload in the request body that carries a lot of information about the GitLab event that led to the webhook invocation. The JSON payload for the GitLab push event does have branch name information in it -- see the "ref" field in the GitLab webhook push event payload:
{
"object_kind": "push",
"before": "95790bf891e76fee5e1747ab589903a6a1f80f22",
"after": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"ref": "refs/heads/master",
"checkout_sha": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"user_id": 4,
"user_name": "John Smith",
"user_username": "jsmith",
"user_email": "john#example.com",
"user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80",
"project_id": 15,
"project":{
"id": 15,
"name":"Diaspora",
"description":"",
"web_url":"http://example.com/mike/diaspora",
"avatar_url":null,
"git_ssh_url":"git#example.com:mike/diaspora.git",
"git_http_url":"http://example.com/mike/diaspora.git",
"namespace":"Mike",
"visibility_level":0,
"path_with_namespace":"mike/diaspora",
"default_branch":"master",
"homepage":"http://example.com/mike/diaspora",
"url":"git#example.com:mike/diaspora.git",
"ssh_url":"git#example.com:mike/diaspora.git",
"http_url":"http://example.com/mike/diaspora.git"
},
"repository":{
"name": "Diaspora",
"url": "git#example.com:mike/diaspora.git",
"description": "",
"homepage": "http://example.com/mike/diaspora",
"git_http_url":"http://example.com/mike/diaspora.git",
"git_ssh_url":"git#example.com:mike/diaspora.git",
"visibility_level":0
},
"commits": [
{
"id": "b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327",
"message": "Update Catalan translation to e38cb41.",
"timestamp": "2011-12-12T14:27:31+02:00",
"url": "http://example.com/mike/diaspora/commit/b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327",
"author": {
"name": "Jordi Mallach",
"email": "jordi#softcatala.org"
},
"added": ["CHANGELOG"],
"modified": ["app/controller/application.rb"],
"removed": []
},
{
"id": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"message": "fixed readme",
"timestamp": "2012-01-03T23:36:29+02:00",
"url": "http://example.com/mike/diaspora/commit/da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"author": {
"name": "GitLab dev user",
"email": "gitlabdev#dv6700.(none)"
},
"added": ["CHANGELOG"],
"modified": ["app/controller/application.rb"],
"removed": []
}
],
"total_commits_count": 4
}
You may not bind any field in the webhook payload to a Jenkins job parameter automatically. However, there is still a way to access webhook payload information in the Jenkins job. The Jenkins GitLab plugin makes this webhook payload information available in the Jenkins Global Variable env. The available env variables are as follows and they do include branch information:
gitlabBranch
gitlabSourceBranch
gitlabActionType
gitlabUserName
gitlabUserEmail
gitlabSourceRepoHomepage
gitlabSourceRepoName
gitlabSourceNamespace
gitlabSourceRepoURL
gitlabSourceRepoSshUrl
gitlabSourceRepoHttpUrl
gitlabMergeRequestTitle
gitlabMergeRequestDescription
gitlabMergeRequestId
gitlabMergeRequestIid
gitlabMergeRequestState
gitlabMergedByUser
gitlabMergeRequestAssignee
gitlabMergeRequestLastCommit
gitlabMergeRequestTargetProjectId
gitlabTargetBranch
gitlabTargetRepoName
gitlabTargetNamespace
gitlabTargetRepoSshUrl
gitlabTargetRepoHttpUrl
gitlabBefore
gitlabAfter
gitlabTriggerPhrase
Just as you would read Jenkins job parameters from Jenkins Global Variable params in your job pipeline script, you could read webhook payload fields from Jenkins Global Variable env:
params.MY_PARAM_NAME
env.gitlabBranch
Hope, the above information helps solve your problem.

You can do that with the Generic Webhook Trigger plugin. There is also an example of using that plugin with GitLab in the Violation Comments to GitLab page.
The source branch is resolved with the following JSONPath:
$.object_attributes.source_branch

Related

Bitbucket cloud webhook Does not trigger Jenkins Build: No branch information

I am trying to integrate Bitbucket cloud with Jenkins. I have Bitbucket plugin installed in Jenkins. I have configured the webhook in bitbucket cloud for the repository with url http://<jenkins_url>/bitbucket-hook/
I set up Jenkins job (I tried both pipeline and multibranch pipeline). In the Pipeline I have selected "Build when a change is pushed to BitBucket" and In the pipeline I have stages to fetch from bitbucket and build. I have specified to get from branch develop.
case 1: After I set this up I pushed some code to my repository(branch develop). I expected the pipeline to be triggered(since I have a webhook setup) nothing happens.
I triggered the build manually. It builds the project. Now if I push something to the repository, the build is getting triggered. Question here is Why didn't it triggered the first time and works only after a manual build.
case 2: I setup another pipeline. Exactly as the one before. But with branch as master. I tried to follow the same step(triggered manually first). Now when i pushed something to master. Nothing happens(even after manual build).
When I analysed the jenkins log. I see that there is a notification from bitbucket cloud as soon as something is pushed. But the json which is sent doesn't has branch information. May be that is the reason Jenkins is not able to decide which pipeline to trigger when there are multiple pipelines talking to same repository. Is my assumption correct? Is it a bug in Bitbucket cloud
The json recieved from bitbucket cloud looks like this
{
"scm": "git",
"website": null,
"name": "myproject-api",
"links": {
"self": {
"href": "https://api.bitbucket.org/2.0/repositories/my-team/myproject-api"
},
"html": {
"href": "https://bitbucket.org/my-team/myproject-api"
},
"avatar": {
"href": "https://bytebucket.org/ravatar/%7Bb07e7384-c622-44b1-8c5f-96104d36cfae%7D?ts=java"
}
},
"project": {
"key": "PB",
"type": "project",
"uuid": "{d9c0caa2-7d43-4a0e-8459-2c7c8c2bd287}",
"links": {
"self": {
"href": "https://api.bitbucket.org/2.0/teams/my-team/projects/PB"
},
"html": {
"href": "https://bitbucket.org/account/user/my-team/projects/PB"
},
"avatar": {
"href": "https://bitbucket.org/account/user/my-team/projects/PB/avatar/32"
}
},
"name": "MyProject Backend"
},
"full_name": "my-team/myproject-api",
"owner": {
"username": "my-team",
"display_name": "my-team",
"type": "team",
"uuid": "{848f7612-ffbc-4dfb-8f67-ade4b604761c}",
"links": {
"self": {
"href": "https://api.bitbucket.org/2.0/teams/%7B848f7612-ffbc-4dfb-8f67-ade4b604761c%7D"
},
"html": {
"href": "https://bitbucket.org/%7B848f7612-ffbc-4dfb-8f67-ade4b604761c%7D/"
},
"avatar": {
"href": "https://bitbucket.org/account/my-team/avatar/"
}
}
},
"type": "repository",
"is_private": true,
"uuid": "{b07e7384-c622-44b1-8c5f-96104d36cfae}"
}
Does anyone have any idea how to successfully get webhooks working with bitbucket cloud and Jenkins. Most of the questions on SO regarding this talks about Bitbucket server and not Bitbucket cloud.
I had similar issues solved by doing following steps:
Uninstall old 'Bitbucket' plugin
Removed webhook at bitbucket that was set as:
http://<Jenkins URL>/bitbucket-hook/
Install 'Bitbucket Branch Source Plugin'
Set new webhook at bitbucket repository as:
http://<Jenkins URL>/bitbucket-scmsource-hook/notify/
Cheers!!!

Pipeline stage ID doesn't follow the stage when reordered

I'm using the Jenkins pipeline stage view plugin REST API. When I use the path /job/:job-name/wfapi/runs I get a list of the last jobs.
Yesterday I renamed all the stages and also changed the order of the stages a bit. Here is a part of the JSON file, #152 is before the change and #153 is after:
"name": "#153",
"id": "6",
"name": "Compile",
"id": "13",
"name": "Flash image",
"id": "19",
"name": "Generate log",
...
"name": "#152",
"id": "6",
"name": "1. Compile",
"id": "13",
"name": "2. Flash image",
"id": "19",
"name": "3. Prepare hardware",
...
"id": "31",
"name": "6. Generate log",
What I feel is very strange is that the "Generate log" stage in #153 has taken over the stage id 19 from "3. Prepare hardware" in #152 ("Generate log" failed so I don't have any id number for "Prepare hardware" that comes later in the pipeline). I thought that the id was unique for a stage setup and that the id would change if you change the stage settings more than just renaming it.
My goal is to gather statistics over failed stages. But if the id isn't following the stages when I reorder them I don't know how to distinguish one test from another. Well, as long as I don't rename the stages I can use the name as my unique identifier, but that seems not so smart.
Does anyone have some solution for this or information about how the stage id tag works?
Version of my pipeline plugins (not the latest):
Pipeline: Declarative v1.1.9
Pipeline: Declarative Extension Points API v1.1.9
Pipeline: Groovy v2.39
Pipeline: Job v2.12.2
Pipeline: Multibranch v2.16
Pipeline: Nodes and Processes v2.13
Pipeline: Supporting APIs v2.14

Jenkins doesn't build on first push event, triggered by gitlab

I configured my gitlab to send a webhook to jenkins on each push event.
My problem is, that jenkins don't start to build for the first push event of a branch, i.e. if I create a new branch.
My command is this:
git checkout -b foo ; git push --set-upstream origin foo
And the webhook message looks like that:
{
"before": "0000000000000000000000000000000000000000",
"after": "30677c0db88ea188e501199ba33f05df06dba4fc",
"ref": "refs/heads/foo",
"user_id": 4,
"user_name": "Sascha Schimke",
"project_id": 3,
"repository": {
"name": "...",
"url": "git#gitlab...",
"description": "",
"homepage": "http://gitlab..."
},
"commits": [],
"total_commits_count": 0
}
I would expect the jenkins to start building my project, but it does not.
The 2nd push works fine.
Any ideas?
I found this bug on the GitLab Jenkins plugin: https://github.com/jenkinsci/gitlab-plugin/issues/238
Upgrading to 8.6.4 (using the Docker images) fixed the issue for me.

Gitlab Webhook and Jenkins: No Data Received

I have been following the below links in order to integrate Gitlab with Jenkins using web hooks. All the below links mention to receive 'JSON' or 'payload' or token over at Jenkins side, but I do not see anything when I try to echo or print these parameters in the Shell script of Jenkins configurations.
In shell script I have this, but I never receive any payloads:
echo "the build worked! The payload is $payload"
I do see some JSON coming through on /var/log/Jenkins/Jenkins.logs, but I want to see the messages coming in inside my 'Console Output', so that I can use the messages coming in from Gitlab to whether trigger a build or not.
Most of these links mention options are not available via Gitlab. One article was mentioning to convert web hook format to application/json, but there are no such options on the Gitlab UI.
How to process a github webhook payload in Jenkins?
http://chloky.com/github-json-payload-in-jenkins/
Jenkins Settings:
Gitlab webhook:
http://xx.xx.xx.xxx:8080/job/Interim_Build/buildWithParameters?token=TOKEN_NAME
Any help would be great. Thanks.
I suggest you to try two solutions (both working for me):
convert json data from Gitlab webhook using this elegant proxy written in Go https://github.com/akira/githookproxy .
It will take the webhook request, and translate it to a request to the target_url in the format of:
payload: JSON body
START: Start commit hash
END: End commit hash
REFNAME: Ref name
emulate jenkins as a Gitlab CI using this Jenkins plugin https://github.com/jenkinsci/gitlab-plugin
For me the best is the first because it is simple and more transparent.
GitLab and GitHub are two separate products. So, the documentation or links for GitHub webhooks that you are referring will not apply to GitLab webhooks.
GitLab invokes the webhook URL with a JSON payload in the request body that carries a lot of information about the GitLab event that led to the webhook invocation. For example, the GitLab webhook push event payload carries the following information in it:
{
"object_kind": "push",
"before": "95790bf891e76fee5e1747ab589903a6a1f80f22",
"after": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"ref": "refs/heads/master",
"checkout_sha": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"user_id": 4,
"user_name": "John Smith",
"user_username": "jsmith",
"user_email": "john#example.com",
"user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80",
"project_id": 15,
"project":{
"id": 15,
"name":"Diaspora",
"description":"",
"web_url":"http://example.com/mike/diaspora",
"avatar_url":null,
"git_ssh_url":"git#example.com:mike/diaspora.git",
"git_http_url":"http://example.com/mike/diaspora.git",
"namespace":"Mike",
"visibility_level":0,
"path_with_namespace":"mike/diaspora",
"default_branch":"master",
"homepage":"http://example.com/mike/diaspora",
"url":"git#example.com:mike/diaspora.git",
"ssh_url":"git#example.com:mike/diaspora.git",
"http_url":"http://example.com/mike/diaspora.git"
},
"repository":{
"name": "Diaspora",
"url": "git#example.com:mike/diaspora.git",
"description": "",
"homepage": "http://example.com/mike/diaspora",
"git_http_url":"http://example.com/mike/diaspora.git",
"git_ssh_url":"git#example.com:mike/diaspora.git",
"visibility_level":0
},
"commits": [
{
"id": "b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327",
"message": "Update Catalan translation to e38cb41.",
"timestamp": "2011-12-12T14:27:31+02:00",
"url": "http://example.com/mike/diaspora/commit/b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327",
"author": {
"name": "Jordi Mallach",
"email": "jordi#softcatala.org"
},
"added": ["CHANGELOG"],
"modified": ["app/controller/application.rb"],
"removed": []
},
{
"id": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"message": "fixed readme",
"timestamp": "2012-01-03T23:36:29+02:00",
"url": "http://example.com/mike/diaspora/commit/da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"author": {
"name": "GitLab dev user",
"email": "gitlabdev#dv6700.(none)"
},
"added": ["CHANGELOG"],
"modified": ["app/controller/application.rb"],
"removed": []
}
],
"total_commits_count": 4
}
The Jenkins GitLab plugin makes this webhook payload information available in the Jenkins Global Variable env. The available env variables are as follows:
gitlabBranch
gitlabSourceBranch
gitlabActionType
gitlabUserName
gitlabUserEmail
gitlabSourceRepoHomepage
gitlabSourceRepoName
gitlabSourceNamespace
gitlabSourceRepoURL
gitlabSourceRepoSshUrl
gitlabSourceRepoHttpUrl
gitlabMergeRequestTitle
gitlabMergeRequestDescription
gitlabMergeRequestId
gitlabMergeRequestIid
gitlabMergeRequestState
gitlabMergedByUser
gitlabMergeRequestAssignee
gitlabMergeRequestLastCommit
gitlabMergeRequestTargetProjectId
gitlabTargetBranch
gitlabTargetRepoName
gitlabTargetNamespace
gitlabTargetRepoSshUrl
gitlabTargetRepoHttpUrl
gitlabBefore
gitlabAfter
gitlabTriggerPhrase
Just as you would read Jenkins job parameters from Jenkins Global Variable params in your job pipeline script, you could read webhook payload fields from Jenkins Global Variable env:
echo "My Jenkins job parameter is ${params.MY_PARAM_NAME}"
echo "One of Jenkins job webhook payload field is ${env.gitlabMergedByUser}"
Hope, the above information helps solve your problem.

Read json payload from gitlab webhook in Jenkins

I followed this tutorial to setup a Jenkins job to run whenever a push is made to the gitlab repository. I tested the webhook and I can see that the job is triggered. However, I don't see anything in the payload.
Just wondering, if anyone has ever tried to read the payload received from gitlab webhook?
Jenkins Gitlab Plugin sends these POST parameters to Jenkins whenever any event occurs in the Gitlab repo.
You can add env in the Jenkins console to get what all Gitlab parameters are exported to the Jenkins environment. Then you can print or use the required variables.
e.g
echo $gitlabSourceRepoURL
echo $gitlabAfter
echo $gitlabTargetBranch
echo $gitlabSourceRepoHttpUrl
echo $gitlabMergeRequestLastCommit
echo $gitlabSourceRepoSshUrl
echo $gitlabSourceRepoHomepage
echo $gitlabBranch
echo $gitlabSourceBranch
echo $gitlabUserEmail
echo $gitlabBefore
echo $gitlabSourceRepoName
echo $gitlabSourceNamespace
echo $gitlabUserName
The tutorial you have mentioned talks about GitHub webhooks. GitLab and GitHub are two separate products. So, the documentation or links for GitHub webhooks will not apply to GitLab webhooks.
GitLab invokes the webhook URL with a JSON payload in the request body that carries a lot of information about the GitLab event that led to the webhook invocation. For example, the GitLab webhook push event payload carries the following information in it:
{
"object_kind": "push",
"before": "95790bf891e76fee5e1747ab589903a6a1f80f22",
"after": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"ref": "refs/heads/master",
"checkout_sha": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"user_id": 4,
"user_name": "John Smith",
"user_username": "jsmith",
"user_email": "john#example.com",
"user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80",
"project_id": 15,
"project":{
"id": 15,
"name":"Diaspora",
"description":"",
"web_url":"http://example.com/mike/diaspora",
"avatar_url":null,
"git_ssh_url":"git#example.com:mike/diaspora.git",
"git_http_url":"http://example.com/mike/diaspora.git",
"namespace":"Mike",
"visibility_level":0,
"path_with_namespace":"mike/diaspora",
"default_branch":"master",
"homepage":"http://example.com/mike/diaspora",
"url":"git#example.com:mike/diaspora.git",
"ssh_url":"git#example.com:mike/diaspora.git",
"http_url":"http://example.com/mike/diaspora.git"
},
"repository":{
"name": "Diaspora",
"url": "git#example.com:mike/diaspora.git",
"description": "",
"homepage": "http://example.com/mike/diaspora",
"git_http_url":"http://example.com/mike/diaspora.git",
"git_ssh_url":"git#example.com:mike/diaspora.git",
"visibility_level":0
},
"commits": [
{
"id": "b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327",
"message": "Update Catalan translation to e38cb41.",
"timestamp": "2011-12-12T14:27:31+02:00",
"url": "http://example.com/mike/diaspora/commit/b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327",
"author": {
"name": "Jordi Mallach",
"email": "jordi#softcatala.org"
},
"added": ["CHANGELOG"],
"modified": ["app/controller/application.rb"],
"removed": []
},
{
"id": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"message": "fixed readme",
"timestamp": "2012-01-03T23:36:29+02:00",
"url": "http://example.com/mike/diaspora/commit/da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"author": {
"name": "GitLab dev user",
"email": "gitlabdev#dv6700.(none)"
},
"added": ["CHANGELOG"],
"modified": ["app/controller/application.rb"],
"removed": []
}
],
"total_commits_count": 4
}
The Jenkins GitLab plugin makes this webhook payload information available in the Jenkins Global Variable env. The available env variables are as follows:
gitlabBranch
gitlabSourceBranch
gitlabActionType
gitlabUserName
gitlabUserEmail
gitlabSourceRepoHomepage
gitlabSourceRepoName
gitlabSourceNamespace
gitlabSourceRepoURL
gitlabSourceRepoSshUrl
gitlabSourceRepoHttpUrl
gitlabMergeRequestTitle
gitlabMergeRequestDescription
gitlabMergeRequestId
gitlabMergeRequestIid
gitlabMergeRequestState
gitlabMergedByUser
gitlabMergeRequestAssignee
gitlabMergeRequestLastCommit
gitlabMergeRequestTargetProjectId
gitlabTargetBranch
gitlabTargetRepoName
gitlabTargetNamespace
gitlabTargetRepoSshUrl
gitlabTargetRepoHttpUrl
gitlabBefore
gitlabAfter
gitlabTriggerPhrase
Just as you would read Jenkins job parameters from Jenkins Global Variable params in your job pipeline script, you could read webhook payload fields from Jenkins Global Variable env:
echo "My Jenkins job parameter is ${params.MY_PARAM_NAME}"
echo "One of Jenkins job webhook payload field is ${env.gitlabTargetBranch}"
Hope, the above information helps solve your problem.
Yes, I did it. And it works for some scenarios.
If you use /gitlab/buildnow, you can have access to payload objects. All of them.
But you have to name them under "this build is parametrized".
Then you can access them by name, like ${AUTHOR_NAME}.
Doc: https://github.com/elvanja/jenkins-gitlab-hook-plugin#parameterized-projects
But please note that if you use /gitlab/notifycommit, it will not work, since there is a gap (the poll) between triggering jenkins, and actually starting the job. All payload data in this situation is empty.
But be carefull to use /gitlab/buildnow, because you cannot control if you want or not to build, like when Maven commit back some files, and build is not supposed to be triggered.
What I did was to write a little tool in Python to receive all gitlab notification, and this tool talks back to GitLab and Jenkins, to fire (or not) jobs, and collect back statuses.
My start point:
How do I receive Github Webhooks in Python (last answer, not the choosen one).
I started developing it 2 days ago. It's done, but I am still validating it.

Resources