Related
I have a AQL that is used to delete files from the artifactory. I am designing a pipeline so that it can delete files from artifactory based on the AQL.
I know for artifactory upload and download we can specify the AQL within the pipeline using rtupload/rtdownload, but is there a way to do for delete as well?
#AQL delete.spec
{
"files": [
{
"aql": {
"items.find": {
"repo": {"$eq":"app-java-repo"},
"path": "archives/test/app",
"type": "folder",
"$or": [
{
"$and": [
{
"name": {"$nmatch" : "*build*"}
}
]
}
]
}
}
}
]
}
jf rt del --spec delete.spec
Using the jfrog cli, the spec file can be passed and the files from artifactory repo can be deleted. This is working out manually, but how can the same be achieved by having the AQL in the pipeline stage instead of having AQL as a file.
I'm using the Jenkins pipeline stage view plugin REST API. When I use the path /job/:job-name/wfapi/runs I get a list of the last jobs.
Yesterday I renamed all the stages and also changed the order of the stages a bit. Here is a part of the JSON file, #152 is before the change and #153 is after:
"name": "#153",
"id": "6",
"name": "Compile",
"id": "13",
"name": "Flash image",
"id": "19",
"name": "Generate log",
...
"name": "#152",
"id": "6",
"name": "1. Compile",
"id": "13",
"name": "2. Flash image",
"id": "19",
"name": "3. Prepare hardware",
...
"id": "31",
"name": "6. Generate log",
What I feel is very strange is that the "Generate log" stage in #153 has taken over the stage id 19 from "3. Prepare hardware" in #152 ("Generate log" failed so I don't have any id number for "Prepare hardware" that comes later in the pipeline). I thought that the id was unique for a stage setup and that the id would change if you change the stage settings more than just renaming it.
My goal is to gather statistics over failed stages. But if the id isn't following the stages when I reorder them I don't know how to distinguish one test from another. Well, as long as I don't rename the stages I can use the name as my unique identifier, but that seems not so smart.
Does anyone have some solution for this or information about how the stage id tag works?
Version of my pipeline plugins (not the latest):
Pipeline: Declarative v1.1.9
Pipeline: Declarative Extension Points API v1.1.9
Pipeline: Groovy v2.39
Pipeline: Job v2.12.2
Pipeline: Multibranch v2.16
Pipeline: Nodes and Processes v2.13
Pipeline: Supporting APIs v2.14
I'm trying to build a job on my jenkins using Gitlab webhook when pushing into a specific branch.
The only issue I have is that the jenkins job is parameterized. The parameter is the branch name.
Is there an option to add the branch name (its a different branch every time) to the webhook so it will be inserted automatically into the parameter in the jenkins job parameter?
Thanks
GitLab invokes the webhook URL with a JSON payload in the request body that carries a lot of information about the GitLab event that led to the webhook invocation. The JSON payload for the GitLab push event does have branch name information in it -- see the "ref" field in the GitLab webhook push event payload:
{
"object_kind": "push",
"before": "95790bf891e76fee5e1747ab589903a6a1f80f22",
"after": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"ref": "refs/heads/master",
"checkout_sha": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"user_id": 4,
"user_name": "John Smith",
"user_username": "jsmith",
"user_email": "john#example.com",
"user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80",
"project_id": 15,
"project":{
"id": 15,
"name":"Diaspora",
"description":"",
"web_url":"http://example.com/mike/diaspora",
"avatar_url":null,
"git_ssh_url":"git#example.com:mike/diaspora.git",
"git_http_url":"http://example.com/mike/diaspora.git",
"namespace":"Mike",
"visibility_level":0,
"path_with_namespace":"mike/diaspora",
"default_branch":"master",
"homepage":"http://example.com/mike/diaspora",
"url":"git#example.com:mike/diaspora.git",
"ssh_url":"git#example.com:mike/diaspora.git",
"http_url":"http://example.com/mike/diaspora.git"
},
"repository":{
"name": "Diaspora",
"url": "git#example.com:mike/diaspora.git",
"description": "",
"homepage": "http://example.com/mike/diaspora",
"git_http_url":"http://example.com/mike/diaspora.git",
"git_ssh_url":"git#example.com:mike/diaspora.git",
"visibility_level":0
},
"commits": [
{
"id": "b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327",
"message": "Update Catalan translation to e38cb41.",
"timestamp": "2011-12-12T14:27:31+02:00",
"url": "http://example.com/mike/diaspora/commit/b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327",
"author": {
"name": "Jordi Mallach",
"email": "jordi#softcatala.org"
},
"added": ["CHANGELOG"],
"modified": ["app/controller/application.rb"],
"removed": []
},
{
"id": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"message": "fixed readme",
"timestamp": "2012-01-03T23:36:29+02:00",
"url": "http://example.com/mike/diaspora/commit/da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"author": {
"name": "GitLab dev user",
"email": "gitlabdev#dv6700.(none)"
},
"added": ["CHANGELOG"],
"modified": ["app/controller/application.rb"],
"removed": []
}
],
"total_commits_count": 4
}
You may not bind any field in the webhook payload to a Jenkins job parameter automatically. However, there is still a way to access webhook payload information in the Jenkins job. The Jenkins GitLab plugin makes this webhook payload information available in the Jenkins Global Variable env. The available env variables are as follows and they do include branch information:
gitlabBranch
gitlabSourceBranch
gitlabActionType
gitlabUserName
gitlabUserEmail
gitlabSourceRepoHomepage
gitlabSourceRepoName
gitlabSourceNamespace
gitlabSourceRepoURL
gitlabSourceRepoSshUrl
gitlabSourceRepoHttpUrl
gitlabMergeRequestTitle
gitlabMergeRequestDescription
gitlabMergeRequestId
gitlabMergeRequestIid
gitlabMergeRequestState
gitlabMergedByUser
gitlabMergeRequestAssignee
gitlabMergeRequestLastCommit
gitlabMergeRequestTargetProjectId
gitlabTargetBranch
gitlabTargetRepoName
gitlabTargetNamespace
gitlabTargetRepoSshUrl
gitlabTargetRepoHttpUrl
gitlabBefore
gitlabAfter
gitlabTriggerPhrase
Just as you would read Jenkins job parameters from Jenkins Global Variable params in your job pipeline script, you could read webhook payload fields from Jenkins Global Variable env:
params.MY_PARAM_NAME
env.gitlabBranch
Hope, the above information helps solve your problem.
You can do that with the Generic Webhook Trigger plugin. There is also an example of using that plugin with GitLab in the Violation Comments to GitLab page.
The source branch is resolved with the following JSONPath:
$.object_attributes.source_branch
I have been following the below links in order to integrate Gitlab with Jenkins using web hooks. All the below links mention to receive 'JSON' or 'payload' or token over at Jenkins side, but I do not see anything when I try to echo or print these parameters in the Shell script of Jenkins configurations.
In shell script I have this, but I never receive any payloads:
echo "the build worked! The payload is $payload"
I do see some JSON coming through on /var/log/Jenkins/Jenkins.logs, but I want to see the messages coming in inside my 'Console Output', so that I can use the messages coming in from Gitlab to whether trigger a build or not.
Most of these links mention options are not available via Gitlab. One article was mentioning to convert web hook format to application/json, but there are no such options on the Gitlab UI.
How to process a github webhook payload in Jenkins?
http://chloky.com/github-json-payload-in-jenkins/
Jenkins Settings:
Gitlab webhook:
http://xx.xx.xx.xxx:8080/job/Interim_Build/buildWithParameters?token=TOKEN_NAME
Any help would be great. Thanks.
I suggest you to try two solutions (both working for me):
convert json data from Gitlab webhook using this elegant proxy written in Go https://github.com/akira/githookproxy .
It will take the webhook request, and translate it to a request to the target_url in the format of:
payload: JSON body
START: Start commit hash
END: End commit hash
REFNAME: Ref name
emulate jenkins as a Gitlab CI using this Jenkins plugin https://github.com/jenkinsci/gitlab-plugin
For me the best is the first because it is simple and more transparent.
GitLab and GitHub are two separate products. So, the documentation or links for GitHub webhooks that you are referring will not apply to GitLab webhooks.
GitLab invokes the webhook URL with a JSON payload in the request body that carries a lot of information about the GitLab event that led to the webhook invocation. For example, the GitLab webhook push event payload carries the following information in it:
{
"object_kind": "push",
"before": "95790bf891e76fee5e1747ab589903a6a1f80f22",
"after": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"ref": "refs/heads/master",
"checkout_sha": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"user_id": 4,
"user_name": "John Smith",
"user_username": "jsmith",
"user_email": "john#example.com",
"user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80",
"project_id": 15,
"project":{
"id": 15,
"name":"Diaspora",
"description":"",
"web_url":"http://example.com/mike/diaspora",
"avatar_url":null,
"git_ssh_url":"git#example.com:mike/diaspora.git",
"git_http_url":"http://example.com/mike/diaspora.git",
"namespace":"Mike",
"visibility_level":0,
"path_with_namespace":"mike/diaspora",
"default_branch":"master",
"homepage":"http://example.com/mike/diaspora",
"url":"git#example.com:mike/diaspora.git",
"ssh_url":"git#example.com:mike/diaspora.git",
"http_url":"http://example.com/mike/diaspora.git"
},
"repository":{
"name": "Diaspora",
"url": "git#example.com:mike/diaspora.git",
"description": "",
"homepage": "http://example.com/mike/diaspora",
"git_http_url":"http://example.com/mike/diaspora.git",
"git_ssh_url":"git#example.com:mike/diaspora.git",
"visibility_level":0
},
"commits": [
{
"id": "b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327",
"message": "Update Catalan translation to e38cb41.",
"timestamp": "2011-12-12T14:27:31+02:00",
"url": "http://example.com/mike/diaspora/commit/b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327",
"author": {
"name": "Jordi Mallach",
"email": "jordi#softcatala.org"
},
"added": ["CHANGELOG"],
"modified": ["app/controller/application.rb"],
"removed": []
},
{
"id": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"message": "fixed readme",
"timestamp": "2012-01-03T23:36:29+02:00",
"url": "http://example.com/mike/diaspora/commit/da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"author": {
"name": "GitLab dev user",
"email": "gitlabdev#dv6700.(none)"
},
"added": ["CHANGELOG"],
"modified": ["app/controller/application.rb"],
"removed": []
}
],
"total_commits_count": 4
}
The Jenkins GitLab plugin makes this webhook payload information available in the Jenkins Global Variable env. The available env variables are as follows:
gitlabBranch
gitlabSourceBranch
gitlabActionType
gitlabUserName
gitlabUserEmail
gitlabSourceRepoHomepage
gitlabSourceRepoName
gitlabSourceNamespace
gitlabSourceRepoURL
gitlabSourceRepoSshUrl
gitlabSourceRepoHttpUrl
gitlabMergeRequestTitle
gitlabMergeRequestDescription
gitlabMergeRequestId
gitlabMergeRequestIid
gitlabMergeRequestState
gitlabMergedByUser
gitlabMergeRequestAssignee
gitlabMergeRequestLastCommit
gitlabMergeRequestTargetProjectId
gitlabTargetBranch
gitlabTargetRepoName
gitlabTargetNamespace
gitlabTargetRepoSshUrl
gitlabTargetRepoHttpUrl
gitlabBefore
gitlabAfter
gitlabTriggerPhrase
Just as you would read Jenkins job parameters from Jenkins Global Variable params in your job pipeline script, you could read webhook payload fields from Jenkins Global Variable env:
echo "My Jenkins job parameter is ${params.MY_PARAM_NAME}"
echo "One of Jenkins job webhook payload field is ${env.gitlabMergedByUser}"
Hope, the above information helps solve your problem.
I followed this tutorial to setup a Jenkins job to run whenever a push is made to the gitlab repository. I tested the webhook and I can see that the job is triggered. However, I don't see anything in the payload.
Just wondering, if anyone has ever tried to read the payload received from gitlab webhook?
Jenkins Gitlab Plugin sends these POST parameters to Jenkins whenever any event occurs in the Gitlab repo.
You can add env in the Jenkins console to get what all Gitlab parameters are exported to the Jenkins environment. Then you can print or use the required variables.
e.g
echo $gitlabSourceRepoURL
echo $gitlabAfter
echo $gitlabTargetBranch
echo $gitlabSourceRepoHttpUrl
echo $gitlabMergeRequestLastCommit
echo $gitlabSourceRepoSshUrl
echo $gitlabSourceRepoHomepage
echo $gitlabBranch
echo $gitlabSourceBranch
echo $gitlabUserEmail
echo $gitlabBefore
echo $gitlabSourceRepoName
echo $gitlabSourceNamespace
echo $gitlabUserName
The tutorial you have mentioned talks about GitHub webhooks. GitLab and GitHub are two separate products. So, the documentation or links for GitHub webhooks will not apply to GitLab webhooks.
GitLab invokes the webhook URL with a JSON payload in the request body that carries a lot of information about the GitLab event that led to the webhook invocation. For example, the GitLab webhook push event payload carries the following information in it:
{
"object_kind": "push",
"before": "95790bf891e76fee5e1747ab589903a6a1f80f22",
"after": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"ref": "refs/heads/master",
"checkout_sha": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"user_id": 4,
"user_name": "John Smith",
"user_username": "jsmith",
"user_email": "john#example.com",
"user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80",
"project_id": 15,
"project":{
"id": 15,
"name":"Diaspora",
"description":"",
"web_url":"http://example.com/mike/diaspora",
"avatar_url":null,
"git_ssh_url":"git#example.com:mike/diaspora.git",
"git_http_url":"http://example.com/mike/diaspora.git",
"namespace":"Mike",
"visibility_level":0,
"path_with_namespace":"mike/diaspora",
"default_branch":"master",
"homepage":"http://example.com/mike/diaspora",
"url":"git#example.com:mike/diaspora.git",
"ssh_url":"git#example.com:mike/diaspora.git",
"http_url":"http://example.com/mike/diaspora.git"
},
"repository":{
"name": "Diaspora",
"url": "git#example.com:mike/diaspora.git",
"description": "",
"homepage": "http://example.com/mike/diaspora",
"git_http_url":"http://example.com/mike/diaspora.git",
"git_ssh_url":"git#example.com:mike/diaspora.git",
"visibility_level":0
},
"commits": [
{
"id": "b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327",
"message": "Update Catalan translation to e38cb41.",
"timestamp": "2011-12-12T14:27:31+02:00",
"url": "http://example.com/mike/diaspora/commit/b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327",
"author": {
"name": "Jordi Mallach",
"email": "jordi#softcatala.org"
},
"added": ["CHANGELOG"],
"modified": ["app/controller/application.rb"],
"removed": []
},
{
"id": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"message": "fixed readme",
"timestamp": "2012-01-03T23:36:29+02:00",
"url": "http://example.com/mike/diaspora/commit/da1560886d4f094c3e6c9ef40349f7d38b5d27d7",
"author": {
"name": "GitLab dev user",
"email": "gitlabdev#dv6700.(none)"
},
"added": ["CHANGELOG"],
"modified": ["app/controller/application.rb"],
"removed": []
}
],
"total_commits_count": 4
}
The Jenkins GitLab plugin makes this webhook payload information available in the Jenkins Global Variable env. The available env variables are as follows:
gitlabBranch
gitlabSourceBranch
gitlabActionType
gitlabUserName
gitlabUserEmail
gitlabSourceRepoHomepage
gitlabSourceRepoName
gitlabSourceNamespace
gitlabSourceRepoURL
gitlabSourceRepoSshUrl
gitlabSourceRepoHttpUrl
gitlabMergeRequestTitle
gitlabMergeRequestDescription
gitlabMergeRequestId
gitlabMergeRequestIid
gitlabMergeRequestState
gitlabMergedByUser
gitlabMergeRequestAssignee
gitlabMergeRequestLastCommit
gitlabMergeRequestTargetProjectId
gitlabTargetBranch
gitlabTargetRepoName
gitlabTargetNamespace
gitlabTargetRepoSshUrl
gitlabTargetRepoHttpUrl
gitlabBefore
gitlabAfter
gitlabTriggerPhrase
Just as you would read Jenkins job parameters from Jenkins Global Variable params in your job pipeline script, you could read webhook payload fields from Jenkins Global Variable env:
echo "My Jenkins job parameter is ${params.MY_PARAM_NAME}"
echo "One of Jenkins job webhook payload field is ${env.gitlabTargetBranch}"
Hope, the above information helps solve your problem.
Yes, I did it. And it works for some scenarios.
If you use /gitlab/buildnow, you can have access to payload objects. All of them.
But you have to name them under "this build is parametrized".
Then you can access them by name, like ${AUTHOR_NAME}.
Doc: https://github.com/elvanja/jenkins-gitlab-hook-plugin#parameterized-projects
But please note that if you use /gitlab/notifycommit, it will not work, since there is a gap (the poll) between triggering jenkins, and actually starting the job. All payload data in this situation is empty.
But be carefull to use /gitlab/buildnow, because you cannot control if you want or not to build, like when Maven commit back some files, and build is not supposed to be triggered.
What I did was to write a little tool in Python to receive all gitlab notification, and this tool talks back to GitLab and Jenkins, to fire (or not) jobs, and collect back statuses.
My start point:
How do I receive Github Webhooks in Python (last answer, not the choosen one).
I started developing it 2 days ago. It's done, but I am still validating it.