I'm new to this continuous integration thing. I want to use Jenkins as my CI system, but I can't get it to pull the build everytime there's a new one.
Using mercurial's plugin I'm able to connect to my repository and pull my builds normally, but I don't want Jenkins to keep polling, I want it to update the build only when there's a new one instead. On the plugin's wiki I found this:
As of version 1.38 it's possible to trigger builds using push
notifications instead of polling. In your repository's .hg/hgrc file
add:
[hooks]
commit.jenkins = wget -q -O /dev/null <jenkins root>/mercurial/notifyCommit?url=<repository remote url>
incoming.jenkins = wget -q -O /dev/null <jenkins root>/mercurial/notifyCommit?url=<repository remote url>
For now I'm keeping Jenkis local, so I used this o my hgrc file:
commit.jenkins = wget -q -O /dev/null http://localhost:8080/mercurial/notifyCommit?url=<my repository remote url>
incoming.jenkins = wget -q -O /dev/null http://localhost:8080/mercurial/notifyCommit?url=<my repository remote url>
But builds aren't being triggered. Could someone help me?
[UPDATE]
I didn't pay attention to the wget command, which doesn't exist on windows. Installed it and it's still the same. Jenkins is not pulling the builds.
You must to have wget on PATH (I'll recommend native port of GOW, not Cygwin - or Bash in Win10)
Your hooks must be in working state
wget ... must produce the expected result
you have threenow two possible points of failure and have to test all independently
Does my hooks work?
Replace your current content of hooks with dumb billet like
commit.jenkins = echo Commit hook here
incoming.jenkins = echo Incoming hook here
and test hooks (in console for better visibility) by executing commit into repo with added hook and pull|push to it|unbundle anything. If you'll see hook output - they are usable
Does Jenkins integration work?
After commit to repo you can perform task of hook by hand: run wget -q -O /dev/null ... and check results in Jenkins
Related
I have a CI pipeline in Bitbucket which is building, testing and deploying an application.
The thing is that after the deploy I want to run selenium tests.
Selenium tests are in an another repository in Bitbucket and they have their own pipeline.
Is there a trigger step in the Bitbucket pipeline to trigger a pipeline when a previous one has finished?
I do not want to do a fake push to the test repository to trigger those tests.
The most "correct" way I can think of doing this is to use the Bitbucket REST API to manually trigger a pipeline on the other repository, after your deployment completes.
There are several examples of how to create a pipeline here: https://developer.atlassian.com/bitbucket/api/2/reference/resource/repositories/%7Bworkspace%7D/%7Brepo_slug%7D/pipelines/#post
Copy + pasting the first example. How to trigger a pipeline for the latest commit on master:
$ curl -X POST -is -u username:password \
-H 'Content-Type: application/json' \
https://api.bitbucket.org/2.0/repositories/jeroendr/meat-demo2/pipelines/ \
-d '
{
"target": {
"ref_type": "branch",
"type": "pipeline_ref_target",
"ref_name": "master"
}
}'
according to their official documentation there is no "easy way" to do that, cause the job are isolated in scope of one repository, yet you can achieve your task in following way:
create docker image with minimum required setup for execution of your tests inside
upload to docker hub (or some other repo if you have such)
use docker image in last step of you pipeline after deploy to execute tests
Try out official component Bitbucket pipeline trigger: https://bitbucket.org/product/features/pipelines/integrations?p=atlassian/trigger-pipeline
You can run in after deploy step
script:
- pipe: atlassian/trigger-pipeline:4.1.7
variables:
BITBUCKET_USERNAME: $BITBUCKET_USERNAME
BITBUCKET_APP_PASSWORD: $BITBUCKET_APP_PASSWORD
REPOSITORY: 'your-awesome-repo'
ACCOUNT: 'teams-in-space'
#BigGinDaHouse I did something more or less like you say.
My step is built on top of docker image with headless chrome, npm and git.
I did follow the steps below:
I have set private key for the remote repo in the original repo. Encoded base 64. documentation. The public key is being set into the remote repo in SSH Access option in bitbucket menu.
In the pipeline step I am decoding it and setting it to a file. I am also changing its permission to be 400.
I am adding this Key inside the docker image. ssh-add
Then I am able to do a git clone followed by npm install and npm test
NOTE: The entry.sh is because I am starting the headless browser.
- step:
image: kimy82/headless-selenium-npm-git
script:
- echo $key_in_env_variable_in_bitbucket | base64 --decode > priv_key
- chmod 400 ./priv_key
- eval `ssh-agent -s`
- ssh-agent $(ssh-add priv_key; git clone git#bitbucket.org:project.git)
- cd project
- nohup bash /usr/bin/entry.sh >> out.log &
- npm install
- npm test
Top answers (this and this) are correct, they work.
Just adding that we found out (after a LOT of trial and error) that the user executing the pipeline must have WRITE permissions on the repo where the pipeline is invoked (even though his app password permissions were set to "WRITE" for repos and pipelines...)
Also, this works for executing pipelines in Bitbucket's cloud or on-premise, through local runners.
(Answering as I am lacking reputation for commenting)
I have a requirement where i need to trigger the build jobs remotely using curl command. I am unable to pass the branch/tag name as a parameter to trigger the build.
I used the below command :
& $CURLEXE -k -X POST $dst_job_url --user username:token --data-urlencode json='{"parameters": [{"name":"branch","branch":"branches"}]}'
If i run the above command it was triggers the build for the trunk ( default ).
You omitted the URL, so it's hard to be certain. Jenkins has two urls for building: "build" and "buildWithParameters". If you're not using buildWithParameters, switching to it will probably help.
See:
How to trigger Jenkins builds remotely and to pass parameters
I tried many times to execute a Continuous Integration via BitBucket Pipeline (beta). For the moment I need a very simple task, just update my remote server when a push is made on repository (In the past I used for this purpose CodeShip with a very similar syntax).
In Pipelines is necessary to set up a file called bitbucket-pipelines.yml which contains several rows to differentiate between branches, etc. but the main instruction is:
- lftp -c "open -u $FTP_USER,$FTP_PASSWORD ftp.mydomain.com; set ssl:verify-certificate no; mirror -Rne /opt/atlassian/bitbucketci/agent/build /clone/ /public_html/dev"
Unfortunately it does not run correctly because it failed (apparently with infinite loop and new attempts).
I tried to discuss this topic with Support but I did not recieve any useful help and in the final message, they simply suggested me other resources.
Maybe, is there anybody that set up succesfully a similar things?
Thanks
If its just a git push you want, you could try this.
image: samueldebruyn/debian-git
pipelines:
default:
- step:
script:
- echo "Pipeline Init"
- apt-get update
- apt-get -qq install git-ftp
- echo "'_$(git status -uno --porcelain | wc -l)_'"
- git status -uno --porcelain
- echo "Initiating Push site:Source."
- git config git-ftp.syncroot Source/
- git ftp init --user $Username --passwd $Pwd ftp://domain.com/public_html/
Once you have done the first push (init), change the code git ftp init to git ftp push
I have Jenkins / Mercurial latest versions. Machine is Red Hat Linux 6.6.
I'm using Release plugin in Jenkins. Maven plugins (maven-scm-plugin, maven-version-plugin and maven-enforcer-plugin) for doing release process on a project. All of these versions are using latest available versions and configuration is setup correctly.
In Jenkins job, I'm checking out the source code from a Project which sits behind RhodeCode (Mercurial hg).
Output shows like:
06:00:02 Started by timer
06:00:02 [EnvInject] - Loading node environment variables.
06:00:03 Building on master in workspace /main/jenkins/instance2/workspace/MyCoolProject
06:00:06 [MyCoolProject] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" showconfig paths.default
06:00:06 [MyCoolProject] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" pull --rev default
06:00:12 pulling from http://mercurialserver.my.company.com:9001/csa/MyCoolProject/
06:00:12 no changes found
06:00:12 [MyCoolProject] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" update --clean --rev default
06:00:13 0 files updated, 0 files merged, 0 files removed, 0 files unresolved
06:00:13 [MyCoolProject] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" --config extensions.purge= clean --all
06:00:13 [MyCoolProject] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" log --rev . --template {node}
06:00:13 [MyCoolProject] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" log --rev . --template {rev}
06:00:13 [MyCoolProject] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" log --rev
As you see above, because we specified a user to perform the read/checkout operation in Mercurial DVCS tool, Jenkins passed bunch of parameters while initiating the hg commands i.e.
--config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https"
I'm trying to find out how I can send the same, when I'm using Maven SCM plugin and it's goals.
scm:checkin
scm:tag
etc
which when called, initiates the underlaying version control commands (in my case, hg branch, hg outgoing, hg push commands).
Right now, the above hg commands are failing due to the fact that either the user doesn't have ~/.hgrc file containing username and password variable set OR the user does NOT have "WRITE" access to the target RhodeCode/Mercurial repository.
20:16:26 [INFO] --- maven-scm-plugin:1.9.4:checkin (default-cli) # MyCoolProject ---
20:16:27 [INFO] EXECUTING: /bin/sh -c cd /main/jenkins/instance2/workspace/MyCoolProject && hg branch
20:16:27 [INFO] EXECUTING: /bin/sh -c cd /main/jenkins/instance2/workspace/MyCoolProject && hg outgoing
20:16:29 [INFO] EXECUTING: /bin/sh -c cd /main/jenkins/instance2/workspace/MyCoolProject && hg status
20:16:29 [INFO] [pom.xml:modified]
20:16:29 [INFO] [jenkins-MyCoolProject-43.appVersion.txt:unknown]
20:16:29 [INFO] [pom.xml.versionsBackup:unknown]
20:16:29 [INFO] EXECUTING: /bin/sh -c cd /main/jenkins/instance2/workspace/MyCoolProject && hg commit --message '"CM Jenkins - Release plugin auto check-in and creation of release tag = 0.0.29'
20:16:30 [INFO] EXECUTING: /bin/sh -c cd /main/jenkins/instance2/workspace/MyCoolProject && hg push http://cmprod2merc.my.company.com:9001/csa/MyCoolProject
20:16:34 [ERROR]
20:16:34 EXECUTION FAILED
20:16:34 Execution of cmd : push failed with exit code: 255.
20:16:34 Working directory was:
20:16:34 /main/jenkins/instance2/workspace/MyCoolProject
20:16:34 Your Hg installation seems to be valid and complete.
20:16:34 Hg version: 1.9.2 (OK)
I tried configuring ~/.hgrc file (as per Mercurial docs) and everything works if the username you specify in this file has valid WRITE access on the target source code repository/project then the above errors message won't come.
This can also be resolved if I install/configure keyring and mercurial_keyring extension/plugin on the build machine (which is easy as per the online mercurial keyring documentation).
What I'm trying to find out is: How can I pass the --config xxxxx parameters (like Jenkins is passing while doing checkout / pull operations - coming due to the setting values we have entered under Source Code Management in Jenkins job) to the hg commands which are called when Maven SCM plugin is initiating scm:checkin / scm:tag goals which ends up calling hg commands (hg outgoing, hg push) / all hg commands without setting ~/.hgrc and installing mercurial_keyring?
Found the solution:
Even when the user (jenkins) had valid WRITE access on the target RhodeCode/Merurial repository and either ~/.hgrc or mercurial_keyring setup wsa configured successfully (i.e. username/password less hg operations after one time manual entry), Maven SCM plugin scm:checkin and scm:tag operation were still failing.
The reason for that is scm:checkin / scm:tag goals (of Maven SCM plugin) calls the version tool commands (hg commands in my case) but it was NOT passing the authentication params (username/password). For that either I could have added the and values in project pom.xml OR in ~/.m2/settings.xml withing maven-scm-plugin plugin's configuration ---OR (more secured way is to) create/use 2 new Jenkins global level variables (of password type) to create username/password variables and use / pass them to scm:checkin / scm:tag goals while calling these goals within Release plugin's configuration settings in Jenkins i.e. -Dusername=$username and -Dpassword=$password (as the values are coming from Jenkins, they will be masked automatically).
I went with the Jenkins route and create 2 password type global variables in Jenkins Global configuration under "Configure system" > Global parameters/passwords section and just passed them to scm:checkin/tag goals while calling them in Invoke Maven step (within Release plugin configuration in Jenkins).
I found, if you have ~/.hgrc set with just username in it, then Jenkins checkout /pull commands started to fail as Jenkins process stopped using the credentials what I was using to pull/clone the source code (it seems like it was giving preference to the ~/.hgrc username first as Jenkins job/process was running with jenkins user and because it didn't had a password variable/field set in ~/.hgrc, the pull/clone failed for some reason (it should have given preference to the user/credentials what we specify in the job's configuration itself). If I moved ~/.hgrc to ~/.hgrc-backup, then checkout/pull/clone worked in Jenkins worked fine (as it used the credentials what I mentioned in the Source Code Management section for Mercurial) BUT, it still failed during Maven SCM plugin not handling the under laying hg commands.
PS: running "hg push" on the workspace was working successfully (standalone, at command line) but when Maven SCM plugin was calling these goals and the goals were calling the hg commands, it didn't work for some reason.
Solution was:
pass -Dusername=$username -Dpassword=$password variable to scm:checkin/scm:tag goals
Make sure ~/.hgrc had username / password variables set --OR mercurial_keyring set to work with the repository (without prompting for username and password).
The reason, Maven SCM plugin goals .. which called hg commands didn't work was due to an issue with the plugin I guess. A work around is to call these goals with -DpushChanges=false and this way, the goals will not call the under laying version control push operation and thus it'll succeed. Then you have to manually add another step in the "Release Plugin in Jenkins's configuration" as "Execute Shell / Execute Windows Batch command" way to run "hg push". Then, it will work and in this case, you don't need to pass -Dusername and -Dpassword parameters to these goals.
Is there a way to specify a hook in the single repository?
Now we have specified the hook in the "/etc/mercurial/hgrc" file, but every time it builds twice, and it builds for each commit in each repository.
So we want to specify a build per repository.
This is how we implemented the hook:
[hooks]
changegroup = curl --silent http://jenkins:8080/job/ourProject/build
It's on a Ubuntu server.
Select the Poll SCM option under Build Triggers.
Make sure that schedule form is empty.
You should be creating in the .hg directory, /home/user/mercurial/.hg/hgrc and add hooks as:
[hooks]
commit.jenkins = wget -q http://localhost:8080/mercurial/notifyCommit?url=file:///home/user/mercurial > /dev/null
incoming.jenkins = wget -q http://localhost:8080/mercurial/notifyCommit?url=file:///home/user/mercurial > /dev/null
You should make sure that
Your Jenkins project doesn't poll
You use the proper notifyCommit URLs for your Mercurial hooks: https://wiki.jenkins-ci.org/display/JENKINS/Mercurial+Plugin
Ok, I found what I looked for (I'm the bounty; my case is Mercurial with a specific branch).
In the main/origin repository, place a hook with your desired build script. Pregroupchange is to maintain the incoming changes. I have a rhodecode installed on the main repository and itself has its own hooks.
In this way, I still trigger Jenkins and still have the changes afther the trigger for rhodecode push notifications and others.
[hooks]
pregroupchange = /path/to/script.extention
In the script, place your desired actions, also a trigger for Jenkins. Don't forget to enable in Jenkins:Job:Configure:Build Triggers:checkbox Trigger builds remotely + put here your desired_token (for my case: Mercurial).
Because you can't trigger only to a specific branch in Mercurial, I found the branch name in this way. Also, to trigger from a remote script, you need to give in Jenkins read permission for anonymous overall, or create a specific user with credentials and put them into the trigger URL.
Bash example:
#!/bin/bash
BRANCH_NAME=`hg tip --template "{branch}"`
if [ $BRANCH_NAME = "branch_name" ]; then
curl --silent http://jenkins_domain:port/path/to/job?token=desired_token
fi
For the original question:
In this way you only execute one build, for a desired branch. Hooks are meant only for the main repository in case you work with multiple clones and multiple developers. You may have your local hooks, but don't trigger Jenkins from you local, for every developer. Trigger Jenkins only from the main repository when a push came (commit, incoming, and groupchange). Your local hooks are for other things, like email, logs, configuration, etc.