I have a jenkins setup with a big amount of jobs.
I'd like change all the jobs that clone from git with credentials X to credentials Y (which I already created)
Any way to do that programmatically?
If you haven't got a git credential helper configured (check with git config credential.helper), set one up (e.g. git config credential.helper store).
Then you can store a credential programmatically with
printf "username=<your_username>\npassword=<your_password>\nprotocol=https\nhost=your.git.host.net\n\n" | git credential approve
I ended up just doing a sed command on all the xml files in /var/lib/jenkins/jobs and replace the credentials id with the new id.
not so elegant but worked like a charm
Related
I'm trying to implement iOS pipeline to Azure DevOps using Fastlane. I have already have Fastlane in my project and successfully deploy beta and pilot versions. My problem is that when I run below script on Azure pipeline, It can't pass match clone part. Therefore, can't fetch certificates, provision profiles etc..
P.S: iOS_Certificates repo is different than project repo.
I'm getting timeout error after 1 hour. I think It is about authentication to
pool:
vmImage: 'macos-latest'
steps:
- script: |
fastlane match development --clone_branch_directly --verbose
fastlane beta
displayName: 'Build iOS'
Related code in MatchFile:
git_url("git#ssh.dev.azure.com:v3/myteam/myproject/certificates_repo")
storage_mode("git")
type("development")
EDIT: I'm trying to fetch a repo inside same project inside Azure DevOps (not GitHub or somewhere else). I'm getting timeout error, so no specific error even I run --verbose on match command.
From your information, you are using the SSH key as the authentication method.
Since you are using the macos-latest(microsoft-hosted agent) as build agent, the private key of ssh key will not exist on the target build machine.
So it can't authenticate and gets stuck. As you said, it will run 60 minutes and cancel. I could also reproduce this issue.
You could try to create a self-hosted agent and run the build on it.
In this case, you need to ensure that the private key exists on the machine， then you could authenticate through the ssh key.
On the other hand, you can authenticate with username and password.
For example(matchfile):
git_url "https://organizationname#dev.azure.com/organizationname/projectname/_git/reponame"
type "development"
app_identifier 'xxx'
username "member#companyname.com" #This will be the git username
ENV["FASTLANE_PASSWORD"] = "abcdefgh" #Password to access git repo.
ENV["MATCH_PASSWORD"] = "password" #Password for the .p12 files saved in git repo.
The following code is an "Execute Shell" build step in Jenkins. The job pulls from a repo which contains a file ranger-policies/policies.json. What I'd like to do is update that file (with a curl command, in this case) and then commit the change to source control and update the remote repo. The job successfully pulls from the remote repo in the "Source Code Management" section of the job configuration page over SSH using SSH keys. However, when the job gets to the "git push origin master" line in the "Execute Shell" step, I get a Permission denied (publickey) error, as if those same SSH keys which allowed me to successfully pull the repo are not available in the "Execute Shell" step when I want to push.
curl -X GET --header "text/json" -H "Content-Type: text/json" -u user:pass "http://my-url.com/exportJson" > ranger-policies/policies.json
git add ranger-policies/policies.json
git commit -m "udpate policies.json with latest ranger policies `echo "$(date +'%Y-%m-%d')"`"
git push origin master
I ended up figuring out how to make it work. The solution involves using the SSH Agent plugin. Here's a step-by-step that describes how I did it, hopefully it helps someone else:
First, create a new pipeline job.
Then, as hinted at in this post from Jenkins' documentation, go to the home screen for your new pipeline job, and click on "Pipeline Syntax." Choose "git: Git" as the "Sample Step, and enter the git repo you want to push to in the "Repository URL" field. Then choose the corresponding valid SSH keys for that repo from the "Credentials dropdown." Everything should look like this:
Grab the value of "credentialsId", highlighted with red in the above screenshot. You'll need it later.
Install the "Workspace Cleanup Plugin" (https://wiki.jenkins.io/display/JENKINS/Workspace+Cleanup+Plugin, optional) and the "SSH Agent Plugin" (https://jenkins.io/doc/pipeline/steps/ssh-agent/, not optional, required for this process to work).
Now go back to your new pipeline job and hit "Configure," which will take you to the screen where you define the job. Drop the following code into the "Pipeline" section ("Definition" should be set to "Pipeline script"): https://gist.github.com/ScottNeaves/5cdce294296437043b24f0f3f0a8f1d8
Drop your "credentialsId" into the appropriate places in the above Jenkinsfile, and fix up the repo names to target the repo you want, and you should be good to go.
Relevant documentation:
https://jenkins.io/doc/pipeline/examples/#push-git-repo
https://gist.github.com/blaisep/eb8aa720b06eff4f095e4b64326961b5#file-jenkins-pipeline-git-cred-md
https://issues.jenkins-ci.org/browse/JENKINS-28335?focusedCommentId=269000&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-269000
As per this gist, you need to set the remote origin url as per:
git remote set-url origin git#github.com:username/your-repository.git
I've set up some credentials in Jenkins for bitbucket and double-checked the Credentials settings (e.g. logging in manually) however when I try it in Jenkins it just spins forever giving this output:
> git config remote.origin.url <bitbucket url> # timeout=10
Fetching upstream changes from <bitbucket url>
> git --version # timeout=10
using GIT_ASKPASS to set credentials <bitbucket account email> Bitbucket
> git fetch --tags --progress <bitbucket url> +refs/heads/*:refs/remotes/origin/*
> git fetch --tags --progress <bitbucket url> +refs/heads/*:refs/remotes/origin/*
Note that the URL is fine when public. But when set to Private it simply fails with no output.
Is there anyway to debug this in a bit more detail?
I had a similar issue, with Jenkins on a Windows server. I installed git with credentials manager and whenever it tried to checkout a private repository, it would wait for me to input credentials manually in the server. Disabling the git credential manager fixed it for me.
I already had an option to input credentials in the git plugin so didn't need a separate credentials manager.
This is on MacOSX.
I changed the Jenkins setting on Git path to /usr/local/git as well as unset the credential.helper using git config, both don't work.
Finally, the problem was resolved by creating a default keychain file for jenkins in ~jenkins/Library/Keychains folder. Herewith is the steps...
sudo su jenkins
mkdir ~jenkins/Library/Keychains
cd ~jenkins/Library/Keychains
security create-keychain -p [pwd] ./Login.keychain
security login-keychain -d user -s ./Login.keychain
check default keychain setup properly
security default-keychain
git fetch --tags --progress https://github.com/....git +refs/heads/:refs/remotes/origin/
After that, the github userid/password is stored in jenkins default keychain and it will be used on jenkins build.
I had this problem on OSX. My issue was that Jenkins was using the wrong Git executable (I verified this by disabling the checkout step and adding which git before anything else).
I ran which git in terminal and copy-pasted the path into Manage Jenkins -> Global Tool Configuration -> Git -> Path to Git executable. It worked after that.
I had a similar issue, with Jenkins on a Windows server. I installed git with credentials manager and whenever it tried to checkout a private repository, it would wait for me to input credentials manually in the server. Disabling the git credential manager fixed it for me.
Actually, that should now (Q1 2021) work without having to disable the credential helper with Git 2.30.
"git credential(man)' didn't honor the core.askPass configuration variable (among other things), which has been corrected with Git 2.30 (Q1 2021).
See commit 567ad2c (15 Oct 2020) by Thomas Koutcher (koutcher).
(Merged by Junio C Hamano -- gitster -- in commit e0f6ad2, 02 Nov 2020)
credential: load default config
Signed-off-by: Thomas Koutcher
[jk: added test]
Signed-off-by: Jeff King peff#peff.net
Signed-off-by: Junio C Hamano gitster#pobox.com
Make git credential fill(man) honour the core.askPass variable.
As noted by kymikoloco in the comments, upgrading to the latest (Nov. 2021) Git for Windows 2.34 seems enough to solve the issue.
I had such problem with Jenkins on Windows 10. It always showed failure with permission denied error, code 128 returned by git.exe after ...GIT_ASKPASS for credentials step. Changing credential.helper and env.variable GIT_ASKPASS did not help me at all, the same behaviour.
Then i checked up which git.exe in cmd where git and was suprised, it was SmartGit's internal one.
I changed it to JGit in Jenkins global settings and it works now.
https://i.stack.imgur.com/IBfIi.png
Its works for me!
Goto Dashboard --> configuration
Scroll down to find Git plugin
input Global Config user.name Value: ex: jenkins
input Global Config user.email Value ex: youremail#gmail.com
you need to generate an SSH key from Git and add it to Bitbucket
I am trying to set git private repo on jenkins server. I have installed git plugin and also github. when I set repo url in jenkins project ui the error is
Failed to connect to repository : Command "git -c core.askpass=true
ls-remote -h git#github.com:repo/project.git HEAD" returned
status code 128:
stdout:
stderr: Permission denied (publickey).
fatal: The remote end hung up unexpectedl
What I have done up to now:
My server user and jenkin user( both are in same server) are different. Though it seems to me these are not related. jenkins user are given all credintial.
In my server under var/lib/jenkins/.ssh(.ssh is created by me) I added ssh key . Public key is added to github repo.
By swithcting user to jenkins i can clone the project by this ssh. So i think there is not any public key adding problem.
I have googled the problem. there are many solutions. I tried most of them. But still no solution. Probably I am missing something.
My repo url is something like this
git#github.com:repo/project.git
If your HOME set in /var/lib/jenkins/ then i hope all the step you have been done successfully :)
Then one thing may be happen for your case. Like when you switch the user by using:
su jenkins
This command means that you switch the user but the home directory will be same as a root's home!
So you need to switch user by confirming the specific user home also switched. TO doing so, you need to follow:
su -s /bin/bash jenkins
Then you need to generate either the ssh public key once again or just update the known host. This will work.
Related Link
It depends on what HOME is set to when Jenkins is running: git will look for the ssh (public and private) keys under $HOME/.ssh.
Simply add a build step with an echo $HOME, and make sure your .ssh is in that folder.
I found this gist, showing how to check out a pull request locally from GitHub.
I'm using bitbucket and I'm looking for a similar function.
Can you help me?
Thank you
One may fetch the code from Bitbucket Server's pull requests using:
git fetch origin refs/pull-requests/$PR_NO/from:$LOCAL_BRANCH
I found this answer and thought that it was actually possible to fetch refs for a pull request on bitbucket.
But it's not.
The answer for the OP's question is that it is NOT possible: there's been an open feature request issue about it that has been unanswered and unattended for four five SIX SEVEN years.
The workaround?
You can get the PR as a downloadable .patch file you can download and apply to a new branch you create manually. But you won't easily be able to apply updates.
I figured another way out, which I've implemented in git-repo, so everybody can use it. What I'm doing is use the API to get the PR's remote and branch, and automatically create a new upstream and branch locally. That way you can get updates from the PR poster. The downside is the clutter of git remotes.
edit: I hope this gets done and the feature request is closed. But there has been a solution for this on dedicated bitbucket servers for some time now, but not on the bitbucket.org service. On June 5th, a bitbucket staff member commented on this ticket:
Hi y'all -- thanks again for your feedback and patience on this issue. This feature is still high on the priority list in the backlog. When we have more information to share about the expected delivery of this feature, we will share it here.
I followed this article Pull request Fetching.
It worked but I found out I just need add one line to the current repo, rather than create a folk repo and an upstream repo. Run this line
git config --add remote.origin.fetch '+refs/pull-requests/*/from:refs/remotes/origin/pr/*'
You can also add it manually to the file .git/config in your project.
Next run git pull you should see a list:
[new ref] refs/pull-requests/488/from -> origin/pr/488
[new ref] refs/pull-requests/666/from -> origin/pr/666
Then you can run git checkout origin/pr/666 to get the pull request changes.
Fetch/Checkout Pull Requests
This works for bitbucket. Other server could have different refs: (refspecs) or no refs: at all.
First Time
First of all you need to add the pull request refs: of the remote repository. To do that to a repository (e.g. aliased 'upstream'):
git config --add remote.upstream.fetch '+refs/pull-requests/*/from:refs/remotes/upstream/pull-requests/*'
That is, you add the last line on git .config file:
[remote "origin"]
url = ssh://git#git.blablabla.net/~user/repository.git
fetch = +refs/heads/*:refs/remotes/origin/*
fetch = +refs/pull/*/head:refs/remotes/origin/pull-requests/*
Fetching
Then if you perform the remote fetch you should see the retrieval of (also) all the pull requests:
git fetch upstream
From ssh://git.blablabla.net/somepath/repository
* [new ref] refs/pull-requests/1188/from -> upstream/pull-requests/1188
* [new ref] refs/pull-requests/1741/from -> upstream/pull-requests/1741
* [new ref] refs/pull-requests/2394/from -> upstream/pull-requests/2394
Checking out
Finally you can checkout the pull-request you prefer:
git checkout pull-requests/2723
Successfully tested on dedicated bitbucket server 27/02/19.
When is not possible to checkout the pull request, a trick is that you can checkout the last commit of that pull request
git checkout <hash code of last commit>
If you use forks probably "origin" is your fork, so first of all you should add the main remote.
Take the URL of the main remote clicking the "Clone" button in the repository page the same way you do when you clone a repository
git remote add upstream $UPSTREAM_URL
fetch the pull request
git fetch upstream refs/pull-requests/$PR_NO/from:$LOCAL_BRANCH
checkout the new branch
git checkout $LOCAL_BRANCH
If you are using forked repository and you want to pull PR from original or other repo then use below commands.
1. git fetch ${URLofOriginalRepo}
“+refs/pull-requests/*/from:refs/remotes/origin/pr/*”
2. git checkout origin/pr/${PR_NUMBER}
URLOfOriginalRepo is the url repository from which you want to pull the PR. This url is the one you use to clone the repo using ssh key.
After running these command you can see the open PR's on this repo. Then pull the one you want.
for eg.
git fetch ssh://hostname.net:8080/repofolder/repo.git.git “+refs/pull-requests//from:refs/remotes/origin/pr/”
&&
git fetch origin/pr/854
It seems the easiest way to do this is still to get a patch of the pull request. Based on this question's answer, Alexandre's comment is still the only way to do this. It uses this BitBucket API call.
I used the following bash script:
USER=username
PASSWORD=password
REPO=repo-name
PULL_NO=42
OUTPUT_FILE=output.patch
# Add -i to include the HTTP-header in the output for debugging
curl -u $USER:$PASSWORD https://bitbucket.org/api/2.0/repositories/$USER/$REPO/pullrequests/$PULL_NO/patch -L -o $OUTPUT_FILE
Save that to a file called pull-patch.sh and fill in the environment variables with your account details. The script requires that you have curl installed (e.g. sudo apt install curl). Then run:
chmod +x pull-patch.sh
./pull-patch.sh
And a file called output.patch should be created from the pull request.
In bitbucket what you can do is
git config remote.origin.fetch "+refs/heads/:refs/remotes/origin/"
and then
git fetch
after that you can checkout to the branch you want to
git checkout BRANCH_NAME
P.S: Hopefully bitbucket will sort this out https://jira.atlassian.com/browse/BCLOUD-5814
I found this difficult in bit bucket, so, I tried a different approach. If a person give a pull request to my repo in bitbucket, i set his(bill) origin by him name(bill) . then to this -
git checkout -b bill-auth bill/bill-auth
Here bill is that contributer repo origin / link , then bill-auth is his branch name.
Here I am creating a branch same name as his(bill) feature branch.
git fetch origin refs/pull-requests/$PR_NO/merge - it works for Butbucket v5.14.1