How can I customize / override the "git clone" step in Travis CI? - travis-ci

On the install step, Travis CI clones the repo, which looks similar to this:
git clone --depth=50 --branch=master https://github.com/user/repo.git user/repo
How can I customize / override this?
Background: I am using tag based deploys. The way Travis checks out tagged builds (--branch=<tagname>), the git repository is in a detached state without access to branches. However, for deployment I need to know on which branch I am. My solution is to do a "normal" clone and then switch to the tagged commit.

You can clone the repository again in the install step. That way you clone the repository twice, but it seems to work.
# .travis.yml
install:
- git clone https://github.com/$TRAVIS_REPO_SLUG.git $TRAVIS_REPO_SLUG
- cd $TRAVIS_REPO_SLUG
- git checkout -qf $TRAVIS_COMMIT

Per the Travis docs you can add the following to your .travis.yml to remove the --depth flag:
git:
depth: false
As --depth implies --single-branch, removing this flag means that all branches will be checked out, which isn't the default behaviour.

I found that in order to get access to your whole repo you need the following:
install:
- git config remote.origin.fetch +refs/heads/*:refs/remotes/origin/*
- git fetch --unshallow --tags
This way you'll have access to remote branches and tags (e.g. can do checkout).
If you're on a tag but no longer want to be in a detached HEAD state you can create a new branch that points to the tag (according to this discussion):
install:
- git config remote.origin.fetch +refs/heads/*:refs/remotes/origin/*
- git fetch --unshallow --tags
- git symbolic-ref --short HEAD || git checkout -b ${TRAVIS_BRANCH}-test $TRAVIS_BRANCH
Note: git symbolic-ref --short HEAD will fail if you're in a detached HEAD state.

The problem is not really that you are in a detached branch. It is that git does not allow you to fetch the tags: git fetch --tags will only fetch the branch spcified by --branch in the git clone command you gave.
I explain this in more details this answer.
To solve you issue (checking out a specific tag) you can call a script that looks like this, after the repo is cloned:
# Keep track of where Travis put us.
# We are on a detached head, and we need to be able to go back to it.
build_head=$(git rev-parse HEAD)
# fetch the tags
git config --replace-all remote.origin.fetch +refs/heads/*:refs/remotes/origin/*
git fetch --tags
# checkout the tagged commit
git checkout -qf <your tag>
# now do your stuff
# go back to where we were at the beginning
git checkout ${build_head}

Run this during your build to have access to origin tags / branches
git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*" 1>/dev/null
git fetch origin -q
After that you can run this command to find branches containing your commit
BRANCHES=`git branch -a --contains "$TRAVIS_TAG"`
I've created a script a loooong time ago to fetch the 'environment' branch where the tag was created for continuous deployment purpose.
It may inspire you: https://gist.github.com/rolebi/a0eb1f783b7f3a5f21a631c8da1582dc
Use it like that:
TARGET_ENV="`test $TRAVIS_TAG && bash scripts/get_branch_for_git_reference.sh $TRAVIS_TAG`"

Disable git clone and then clone the repository again in the install step. The repository will be cloned only once in this way. In this "normal" clone you will able to do what ever you want.
git:
clone: false
install:
- git clone https://github.com/$TRAVIS_REPO_SLUG.git $TRAVIS_REPO_SLUG
- cd $TRAVIS_REPO_SLUG

Or you could just query the remote. Add the following to .travis.yml:
env:
global:
# get all the branches referencing this commit
- REAL_BRANCH=$(git ls-remote origin | sed -n "\|$TRAVIS_COMMIT\s\+refs/heads/|{s///p}")
# or check if we are on a particular branch:
- IS_RELEASE=$(git ls-remote origin | grep "$TRAVIS_COMMIT\s\+refs/heads/release$"
(I am surprised that the git gurus hadn’t come up with this one already)

You can convert the already existing shallow clone to a full clone. To do so execute git fetch --unshallow (available since git version 1.8.3) during the install step.
# .travis.yml
install:
- git fetch --unshallow --tags
The --tags flag forces to fetch all tags even if they don't belong to the checked out branch. This is needed if your build also depends on tags from other branches.

Related

libgit2sharp - How to checkout as "git checkout <commit_hash> ."

With git command, to replace the working directory with a commit, we will call:
git checkout <commit_hash> .
How can I implement that with libgit2sharp?

In gerrit, how to clone project from HEAD:refs/for/master

I try to use gerrit to do some test.
Use "git push origin HEAD:refs/for/master",
before gerrit code review, I want to do some test by Jenkins.
How to use command to clone the commit from "refs/for/master"?
or fetch to Jenkins job workspace in order to test?
There is only one branch.
"git branch" -> *master
When someone push a commit to review on Gerrit, the commit stays on the "magical" refs/for branch until it's finally submitted (merged). To have the commit locally you need to execute one of the download commands:
Checkout:
git fetch "https://GERRIT-SERVER/a/REPO-PATH" refs/changes/CHANGE-NUMBER && git checkout FETCH_HEAD
Cherry Pick:
git fetch "https://GERRIT-SERVER/a/REPO-PATH" refs/changes/CHANGE-NUMBER && git cherry-pick FETCH_HEAD
Format Patch:
git fetch "https://GERRIT-SERVER/a/REPO-PATH" refs/changes/CHANGE-NUMBER && git format-patch -1 --stdout FETCH_HEAD
Pull:
git pull "https://GERRIT-SERVER/a/REPO-PATH" refs/changes/CHANGE-NUMBER
On Gerrit 3.0, you you'll find these commands at the change screen, clicking on the 3 dots at the up-right and then in the "Download patch" option.
See more info at the Gerrit documentation here.
Using Jenkins, you can execute some of these commands too, but it's easier to use the Gerrit Trigger plugin.

Travis CI loading private submodule

At the moment I'm troubling with Travis-CI Pro and a private submodule.
This is some travis.yml code I've found, the $GIT_USER and $GIT_TOKEN envs are set in travis settings.
git:
submodules: false
before_install:
- sed -i 's/git#github.com:/https:\/\/$GIT_USER:$GIT_TOKEN#github.com\//' .gitmodules
- git submodule update --init --recursive
During the build process I get the following error:
$ sed -i 's/git#github.com:/https:\/\/$GIT_USER:$GIT_TOKEN#github.com\//' .gitmodules
0.67s$ git submodule update --init --recursive
Submodule 'ro-realm' (https://github.com/[secure]/ro-realm.git) registered for path 'ro-realm'
Cloning into '/home/travis/build/[secure]/ro-order-worker/ro-realm'...
remote: Invalid username or password.
fatal: Authentication failed for 'https://github.com/[secure]/ro-realm.git/'
Thanks for your help.
I wouldn't call this a recommended solution, but how about this to fix the issue?
# Generate from github, confirm that user and token are correct
echo https://${GIT_USER}:${GIT_TOKEN}#github.com/chaconinc/DbConnector
git submodule add https://${GIT_USER}:${GIT_TOKEN}#github.com/chaconinc/DbConnector
git submodule update --init --recursive

Jenkins - Build loop

Solutions
When configing the branchs to build remove the wildcard *:
*/master to be only master
This error only occours when a GIT SCM is related to the JOB.
Ex: I have a daily job to build an Android project.
First i thought that it was a cron problem, currently im using 50 10 * * *, but i have tried H 10 * * * without any success. When the time comes, it always build the job (Failure or Success) and it queue another job.. and so on...
Would last have run at Friday, May 22, 2015 10:01:58 AM BRT; would next run at Saturday, May 23, 2015 10:01:58 AM BRT.
Even a build with parameter it queue a another job everytime.
So it stays in a build loop, forever..
Config:
Jenkins ver. 1.614.
Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-46-generic x86_64)
GIT client plugin: 1.17.1
GIT plugin: 2.3.5
Console Output:
Started by an SCM change
Building in workspace .../jobs/PROJECT_NAME/workspace
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url ssh://GIT_ADDRESS/ # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Fetching upstream changes from ssh://GIT_ADDRESS/
> git --version # timeout=10
> git -c core.askpass=true fetch --tags --progress ssh://GIT_ADDRESS/ +refs/heads/master:refs/remotes/origin/master
> git rev-parse refs/remotes/origin/master^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Multiple candidate revisions
Scheduling another build to catch up with PROJECT_NAME
Checking out Revision e6d726c2d31cb0d7e6fad4362ee85e6fac1712c6 (refs/remotes/origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f e6d726c2d31cb0d7e6fad4362ee85e6fac1712c6
> git rev-list e6d726c2d31cb0d7e6fad4362ee85e6fac1712c6 # timeout=10
[Gradle] - Launching build.
[android] $ gradle clean build assemble
...
Maybe the problem is this?
> git rev-parse refs/remotes/origin/master^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Multiple candidate revisions
Scheduling another build to catch up with PROJECT_NAME
The build loop:
Job Config:
Global Config:
Review the beginning of the console logs, for both the job that you manually started, and the job that gets queued next automatically.
In the beginning of the console log, it gives a reason for why the build was triggered, for example 09:46:05 Started by an SCM change or 16:06:58 Started by user Slav. Please add to the question the start of both console logs.
Edit:
Now that we know that Jenkins detects an SCM change trigger we can further speculate that what's causing this behaviour is an SCM post-commit hook of some sort, or another script activation sending a request to Jenkins.
Post-commit hooks should not work if SCM polling is not enabled, but nothing else makes sense, so let's investigate that avenue. Add /pollingLog/ to the build URL of the build number that was triggered "by SCM change". There is a chance that nothing would be found, but if there is, but let's make sure.
Next, if possible, bind Jenkins to a different IP address and/or port. If there are "rogue" scripts that are triggering secondary builds, they won't be able to adapt to changed address.
Edit 2:
Looks like there is a bug in git client plugin itself, it was fixed in version 1.6.2. Please check the version of your git plugins.
Sources: https://issues.jenkins-ci.org/browse/JENKINS-10767 and https://issues.jenkins-ci.org/browse/JENKINS-20286
Above screen shot looks fine but i am sure there must be some other configuration.
can you provide detail (rest of the) configuration too?
Please give below code in Build periodically section and try if that resolves the issue
H(0-50) 10 * * *
Make sure your repo doesn't have more than one branch that matches your Branch Specifier by using the full path to the branch: "refs/heads/master" instead of "*/master"

forget to move to another branch than master with Git

I'm using Git with a rails project.
Today i make a lot of changes and i forget to move to another branch, so all these changes are in the master branch but not committed yet.
this is what i get when i do git status command :
# On branch master
# Changes not staged for commit:
# (use "git add <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working directory)
#
# modified: app/controllers/sessions_controller.rb
# modified: app/controllers/users_controller.rb
# modified: app/helpers/sessions_helper.rb
# modified: app/models/user.rb
# .....
#
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# app/assets/javascripts/password_resets.js.coffee
# app/assets/stylesheets/password_resets.css.scss
# app/controllers/password_resets_controller.rb
# .....
# ......
no changes added to commit (use "git add" and/or "git commit -a")
how i can solve this error and move all these untracked files from master to another branch already exists ? thank you.
Just create a new branch and then commit, e.g.:
git checkout -b my_branch
git commit -am "My commit."
If there are no commits marked C1, C2, etc. below
You're in the simple case; just use Henrik N's answer.
If there is a tracking branch
Your updated question/comments suggests that you have both committed changes and uncommitted changes, and you did them all on some tracking branch, such as master that tracks origin/master or devel that tracks origin/devel, but that you have not pushed any of them yet. (I'll assume master but you can just change each master to the feature/development/whatever branch name below.)
As a diagram, your repo's commit tree and working directory/index now looks something like this:
M1 -...- M5 -- M6 <-- origin/master
\
C1 -- C2 <-- master, HEAD=master
\
i, w <-- "index" and working-tree files
Here i is the stuff in the index (git add dir/file1, git rm file2, etc), which git status shows as changes to be committed, and w is the stuff in your working directory that git status shows as changes not staged for commit.
Here's what you want it to look like:
M1 -...- M5 -- M6 <-- master, origin/master
\
C1 -- C2 <-- my_branch, HEAD=my_branch
\
i, w <-- "index" and working-tree files
Remember that branch labels are like sticky-notes ("yellow stickies" or "Post-It® notes" or whatever): they have a name written on them and are pasted on to a commit. So what you have to do is add a new sticky-note, my_branch, pointing to commit C2, and also make HEAD refer to the new branch name:
git checkout -b my_branch
This uses the previous value of HEAD to name the commit that my_branch is pasted-on-to. Since that commit is C2 you now have both master and my_branch here. It then rewrites HEAD to point to the name my_branch. Now you have:
M1 -...- M5 -- M6 <-- origin/master
\
C1 -- C2 <-- master, my_branch, HEAD=my_branch
\
i, w <-- "index" and working-tree files
Now you need to move the sticky-note labeled master to point back where origin/master points. There are two ways to do this, with git reset, or with git branch. Using git reset is harder, so let's do it with git branch:
git branch -f master origin/master
The -f (force) flag tells git branch to change an existing sticky-note, rather than failing because it exists, and origin/master gives which commit master should point-to: so git peels the label off C2 and pastes it on M6, and you have the setup you wanted (the 2nd diagram).
Summary: it's just two commands:
git checkout -b my_branch # create new branch and change HEAD
git branch -f master origin/master # restore master to origin/master
You're now ready to check in changes as usual, on branch my_branch. (git add, git rm, git commit as needed.) That will add a new commit C3:
M1 -...- M5 -- M6 <-- master, origin/master
\
C1 -- C2 -- C3 <-- my_branch, HEAD=my_branch
What if there is no tracking branch?
Well, that's OK, it's just a little harder. Go back to the "have now" and "want" diagrams. In these diagrams, the commit I labeled M6 is the one you want master to name. You now have to find M6. Commits have these long hexadecimal-numeric (SHA-1) "true names" that never change, like 5e013711f5d6eb3f643ef562d49a131852aa4aa1 for example. (git rev-parse HEAD will show the "true name" for the current, i.e., HEAD, commit.) You can find that "true name", or you can find some alternate or abbreviated name that also names the same thing.
There are a whole lot of options for names, but let's stick with numbers that come out of git log and git log --oneline. The latter gives you something like this:
97206f5 peerish: set socket options earlier
4881af5 add semtest.c
b3f8bea fdm: repair example
where the number is an abbreviated SHA-1 and the text is the first line of the commit message. Since the logs are shown in reverse order, the top line is the most recent commit (C2 in the example above), and then each next line is the next one back (well, it's more complicated with merges, but close enough).
Let's say that from the above, you can immediately tell that the commit I labeled M6 is b3f8bea, i.e., that you need to skip two commits. Simply supply that value as the last argument to git branch:
git branch -f master b3f8bea
If your repo is huge and old, or has a lot of merge commits, you might want your logs to include more "decoration". I got these aliases from somewhere long ago:
[alias]
lol = git log --graph --decorate --oneline
lola = git log --graph --decorate --oneline --all
so that I can run git lol and git lola.
Alternatively, if you know for sure there were exactly two commits, master~2 (and in this case, after adding my_branch, my_branch~2) will get you the same SHA-1. The git rev-parse command shows you exactly which SHA-1 any given name "means". In one repo I have here, HEAD~2 is the same as origin/featureX:
$ git rev-parse HEAD~2
0f5a13497dd3da8aff8e452c8f56630f83253e79
$ git rev-parse origin/featureX
0f5a13497dd3da8aff8e452c8f56630f83253e79
but in general git log (or git lol, etc.) will get you the raw SHA-1 and you can add labels from there, as needed.
You simply need to create new branch.
Once you create a branch all the changes will be in the new branch.
and then clean your old stuff from master
// Checkout new branch
git checkout -b my_branch
// Prepare files for commit
git add .
// Commit changes
git commit -m "Your commit message"
// delete the dirty master
git branch -D master
// Now we are going to clean your changes from the master branch
// the -f is important in case you skipped the previous command to delete your master
git checkout -f master

Resources