I've a pipeline that is cloning another repository. Pipeline below is running on yyy repository, and it's supposed to clone xxx repository and do some work on it, then upload the results to AWS S3, I have lots of object to be worked on, so git clone step should be LFS enabled. git lfs client is configured on my base image. LFS is also enabled on my xxx repository. My deployment consumes so much time to clone repo with LFS enabled.
- step:
name: "Pipeline"
services:
- docker
caches:
- docker
script:
- ...
- ...
- ...
- ...
- git clone git#bitbucket.org:xxx.git
I've tried
clone:
lfs: true
option to reduce the consumed time but it seems it doesn't work with the setup above.
How can I reduce time consumption for this setup? Is there any workaround here?
Thank you.
It can be a little bit weird but I came up with the solution below,
I gave
clone:
enabled: false
in the pipeline, and I cloned the repo in the script part. Because my repo is huge, I limited everything to consume less time.
git clone git#bitbucket.org:xxx/<repo-name>.git --depth 1 --no-tags --single-branch -b <branch> --no-checkout .
It's working like a charm :)
Related
i build gerrit server using apache2.
i has 'git push' other lfs files successfully, now i want to upload over 8.1Gb lfs files，but it stuck at 8.1Gb, no return and no error.
android#gerrit-server:~/work/project$ git push http://username#gerrit-server:8888/vendor/qcom/boot_images HEAD:refs/heads/releases-branch**
Locking support detected on remote "http://username#gerrit-server:8888/vendor/qcom/boot_images". Consider enabling it with: $ git config lfs.http://gerrit-server:8888/vendor/qcom/boot_images.git/info/lfs.locksverify true Uploading LFS objects: 98% (9892/10045), 8.1 GB | 0 B/s
Just like that, it always stuck that
I have a repository that uses git submodules, and I configured the automated build on Docker Hub. At the beginning of the build process, it looks like Docker Hub pulls the repository from the default branch (master), update submodules and then checkout to the particular branch (let's say branch feature-a) that triggered the build. It works fine if feature-a branch has the very same submodules as master, but if the submodules are different (let's say, pull one submodule from a different repo), the build fails.
Is there a way to make Docker Hub clone the correct branch directly?
You need to use hooks: https://docs.docker.com/docker-hub/builds/advanced/#custom-build-phase-hooks
TL;DR: Place this in hooks/post_checkout:
#!/bin/bash
# Docker hub does a recursive clone, then checks the branch out,
# so when a PR adds a submodule (or updates it), it fails.
git submodule update --init
It might be failing because the submodule is private.
You can add a build environment variable SSH_PRIVATE. And add a private key with access to the private submodule repository.
A word of caution though… you may want to generate a diff private key than the one you use for anything else and add that to the private submodules repo.
Edit: This is required even if your linked github account has access to the repo because you're most likely specifying the submodule url as ssh based (e.g. git#github.com:Account/repo.git)
Edit2: Adding docs https://docs.docker.com/docker-hub/builds/#build-repositories-with-linked-private-submodules
SSH_PRIVATE sugested by Clintm (and in official documention) doesn't work for us. As far as I understand, this is because Docker hub interface to set environment variables doesn't allow to fill in a line break (!?)
I spent time on my side to find a workaround that match with our needs and it works for us.
I let it here if it can help some of you.
/hooks/post_checkout
#!/bin/bash
# Docker documentation to set up private git submodule for build does not work
# since it's not possible to set environment variable with line break in Docker
# Hub interface. It makes impossible to set SSH_PRIVATE as suggested here:
# https://docs.docker.com/docker-hub/builds/#build-repositories-with-linked-private-submodules
#
# To use the script below:
# - As suggested in the official doc, generate a keypair and push the public
# part to your source code provider account
# - In Docker Hub build configuration, set var SSH_PRIVATE_ESCAPED with the output of
# `awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' /path/to/the/private/key`
SSH_PRIVATE_FILE=~/git_id_rsa
echo "${SSH_PRIVATE_ESCAPED}" | awk '{gsub(/\\n/,"\n")}1' > "${SSH_PRIVATE_FILE}"
chmod 0400 "${SSH_PRIVATE_FILE}"
export GIT_SSH_COMMAND="ssh -i ${SSH_PRIVATE_FILE}"
git submodule update --init
I've got a Lektor site that I'm trying to deploy automatically in response to pull requests and commits, using the Travis CI trigger approach from the Lektor docs.
The Lektor configuration works fine from the command line.
The Travis build starts, and appears to build the site without problem - but when it gets to deployment, the log says the following:
Installing deploy dependencies
!!! Script support is experimental !!!
Preparing deploy
Cleaning up git repository with `git stash --all`. If you need build artifacts for deployment, set `deploy.skip_cleanup: true`. See https://docs.travis-ci.com/user/deployment/#Uploading-Files.
No local changes to save
Deploying application
Deploying to ghpages-https
Build cache: /home/travis/.cache/lektor/builds/d3a411e13041731555222b901cff4248
Target: ghpages+https://pybee/pybee.github.io?cname=pybee.org
Initialized empty Git repository in /home/travis/build/pybee/pybee.github.io/temp/.deploytemp9xhRDc/scratch/.git/
Fetching origin
fatal: repository 'https://github.com/pybee/pybee.github.io/' not found
error: Could not fetch origin
fatal: repository 'https://github.com/pybee/pybee.github.io/' not found
Done!
For a full log, see here.
I've checked the credentials in the Travis CI configuration for the repository; I'm as certain as I can be that they're correct. I've tried using the same configuration (exporting LEKTOR_DEPLOY_USERNAME and LEKTOR_DEPLOY_PASSWORD locally), and it works fine.
hammer:pybee.org rkm$ lektor deploy ghpages-https
Deploying to ghpages-https
Build cache: /Users/rkm/Library/Caches/Lektor/builds/a269cf944d4302f15f78a1dfb1602486
Target: ghpages+https://pybee/pybee.github.io?cname=pybee.org
Initialized empty Git repository in /Users/rkm/projects/beeware/pybee.org/temp/.deploytempOh4p98/scratch/.git/
Fetching origin
From https://github.com/pybee/pybee.github.io
* [new branch] master -> origin/master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Everything up-to-date
Done!
Any suggestions on the cause of this error?
It turns out this is a bug in Lektor.
If you use the following in your <project>.lektorproject:
[servers.ghpages-https]
target = ghpages+https://pybee/pybee.github.io?cname=pybee.org
and the following in your .travis.yml:
language: python
python: 2.7
cache:
directories:
- $HOME/.cache/pip
- $HOME/.cache/lektor/builds
install: "pip install git+https://github.com/singingwolfboy/lektor.git#fix-ghpages-https-deploy#egg=lektor"
script: "lektor build"
deploy:
provider: script
script: "lektor deploy ghpages-https"
on:
branch: lektor
(i.e., use the PR branch for deployment), builds will deploy as expected.
On the install step, Travis CI clones the repo, which looks similar to this:
git clone --depth=50 --branch=master https://github.com/user/repo.git user/repo
How can I customize / override this?
Background: I am using tag based deploys. The way Travis checks out tagged builds (--branch=<tagname>), the git repository is in a detached state without access to branches. However, for deployment I need to know on which branch I am. My solution is to do a "normal" clone and then switch to the tagged commit.
You can clone the repository again in the install step. That way you clone the repository twice, but it seems to work.
# .travis.yml
install:
- git clone https://github.com/$TRAVIS_REPO_SLUG.git $TRAVIS_REPO_SLUG
- cd $TRAVIS_REPO_SLUG
- git checkout -qf $TRAVIS_COMMIT
Per the Travis docs you can add the following to your .travis.yml to remove the --depth flag:
git:
depth: false
As --depth implies --single-branch, removing this flag means that all branches will be checked out, which isn't the default behaviour.
I found that in order to get access to your whole repo you need the following:
install:
- git config remote.origin.fetch +refs/heads/*:refs/remotes/origin/*
- git fetch --unshallow --tags
This way you'll have access to remote branches and tags (e.g. can do checkout).
If you're on a tag but no longer want to be in a detached HEAD state you can create a new branch that points to the tag (according to this discussion):
install:
- git config remote.origin.fetch +refs/heads/*:refs/remotes/origin/*
- git fetch --unshallow --tags
- git symbolic-ref --short HEAD || git checkout -b ${TRAVIS_BRANCH}-test $TRAVIS_BRANCH
Note: git symbolic-ref --short HEAD will fail if you're in a detached HEAD state.
The problem is not really that you are in a detached branch. It is that git does not allow you to fetch the tags: git fetch --tags will only fetch the branch spcified by --branch in the git clone command you gave.
I explain this in more details this answer.
To solve you issue (checking out a specific tag) you can call a script that looks like this, after the repo is cloned:
# Keep track of where Travis put us.
# We are on a detached head, and we need to be able to go back to it.
build_head=$(git rev-parse HEAD)
# fetch the tags
git config --replace-all remote.origin.fetch +refs/heads/*:refs/remotes/origin/*
git fetch --tags
# checkout the tagged commit
git checkout -qf <your tag>
# now do your stuff
# go back to where we were at the beginning
git checkout ${build_head}
Run this during your build to have access to origin tags / branches
git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*" 1>/dev/null
git fetch origin -q
After that you can run this command to find branches containing your commit
BRANCHES=`git branch -a --contains "$TRAVIS_TAG"`
I've created a script a loooong time ago to fetch the 'environment' branch where the tag was created for continuous deployment purpose.
It may inspire you: https://gist.github.com/rolebi/a0eb1f783b7f3a5f21a631c8da1582dc
Use it like that:
TARGET_ENV="`test $TRAVIS_TAG && bash scripts/get_branch_for_git_reference.sh $TRAVIS_TAG`"
Disable git clone and then clone the repository again in the install step. The repository will be cloned only once in this way. In this "normal" clone you will able to do what ever you want.
git:
clone: false
install:
- git clone https://github.com/$TRAVIS_REPO_SLUG.git $TRAVIS_REPO_SLUG
- cd $TRAVIS_REPO_SLUG
Or you could just query the remote. Add the following to .travis.yml:
env:
global:
# get all the branches referencing this commit
- REAL_BRANCH=$(git ls-remote origin | sed -n "\|$TRAVIS_COMMIT\s\+refs/heads/|{s///p}")
# or check if we are on a particular branch:
- IS_RELEASE=$(git ls-remote origin | grep "$TRAVIS_COMMIT\s\+refs/heads/release$"
(I am surprised that the git gurus hadn’t come up with this one already)
You can convert the already existing shallow clone to a full clone. To do so execute git fetch --unshallow (available since git version 1.8.3) during the install step.
# .travis.yml
install:
- git fetch --unshallow --tags
The --tags flag forces to fetch all tags even if they don't belong to the checked out branch. This is needed if your build also depends on tags from other branches.
So I am very new to Github. Please forgive me if i am asking a stupid question. So first did some reading on Github and tried to understand the philosophy about it. So i am working on my first ruby on rails app (very excited) I generated my new app and made my first commit by pushing the empty app to Github. So now i have a master branch.
Then i created a couple of models and added some fields to the tables i made. Now i would like to push this change on Github . What do i do and how? I KNOW some of you will say why would you push a change so small but i am just trying to learn GIT so then when i work on huge projects i am ready.
I basically want to commit the changes to the masters . How do i do this? I USUALLY see other peoples gits and they have messages like "fixed feature 1" 1 hour ago etc. So i want to do the same.
Please advice.
You would do that just like you pushed your earlier code - by pushing the commits you have made to remote repository.
If you are on the master branch, this will do: git push origin master (replace origin with your remote name - git remote -v will tell you all your remote names).
If you are on a feature branch, you can checkout to the master branch and then merge your feature branch - git checkout master and then git merge <feature branch>, and then push your changes again using git push origin master.
EDIT:
You can add a shortname using git remote add origin <link to remote repository>, and then use origin to push.
I am new to git as well and the answer above sounds better then mine. I have bash script on github (https://github.com/caroldomokos/columbo) which is work in progress for some time. I have the master locally as you do. When I make a change on the script I first do "git commit -a" and then "git push". I am in the "columbo" forlder on my pc. If you add a new file you first have to tell git to add it: "git add ". You can always use "git status" to see what is tracked and what has changed.To examplify I created this small sequence for you :-)
hpbcadom#LUBUNTU32:~/columbo$ echo "Git example" > example_file
hpbcadom#LUBUNTU32:~/columbo$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
(use "git add ..." to include in what will be committed)
example_file
nothing added to commit but untracked files present (use "git add" to track)
hpbcadom#LUBUNTU32:~/columbo$ git commit -a
[master d799b3a] to help I hope
1 file changed, 1 insertion(+)
create mode 100644 example_file
hpbcadom#LUBUNTU32:~/columbo$ git push
warning: push.default is unset; its implicit value is changing in
Git 2.0 from 'matching' to 'simple'. To squelch this message
and maintain the current behavior after the default changes, use:
git config --global push.default matching
To squelch this message and adopt the new behavior now, use:
git config --global push.default simple
When push.default is set to 'matching', git will push local branches
to the remote branches that already exist with the same name.
In Git 2.0, Git will default to the more conservative 'simple'
behavior, which only pushes the current branch to the corresponding
remote branch that 'git pull' uses to update the current branch.
See 'git help config' and search for 'push.default' for further information.
(the 'simple' mode was introduced in Git 1.7.11. Use the similar mode
'current' instead of 'simple' if you sometimes use older versions of Git)
Username for 'xxxs://github.com': caroldomokos
Password for 'xxxs://caroldomokos#github.com':
Counting objects: 4, done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 288 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
To xxxs://github.com/caroldomokos/columbo
dadf5af..d799b3a master -> master
I hope it helps :-)