Before deployment, (or after, but this is harder as we are deploying to s3), we need to rename staging.robots.txt to robots.txt (overwriting the default robots.txt) for the staging deployment only, so that we can block crawling on our staging server (but allow it on production).
Any idea if this is possible?
On the Travis documentation site, there is no info on the before_deploy stage, and we cant see any feature to rename files. With Jenkins, I would simply put cp xxx yyy or similar in the build script, as I know my Jenkins is running on Ubuntu, but we don't know the equivalent Travis command for the .travis.yml file.
== UPDATE ==
Having done more research, it might be possible to do this through a script e.g. commit move.sh into your repo, then call it. As you can choose what OS the build is done one (e.g. Linux), you can write the script for that platform. However, it's not clear at what point you can call this script in the .yml file.
You can simply write a script to invoke in your .travis.yml file for deployment. See the documentation.
Here's the example copied from these docs:
deploy:
provider: script
script: scripts/deploy.sh
on:
tags: true
branch: master
The above config for deploy would invoke on tagging the master branch and the the script (scripts/deploy.sh) would be invoked.
Other than that, you can simply write this command under before_install section like this:
before_install:
- mv abc.txt xyz.txt
You've used cp command but you are talking about renaming, not copying. So, I've used mv command to rename the file.
If you want to do something at the end, you can add an after_success section as well.
Hope that helps!
Related
We are thinking to move our ci from jenkins to gitlab. We have several projects that have the same build workflow. Right now we use a shared library where the pipelines are defined and the jenkinsfile inside the project only calls a method defined in the shared library defining the actual pipeline. So changes only have to be made at a single point affecting several projects.
I am wondering if the same is possible with gitlab ci? As far as i have found out it is not possible to define the gitlab-ci.yml outside the repository. Is there another way to define a pipeline and share this config with several projects to simplify maintainance?
GitLab 11.7 introduces new include methods, such as include:file:
https://docs.gitlab.com/ee/ci/yaml/#includefile
include:
- project: 'my-group/my-project'
ref: master
file: '/templates/.gitlab-ci-template.yml'
This will allow you to create a new project on the same GitLab instance which contains a shared .gitlab-ci.yml.
First let me start by saying: Thank you for asking this question! It triggered me to search for a solution (again) after often wondering if this was even possible myself. We also have like 20 - 30 projects that are quite identical and have .gitlab-ci.yml files of about 400 - 500 loc that have to each be changed if one thing changes.
So I found a working solution:
Inspired by the Auto DevOps .gitlab-ci.yml template Gitlab itself created, and where they use one template job to define all functions used and call every before_script to load them, I came up with the following setup.
Multiple project repo's (project-1, project-2) requiring a shared set of CI jobs / functions
Functions script containing all shared functions in separate repo
Files
So using a shared ci jobs scipt:
#!/bin/bash
function list_files {
ls -lah
}
function current_job_info {
echo "Running job $CI_JOB_ID on runner $CI_RUNNER_ID ($CI_RUNNER_DESCRIPTION) for pipeline $CI_PIPELINE_ID"
}
A common and generic .gitlab-ci.yml:
image: ubuntu:latest
before_script:
# Install curl
- apt-get update -qqq && apt-get install -qqqy curl
# Get shared functions script
- curl -s -o functions.sh https://gitlab.com/giix/demo-shared-ci-functions/raw/master/functions.sh
# Set permissions
- chmod +x functions.sh
# Run script and load functions
- . ./functions.sh
job1:
script:
- current_job_info
- list_files
You could copy-paste your file from project-1 to project-2 and it would be using the same shared Gitlab CI functions.
These examples are pretty verbose for example purposes, optimize them any way you like.
Lessons learned
So after applying the construction above on a large scale (40+ projects) I want to share some lessons learned so you don't have to find out the hard way:
Version (tag / release) your shared ci functions script. Changing one thing can now make all pipelines fail.
Using different Docker images could cause an issue in the requirement for bash to load the functions (e.g. I use some Alpine-based images for CLI tool based jobs that have sh by default)
Use project based CI/CD secret variables to personalize build jobs for projects. Like environment URL's etc.
Since gitlab version 12.6, it's possible define a external .gitlab-cy.yml file.
To customize the path:
Go to the project's Settings > CI / CD.
Expand the General pipelines section.
Provide a value in the Custom CI configuration path field.
Click Save changes.
...
If the CI configuration will be hosted on an external site, the URL link must end with .yml:
http://example.com/generate/ci/config.yml
If the CI configuration will be hosted in a different project within
GitLab, the path must be relative to the root directory in the other
project, with the group and project name added to the end:
.gitlab-ci.yml#mygroup/another-project
my/path/.my-custom-file.yml#mygroup/another-project
Use include feature, (available from GitLab 10.6):
https://docs.gitlab.com/ee/ci/yaml/#include
So, i always wanted to post, with what i came up with now:
Right now we use a mixed approach of #stefan-van-gastel's idea of a shared ci library and the relatively new include feature of gitlab 11.7. We are very satisfied with this approach as we can now manage our build pipeline for 40+ repositories in a single repository.
I have created a repository called ci_shared_library containing
a shell script for every single build job containing the execution logic for the step.
a pipeline.yml file containing the whole pipeline config. In the before script we load the ci_shared_library to /tmp/shared to be able to execute the scripts.
stages:
- test
- build
- deploy
- validate
services:
- docker:dind
before_script:
# Clear existing shared library
- rm -rf /tmp/shared
# Get shared library
- git clone https://oauth2:${GITLAB_TOKEN}#${SHARED_LIBRARY} /tmp/shared
- cd /tmp/shared && git checkout master && cd $CI_PROJECT_DIR
# Set permissions
- chmod -R +x /tmp/shared
# open access to registry
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
test:
stage: test
script:
- /tmp/shared/test.sh
build:
stage: build
script:
- /tmp/shared/build.sh
artifacts:
paths:
- $CI_PROJECT_DIR/target/RPMS/x86_64/*.rpm
expire_in: 3h
only:
- develop
- /release/.*/
deploy:
stage: deploy
script:
- /tmp/shared/deploy.sh
artifacts:
paths:
- $CI_PROJECT_DIR/tmp/*
expire_in: 12h
only:
- develop
- /release/.*/
validate:
stage: validate
script:
- /tmp/shared/validate.sh
only:
- develop
- /release\/.*/
Every project that want's to use this pipeline config has to have a .gitlab-ci.yml. In this file the only thing to do is to import the shared pipeline.yml file from the ci_shared_library repo.
# .gitlab-ci.yml
include:
- project: 'ci_shared_library'
ref: master
file: 'pipeline.yml'
With this approach really everything regarding to the pipeline lives in one single repository and is reusable. We have the whole pipeline-template in one file, but i think it would even be possible to split this up to have every single job in a yml-file. This way it would be more flexible and one could create default jobs that can be merged together differently for projects that have similar jobs but not every project needing all jobs...
With GitLab 13.5 (October 2020), the include feature is even more useful:
Validate expanded GitLab CI/CD configuration with the API
Writing and debugging complex pipelines is not a trivial task. You can use the include keyword to help reduce the length of your pipeline configuration files.
However, if you wanted to validate your entire pipeline via the API previously, you had to validate each included configuration file separately which was complicated and time consuming.
Now you have the ability to validate a fully-expanded version of your pipeline configuration through the API, with all the include configuration included.
Debugging large configurations is now easier and more efficient.
See Documentation and Issue.
And:
See GitLab 13.6 (November 2020)
Include multiple CI/CD configuration files as a list
Previously, when adding multiple files to your CI/CD configuration using the include:file syntax, you had to specify the project and ref for each file. In this release, you now have the ability to specify the project, ref, and provide a list of files all at once. This prevents you from having to repeat yourself and makes your pipeline configuration less verbose.
See Documentation) and Issue.
You could look into the concept of Dynamic Child pipeline.
It has evolved with GitLab 13.2 (July 2020):
Dynamically generate Child Pipeline configurations with Jsonnet
We released Dynamic Child Pipelines back in GitLab 12.9, which allow you to generate an entire .gitlab-ci.yml file at runtime.
This is a great solution for monorepos, for example, when you want runtime behavior to be even more dynamic.
We’ve now made it even easier to create CI/CD YAML at runtime by including a project template that demonstrates how to use Jsonnet to generate the YAML.
Jsonnet is a data templating language that provides functions, variables, loops, and conditionals that allow for fully parameterized YAML configuration.
See documentation and issue.
I wrote a python script, which uses boto3 to schedule android app UI test on ADF. My next step is to make gitlab CI work. My test suite program is Java Appium with TestNG, not integrated with the android program.
My problem now is how to get the PATH of files(apk and test.zip) in gitlab repo which needed in curl command to upload the newly build apk(builded after new code checked in) and my test suite.
Actually, foremost, am I in the right track?
Can I use curl in gitlab like that?
If so, what's the path I could use? (if you could shortly explain the storage structure(or say namespace?) or give me some reference). Actually, is it just take the project home directory as '/'?
for test suite, its actually easier, if I figure out what's the path, I could just put it in the home directory.
for the newly built apk, I don't actually know where is it. We use the pipeline, I think the apk file is somewhere in the server. Below is the YAML snippet:
archive_project:
stage: archive
script:
- ./gradlew assembleRelease
only:
- master
- search
artifacts:
paths:
- main/build/outputs/
tags:
- android
- gradle
If not, how can I do that? This actually suit if I am not in the right track as well.
So after you check in code, the runner look at the .gitlab_ci.yml, and runs following its instruction. And this process, happens in the server machine(either yours or gitlab's), so everything is basically the same as you in your computer(sure you will need the right environment, just appoint the right image or docker).
so yes, we could use 'curl' there. For the directory structure, if you have the privilege to login your server(thru ssh, for instance), you could get it easily. Or we could just explore it like what we do locally(pwd, ls, cd). So what I did is I have a script which have some "pwd, ls, cd ", and call this script from yml, then I look at the info it print to help figure out the directory structure. Then I got what I want(the path), then problem solved.
Although I didn't use dependencies, but you might want to read it to get more about how to pass artifacts between jobs.
If you are looking for this problem, hope it helps you.
Whenever I push a new commit, Travis CI fails my build with this message at the top of every log:
WARNING: We were unable to find a .travis.yml file. This may not be
what you want. Build will be run with default settings.
Using worker:
worker-linux-docker-71483f98.prod.travis-ci.org:travis-linux-6
Could not find .travis.yml, using standard configuration.
However, I definitely have a .travis.yaml file in the root of my repository. Here are its contents:
$ cat .travis.yaml
language:
node_js
node_js:
stable
script:
node_modules/grunt-cli/bin/grunt
Some people seem to have encountered similar issues because they renamed their repositories, but I have never changed the name of this repository. Others say it just fixed itself after a couple hours, but it has been 5 days for me and nothing has changed.
Nothing in the Travis CI documentation seems to indicate that I need to do anything more than sync my repos, active the repo I want CI for, and include a .travis.yaml file in the repo. Am I missing something?
You are using the wrong extension for your YAML file.
It needs to be .travis.yml not .travis.yaml.
Recently I started to use Codeship as CI/CD tool for a small website that I am maintaining. I set up my Codeship project to deploy via sftp as described in their guide here.
The part where it fails is in the production script. I created a deploy folder and a production.sh script which contains the line:
put -rp "${HOME}/clone/build/*" /path/to/remote/dir
However when running the build I get the following error:
sftp> put -rp "${HOME}/clone/build/*" /path/to/remote/dir
stat ${HOME}/clone/build/*: No such file or directory
Echoing $HOME in a test script directly in Codeship gives me my home directory, so the environment variable works. However, at the moment the batch script is run, the environment variable is unrecognized.
How can I fix this? I'd rather not hardcode the path in my deployment script. It also doesn't seem possible that this happens because I suffixed production.sh, whereas in the docs they only have a production script?
With no answer coming from the people from Codeship, I resulted to writing the absolute path to the ${HOME} directory. I've been doing this for a time now with a few different projects and it all seems to work.
replace ${HOME}/clone with ~/clone
this worked for me
I have a jenkins server running, and for a job I need to download a file which is in the jobs/builds/buildname folder.
How to download that file from jenkins job?
If you would use the workspace as suggested by previous post, you can access it within a Pipeline:
sh "wget http://<servername:port>/job/<jobname>/ws/index.txt"
Or inside a script:
wget http://<servername:port>/job/<jobname>/ws/index.txt
Where index.txt is the file you want to download.
I rock a Unix based development machine and a Unix based Jenkins machine up in the cloud. This means I can use the SCP Command to download the remote file over an ssh connection. This is the anatomy of my scp commands:
scp -i <path/to/ssh.pem/file> <user>#<jenkins.remote.url>:<path/to/remote/file> <local/path/where/download/goes>
This works for directories too, for instance I use this to download backups generated by the ThinBackup Plugin
You had already been given the answer for getting the file from the workspace
http://<servername:port>/job/<jobname>/ws/filename.ext
Obviously replace stuff in <..> with values relevant to your setup, and make sure anonymous user has access to read from workspace, else you may have to login.
The only other files you could access are those that are archived from previous job runs.
http://<servername:port>/job/<jobname>/<buildnumber>/artifact/filename.ext
Where <buildnumber> is the build number you see in job build history, or one of the permalinks provided by Eldad (such as lastStableBuild). But this will only have access to archived artifacts.
You cannot arbitrarily access files from Jenkin's filesystem through the web interface... it wouldn't be very secure if it did let you.
The Jenkins job's build folder is meant for logging and plugins reports. You should not need to access it directly.
If you must, you can access it relative to the workspace: $WORKSPACE/../builds/$BUILD_ID/
You can also replace the $BUILD_ID with one of the links Jenkins creates:
lastFailedBuild
lastStableBuild
lastSuccessfulBuild
lastUnstableBuild
lastUnsuccessfulBuild
I hope this helps.
As others have pointed out this path should work, I like to highlight that the "ws" is a directory in Jenkins:
http://<servername:port>/job/<your job>/ws/<your file>
Download the Package lynx (Command line browser)
$ apt-get install lynx
or
$ yum install lynx
then use the command
# lynx http://<servername:port>/job/<jobname>/ws/file
The App Will Ask you to allow cookies and if there are authentication will direct you to login page like the browser.