Travis has an easy way to test a project against different PHP versions.
Now I want to run tests for plugins. For that I wrote a script that is called in the install phase of .travis.yml which checks out the main project and moves my plugin source into the correct directory. Then the tests are run. So far so good.
Now I would like to provide two of these scripts. One that checks out the main project's current master branch and one that checks out the latest stable version. The plugin should be tested against both checkouts in completely separate test runs just like it is ran against different PHP versions.
Is there a simple way to configure this in .travis.yml?
You need to use env option:
env:
- TEST_NAME=my_test_1
- TEST_NAME=my_test_2
- TEST_NAME=my_test_3
script:
- ./test-run.sh --test-name=${TEST_NAME}
documentation (Set environment variables section)
example
Related
We are thinking to move our ci from jenkins to gitlab. We have several projects that have the same build workflow. Right now we use a shared library where the pipelines are defined and the jenkinsfile inside the project only calls a method defined in the shared library defining the actual pipeline. So changes only have to be made at a single point affecting several projects.
I am wondering if the same is possible with gitlab ci? As far as i have found out it is not possible to define the gitlab-ci.yml outside the repository. Is there another way to define a pipeline and share this config with several projects to simplify maintainance?
GitLab 11.7 introduces new include methods, such as include:file:
https://docs.gitlab.com/ee/ci/yaml/#includefile
include:
- project: 'my-group/my-project'
ref: master
file: '/templates/.gitlab-ci-template.yml'
This will allow you to create a new project on the same GitLab instance which contains a shared .gitlab-ci.yml.
First let me start by saying: Thank you for asking this question! It triggered me to search for a solution (again) after often wondering if this was even possible myself. We also have like 20 - 30 projects that are quite identical and have .gitlab-ci.yml files of about 400 - 500 loc that have to each be changed if one thing changes.
So I found a working solution:
Inspired by the Auto DevOps .gitlab-ci.yml template Gitlab itself created, and where they use one template job to define all functions used and call every before_script to load them, I came up with the following setup.
Multiple project repo's (project-1, project-2) requiring a shared set of CI jobs / functions
Functions script containing all shared functions in separate repo
Files
So using a shared ci jobs scipt:
#!/bin/bash
function list_files {
ls -lah
}
function current_job_info {
echo "Running job $CI_JOB_ID on runner $CI_RUNNER_ID ($CI_RUNNER_DESCRIPTION) for pipeline $CI_PIPELINE_ID"
}
A common and generic .gitlab-ci.yml:
image: ubuntu:latest
before_script:
# Install curl
- apt-get update -qqq && apt-get install -qqqy curl
# Get shared functions script
- curl -s -o functions.sh https://gitlab.com/giix/demo-shared-ci-functions/raw/master/functions.sh
# Set permissions
- chmod +x functions.sh
# Run script and load functions
- . ./functions.sh
job1:
script:
- current_job_info
- list_files
You could copy-paste your file from project-1 to project-2 and it would be using the same shared Gitlab CI functions.
These examples are pretty verbose for example purposes, optimize them any way you like.
Lessons learned
So after applying the construction above on a large scale (40+ projects) I want to share some lessons learned so you don't have to find out the hard way:
Version (tag / release) your shared ci functions script. Changing one thing can now make all pipelines fail.
Using different Docker images could cause an issue in the requirement for bash to load the functions (e.g. I use some Alpine-based images for CLI tool based jobs that have sh by default)
Use project based CI/CD secret variables to personalize build jobs for projects. Like environment URL's etc.
Since gitlab version 12.6, it's possible define a external .gitlab-cy.yml file.
To customize the path:
Go to the project's Settings > CI / CD.
Expand the General pipelines section.
Provide a value in the Custom CI configuration path field.
Click Save changes.
...
If the CI configuration will be hosted on an external site, the URL link must end with .yml:
http://example.com/generate/ci/config.yml
If the CI configuration will be hosted in a different project within
GitLab, the path must be relative to the root directory in the other
project, with the group and project name added to the end:
.gitlab-ci.yml#mygroup/another-project
my/path/.my-custom-file.yml#mygroup/another-project
Use include feature, (available from GitLab 10.6):
https://docs.gitlab.com/ee/ci/yaml/#include
So, i always wanted to post, with what i came up with now:
Right now we use a mixed approach of #stefan-van-gastel's idea of a shared ci library and the relatively new include feature of gitlab 11.7. We are very satisfied with this approach as we can now manage our build pipeline for 40+ repositories in a single repository.
I have created a repository called ci_shared_library containing
a shell script for every single build job containing the execution logic for the step.
a pipeline.yml file containing the whole pipeline config. In the before script we load the ci_shared_library to /tmp/shared to be able to execute the scripts.
stages:
- test
- build
- deploy
- validate
services:
- docker:dind
before_script:
# Clear existing shared library
- rm -rf /tmp/shared
# Get shared library
- git clone https://oauth2:${GITLAB_TOKEN}#${SHARED_LIBRARY} /tmp/shared
- cd /tmp/shared && git checkout master && cd $CI_PROJECT_DIR
# Set permissions
- chmod -R +x /tmp/shared
# open access to registry
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
test:
stage: test
script:
- /tmp/shared/test.sh
build:
stage: build
script:
- /tmp/shared/build.sh
artifacts:
paths:
- $CI_PROJECT_DIR/target/RPMS/x86_64/*.rpm
expire_in: 3h
only:
- develop
- /release/.*/
deploy:
stage: deploy
script:
- /tmp/shared/deploy.sh
artifacts:
paths:
- $CI_PROJECT_DIR/tmp/*
expire_in: 12h
only:
- develop
- /release/.*/
validate:
stage: validate
script:
- /tmp/shared/validate.sh
only:
- develop
- /release\/.*/
Every project that want's to use this pipeline config has to have a .gitlab-ci.yml. In this file the only thing to do is to import the shared pipeline.yml file from the ci_shared_library repo.
# .gitlab-ci.yml
include:
- project: 'ci_shared_library'
ref: master
file: 'pipeline.yml'
With this approach really everything regarding to the pipeline lives in one single repository and is reusable. We have the whole pipeline-template in one file, but i think it would even be possible to split this up to have every single job in a yml-file. This way it would be more flexible and one could create default jobs that can be merged together differently for projects that have similar jobs but not every project needing all jobs...
With GitLab 13.5 (October 2020), the include feature is even more useful:
Validate expanded GitLab CI/CD configuration with the API
Writing and debugging complex pipelines is not a trivial task. You can use the include keyword to help reduce the length of your pipeline configuration files.
However, if you wanted to validate your entire pipeline via the API previously, you had to validate each included configuration file separately which was complicated and time consuming.
Now you have the ability to validate a fully-expanded version of your pipeline configuration through the API, with all the include configuration included.
Debugging large configurations is now easier and more efficient.
See Documentation and Issue.
And:
See GitLab 13.6 (November 2020)
Include multiple CI/CD configuration files as a list
Previously, when adding multiple files to your CI/CD configuration using the include:file syntax, you had to specify the project and ref for each file. In this release, you now have the ability to specify the project, ref, and provide a list of files all at once. This prevents you from having to repeat yourself and makes your pipeline configuration less verbose.
See Documentation) and Issue.
You could look into the concept of Dynamic Child pipeline.
It has evolved with GitLab 13.2 (July 2020):
Dynamically generate Child Pipeline configurations with Jsonnet
We released Dynamic Child Pipelines back in GitLab 12.9, which allow you to generate an entire .gitlab-ci.yml file at runtime.
This is a great solution for monorepos, for example, when you want runtime behavior to be even more dynamic.
We’ve now made it even easier to create CI/CD YAML at runtime by including a project template that demonstrates how to use Jsonnet to generate the YAML.
Jsonnet is a data templating language that provides functions, variables, loops, and conditionals that allow for fully parameterized YAML configuration.
See documentation and issue.
Anyone got a boilerplate the world can use for Travis CI build testing for modules?
I havn't used Travis CI before but I got it hooked up however all my builds are failing, and the logs just say phpunit exited with code 1. I assume I'm definitely missing something and I have a feeling I need to download all silverstripe composer dependecies but have no idea where to start
I'm wanting it to run tests against (mymodule)/tests folder and hoping it's possible
The section on "Connecting to CI" isn't very helpful!
To set up with travis you'll need to use the travis-support module.
It's quite straightforward to do with a boilerplate .travis.yml which you can see on pretty much any SilverStripe module that's using travis. Here's a pretty standard one.
That file will test against PHP 5.3-5.6 as well as against SilverStripe versions 3.1.x-dev (latest 3.1 development version) all the way to 3.x-dev (3.5 development version).
You'll also need to customise the final line to run the correct test suite.
I trying some test with travis-ci/travis-build.
Can i made script which include setup environment, like apt-get and start service for addon part in .travis.yml?
when i tried to compile travis compile it looks totally ignored addons part.
Some parts of the compilation of .travis.yml happens a phase earlier, when it is decided which image on which infrastructure the build should be run. The command travis build from travis-ci/travis-build makes the shell script that runs inside of the container, and there are some parts which it does not take into account because it is handled by another layer.
I have a sbt project with 4 modules: module-a, module-b, module-c, module-d.
Each module can be packaged as a WAR. I want to set up a deployment on Jenkins that would build only one of the 4 modules and deploy it to a container.
In detail, I want to have 4 Jenkins jobs - job-a, job-b, job-c, job-d, each building only the defined module (a to d).
For now, I am using clean update test package as the command for the Jenkins sbt build, but this results in packaging all 4 modules that is not necessary.
I already tried project -module-a clean update test package but with no luck.
You may also like to execute project-scoped clean and test tasks as follows:
sbt module-a/clean module-a/test
The solution is slightly shorter and clearer as to what project the following commands apply to.
You don't need to execute update task since it's implicitly executed by test as described in inspect tree test.
There's a way to make it cleaner with an alias. Use the following in the build.sbt:
addCommandAlias("jenkinsJob4ModuleA", "; module-a/clean; module-a/test")
With the alias, execute jenkinsJob4ModuleA to have the same effect as the above solution.
Quote the argument to project, i.e. project module-a, and don't use a dash before the name of the submodule.
The entire command line for the Jenkins job would than be as follows:
./sbt "project module-a" clean update test
I have about 100 jobs on my hudson CI, possible to mass delete them ?
The easiest way, IMHO, would be to use script. Go to http://your.hudson.url/script/
Delete jobs by running:
for(j in hudson.model.Hudson.theInstance.getProjects()) {
j.delete();
}
And this way gives you an option to easily use condition to filter out jobs to delete.
FOR JENKINS
Current versions (2.x):
for(j in jenkins.model.Jenkins.theInstance.getAllItems()) {
j.delete()
}
Older versions:
for(j in jenkins.model.Jenkins.getInstance().getProjects()) {
j.delete();
}
Just delete the job directories:
cd $HUDSON_HOME/jobs
rm -rf <JOB_NAME>
See: Administering Hudson
You can programmatically use the XML api (or use the JSON flavor if you prefer that):
http://your.hudson.url/api/xml?xpath=//job/name&wrapper=jobs
Returns:
<jobs>
<name>firstJob</name>
<name>secondJob</name>
<!-- etc -->
</jobs>
Now iterate over the job names and do a post request to
http://your.hudson.url/job/your.job.name/doDelete
(You can do this with any programming language you like that supports XML and HTTP)
I had similar manageability problems with a Hudson instance that was running 500+ build jobs - it was impractical to manually maintain that many jobs using the gui. However, you can provision jobs in Hudson remotely and programatically by using the CLI - which is supplied as a jar file [http://wiki.hudson-ci.org/display/HUDSON/Hudson+CLI].
The command to delete a job would be something like:
**java -jar hudson-cli.jar -s http://host:port/ delete-job jobname**
And the rest of the commands you will need are here:
**java -jar hudson-cli.jar -s http://host:port/** help
I wrapped the cli in python and created an XML file from which to hold the build configuration - then I could use this to manipulate my running instances of Hudson. This also provided the ability to 'reset' the CI instance back to a known configuration - handy if you suspect build failures were caused by manual changes in the UI or if you are using a different CI server for each environment you deploy to (ie dev, test, prod) and need to provision a new one.
This has also got me out of a few binds when badly written plugins have mangled Hudson's own XML and I've needed to rebuild my instances. Hudson is also I/O bound and for really loaded instances it is often faster to boot Hudson from scratch and populate it's configuration this way.