Travis-ci file for multi-module multi-language project - travis-ci

Is there a way to setup travis-ci to build multiple projects in different languages ?
Lets say my repo is
AwesomeRepoThatDoestOneThingInDifferentLanguages
-cpp
-csharp
-python2
-python3
-java
-go
-ruby
-lua
-objectivec
Now, I wish to apply ci to this.
If I just add .travis.yml file to my python2 project, travis-ci complains that no .travis.yml found, since its looking for one at the root of the repo.
So, is there a way where I can put a .travis.yml at the top, which then further invokes the travis.yml in each of my sub-modules ?

I was able to find a way to actually do this. This is not complete yet since some of the jobs are still failing and I need to fix them individually, but overall the idea works just fine.
I am using travis ci's matrix way. Here is my .travis.yml
Thought this might help if someone else had the same question or scenario.
---
matrix:
include:
-
before_script:
- "cd java"
jdk: oraclejdk8
language: java
script: "mvn clean package"
-
before_script:
- "cd python2"
- "pip install --upgrade setuptools"
- "pip install nose"
- "pip freeze > requirements.txt"
- "pip install -r requirements.txt"
- "cd tests"
cache: pip
language: python
python: 2.7
script: nosetests
-
before_script:
- "cd python3"
- "pip install --upgrade setuptools"
- "pip install nose"
- "pip freeze > requirements.txt"
- "pip install -r requirements.txt"
- "cd tests"
language: python
python: 3.6
script: nosetests
-
before_script:
- "cd objectivec"
language: objective-c
os: osx
osx_image: xcode10
script:
- "xcodebuild clean test -project DiffMatchPatch.xcodeproj -scheme DiffMatchPatch -destination 'platform=OS X,arch=x86_64'"
-
before_script:
- "sudo add-apt-repository --yes ppa:ubuntu-sdk-team/ppa"
- "sudo apt-get update -qq"
- "sudo apt-get install -qq qt5-qmake qtbase5-dev qtdeclarative5-dev libqt5webkit5-dev libsqlite3-dev"
language: cpp
script:
- "qmake -qt=qt5 -v"
- "qmake -qt=qt5"
- make
-
before_script:
- "cd csharp"
dotnet: "1.1.5"
env: DOTNETCORE=1
language: csharp
mono: latestscript
script:
- "dotnet restore"
solution: solution-name.sln
-
before_script:
- "cd dart"
dart: "1.15.0"
dart_task:
-
test: "--exclude-tags no-xvfb"
-
test: "--tags no-xvfb"
language: dart
with_content_shell: true
-
after_success:
- coveralls -b .. -r .. -i ./lua --dump c.report.json
- luacov-coveralls -j c.report.json -v
before_install:
- "cd lua"
- "pip install --upgrade pip"
- "pip install --upgrade setuptools"
- "pip freeze > requirements.txt"
- "pip install -r requirements.txt"
- "sudo wget https://luarocks.org/releases/luarocks-2.4.4.tar.gz"
- "sudo tar zxpf luarocks-2.4.4.tar.gz"
- "cd luarocks-2.4.4"
- "./configure --prefix=/usr/local/openresty/luajit \
--with-lua=/usr/local/openresty/luajit \
--lua-suffix=jit-2.1.0-beta2 \
--with-lua-include=/usr/local/openresty/luajit/include/luajit-2.1; sudo make bootstrap"
- "sudo luarocks install luasocket"
- "pip install cpp-coveralls"
- "luarocks install Lua-cURL --server=https://luarocks.org/dev"
- "luarocks install luacov-coveralls --server=https://luarocks.org/dev"
- "luarocks install lunitx"
- "luarocks install luabitop"
language: python
script:
- "lunit.sh tests/diff_match_patch_test.lua"
- "lunit.sh tests/speedtest.lua"
notifications:
email:
-
sudo: required

Related

GitLab CI: UI Testing with Selenium

I have built some UI tests for my application using Selenium and PyTest. I want to automate these tests by integrating the in the GitLab CI. As the latter is docker-based and I am building by UI using a docker-compose script, I need docker in docker (docker:dind) capabilities besides Firefox, Selenium and Python.
My current gitlab-ci.yml looks like this:
ui_test:
stage: Build_and_Test
image:
name: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/docker/compose
entrypoint: [ '/bin/sh', '-c' ]
services:
- name: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/docker:dind
alias: docker
- selenium/standalone-firefox:latest
variables:
GET_SOURCES_ATTEMPTS: 1
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ''
PROXY_REGISTRY_PREFIX: '${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/'
before_script:
- apk add --update --no-cache bash git openssh gcc musl-dev g++ make build-base libffi-dev python3-dev libressl-dev python3 wget tar npm
- apk --no-cache add --repository http://dl-cdn.alpinelinux.org/alpine/edge/community firefox
- echo which firefox
- python3 -m ensurepip
- wget https://github.com/mozilla/geckodriver/releases/download/v0.31.0/geckodriver-v0.31.0-linux64.tar.gz
- tar -xvf geckodriver-v0.31.0-linux64.tar.gz
- chmod +x geckodriver
- export PATH=$PATH:/builds/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME
script:
- docker login -u "$CI_DEPENDENCY_PROXY_USER" -p "$CI_DEPENDENCY_PROXY_PASSWORD" "$CI_DEPENDENCY_PROXY_SERVER"
- alias python=python3
- echo 'alias python=python3' >> ~/.bashrc
- source ~/.bashrc
- python3 -m pip install --upgrade pip setuptools wheel
- python3 -m pip install --no-cache-dir --upgrade cryptography requests webdriver-manager
- python3 -m pip install --no-cache-dir --upgrade pytest
- python3 -m pip install --no-cache-dir --upgrade pytest-asyncio
- python3 -m pip install --no-cache-dir selenium==4.1.5
- mkdir -p ./test_output
- ./docker/build_hdssl_mock.sh
- ./docker/start_hdssl_mock.sh & cd ui & npm run serve & python3 -m pytest hdssl_control/tests/acceptance_tests
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH || $CI_PIPELINE_SOURCE == "merge_request_event" || $CI_COMMIT_BRANCH == "develop"
The problem here is that Firefox requires all kinds of dependencies that I do not know how to install in Alpine:
ERROR: unsatisfiable constraints:
so:libcjson.so.1 (missing):
required by: librist-0.2.6-r1[so:libcjson.so.1]
so:libmbedcrypto.so.7 (missing):
required by: librist-0.2.6-r1[so:libmbedcrypto.so.7]
so:libffi.so.8 (missing):
required by: firefox-101.0.1-r0[so:libffi.so.8]
so:libicui18n.so.71 (missing):
required by: firefox-101.0.1-r0[so:libicui18n.so.71]
so:libicuuc.so.71 (missing):
required by: firefox-101.0.1-r0[so:libicuuc.so.71]
Any help is greatly appreciated.

In Gitlab-ci is there a way to run before scripts only after rule is match

in gitlab-ci I have
before_script:
- apt update && apt upgrade -y
- apt install -y
and in my job on stages I added a rule
merge_request:
stage: test
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != "master"
- if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == "master"
what's happening is that the job is triggered in his stage as the before script is running, then it gets to the rule and sees that there's nothing to do.
That slows down the pipeline.
Is there a way to have the rule before the "before_script"
here's the pipeline
Thank you
Found the answer here
.before_script_template:
before_script:
- apt-get update
- apt-get -y install apt-transport-https ca-certificates curl gnupg2 software-properties-common
- curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
- add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
- apt-get update
- apt-get -y install docker-ce
- curl -L "https://github.com/docker/compose/releases/download/1.23.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
- chmod +x /usr/local/bin/docker-compose
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
test_integration:
extends: .before_script_template
stage: test
script:
- docker-compose -f CI/backend-service/docker-compose.yml up -d
- npm install
- npm run-script integration-test
build:
extends: .before_script_template
stage: build
script:
- npm install
- export VERSION=`git describe --tags --always`
- docker build -t $CI_REGISTRY_IMAGE:$VERSION .
- docker push $CI_REGISTRY_IMAGE
etc
I would rather put the before script commands in a separate stage, and keep the rules in the first stage.
That way, the first stage triggers only if the rules match.
It can run the commands of the formerly "before_script".
And the next stages can go on with your script.

GitLab: How to reuse the applications installed in before_script stage into the next stages (Eg, test) without reinstalling

I have a testing project in GitLab which uses Python, Robot Framework, Chrome and Selenium. I am using the following gitlab-ci.yml:
image: python:3.9
variables:
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
cache:
paths:
- .cache/pip
- venv/
stages:
- test_API1
- test_API2
before_script:
- python -V # Print out python version for debugging
- wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
- echo 'deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main' | tee /etc/apt/sources.list.d/google-chrome.list
- apt update -y
- apt install -y google-chrome-stable
- pip install virtualenv
- virtualenv venv
- source venv/bin/activate
- pip install -r requirements.txt
API1:
stage: test_API1
when: manual
script:
- python3 -m robot.run -d Results1 Test/test_API1.robot
API2:
stage: test_API2
script:
- python3 -m robot.run -d Results2 Test/test_API2.robot
I specifically want the two Test Stages to be separate and in doing so, i have to go through the before_script stage two times, which slows down the overall pipeline.
Is there a better and faster way to have Python, Robot Framework, Chrome and Selenium installed in one stage and then reused in test stages? Can someone suggest any docker images which are already present to work for this scenario?

Why does Travis fail to find rospy when running unit tests?

We have unit tests that require rospy (One of the test uses geometry_msgs/Twist).
When we run the unit tests locally, everything is fine.
pytest
============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/cedric/Programming/robotics-prototype, inifile: pytest.ini
collected 10 items
tests/unit/branch_name_test.py ...... [ 60%]
tests/unit/drive_controls_test.py .. [ 80%]
tests/unit/import_test.py .. [100%]
========================== 10 passed in 2.73 seconds ===========================
When we run it on Travis, it fails. This is because it fails to find rospy. Our Travis setup installs ROS, runs catkin_make successfully, but fails at the unit tests.
The repo is public so if you're interesting in seeing the complete travis logs click here.
os: linux
dist: bionic
language: python
cache:
- apt
env:
global:
- REPO_ROOT=$(pwd)
- ARDUINO_IDE_VERSION="1.8.12"
- ARDUINO_PATH="$HOME/arduino-$ARDUINO_IDE_VERSION" # Required for CMake script
- TEENSYDUINO_VERSION="152"
- ROS_PYTHON_VERSION="3.6"
- ROS_VERSION="melodic"
- OS_VERSION="bionic"
install:
# Taken from this tutorial https://github.com/newdigate/teensy-blink
- mkdir -p robot/rover/build
- mkdir -p ~/Arduino/libraries
- wget --quiet https://downloads.arduino.cc/arduino-$ARDUINO_IDE_VERSION-linux64.tar.xz
- tar xf arduino-$ARDUINO_IDE_VERSION-linux64.tar.xz -C /home/$USER/
- curl -fsSL https://www.pjrc.com/teensy/td_$TEENSYDUINO_VERSION/TeensyduinoInstall.linux64 -o TeensyduinoInstall.linux64
- chmod +x TeensyduinoInstall.linux64
- /sbin/start-stop-daemon --start --quiet --pidfile /tmp/custom_xvfb_1.pid --make-pidfile --background --exec /usr/bin/Xvfb -- :1 -ac -screen 0 1280x1024x16
- sleep 3
- export DISPLAY=:1.0
- ./TeensyduinoInstall.linux64 --dir=$ARDUINO_PATH
# Install pip dependencies and setup robot python package
- pip install -r requirements.txt -r requirements-dev.txt
- pip install -e .
# Install ros-base and necessary packages
- sudo sh -c "echo \"deb http://packages.ros.org/ros/ubuntu $OS_VERSION main\" > /etc/apt/sources.list.d/ros-latest.list"
- sudo apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654
- sudo apt update -q
- sudo apt install -y ros-$ROS_VERSION-ros-base ros-$ROS_VERSION-cv-camera ros-$ROS_VERSION-web-video-server
- sudo apt install -y python-rosdep python-wstool
# Install ros package dependencies
- cd $REPO_ROOT/robot/rospackages
- sudo rosdep init
- rosdep update
- rosdep install -y -r --from-paths src --ignore-src --rosdistro $ROS_VERSION
script:
# Set up 'robot' python module and test python installation
- cd $REPO_ROOT
- python setup.py develop
- pytest --continue-on-collection-errors
# Set up and initialize catkin workspace for ros packages
- cd $REPO_ROOT/robot/rospackages
- source /opt/ros/$ROS_VERSION/setup.bash
- catkin_make
# Allow to compile arduino code in external IDEs or something
- cd $REPO_ROOT/robot/rover/build
- cmake ..
- make
notifications:
email:
recipients:
- (hidden for the purpose of the question)
on_success: change
on_failure: change
python:
- "3.6"
git:
submodules: true
It seems that you are calling the test before sourcing your ROS environment. You need change that around:
script:
# Set up and initialize catkin workspace for ros packages
- cd $REPO_ROOT/robot/rospackages
- source /opt/ros/$ROS_VERSION/setup.bash
- catkin_make
# Set up 'robot' python module and test python installation
- cd $REPO_ROOT
- python setup.py develop
- pytest --continue-on-collection-errors

.gitlab-ci.yml error: "apk: command not found"

I have the following .gitlab-ci.yml file:
image: docker
services:
- docker:dind
stages:
- test
- build
- deploy
test:
stage: test
before_script:
- apk add --update -y python-pip
- pip install docker-compose
script:
- echo "Testing the app"
- docker-compose run app sh -c "python manage.py test && flake8"
build:
stage: build
only:
- develop
- production
- feature/deploy-debug-gitlab
before_script:
- apk add --update -y python-pip
- pip install docker-compose
script:
- echo "Building the app"
- docker-compose build
deploy:
stage: deploy
only:
- master
- develop
- feature/deploy
- feature/deploy-debug-gitlab
before_script:
- apk add --update -y python-pip
- pip install docker-compose
script:
- echo "Deploying the app"
- docker-compose up -d
environment: production
when: manual
When the Gitlab runner executes it, I get the following error:
$ apk add --update -y python-pip
bash: line 82: apk: command not found
ERROR: Job failed: exit status 1
How am I supposed to install apk? Or what image other than docker should I be using to run this gitlab-ci.yml file?
Well, it turns out I had two different runners: one marked as "shell executor" (Ubuntu) and the other marked as "docker executor“.
This error was being thrown out only when the Ubuntu runner was dispatching the job, since Ubuntu doesn´t come with apk.
I disabled the Ubuntu runner and solved the problem.
The alternative is to set your installation on step above test, as in this issue
image: docker:latest
services:
- docker:dind
before_script:
- apk add --update python-pip

Resources