Why does Travis fail to find rospy when running unit tests? - travis-ci

We have unit tests that require rospy (One of the test uses geometry_msgs/Twist).
When we run the unit tests locally, everything is fine.
pytest
============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/cedric/Programming/robotics-prototype, inifile: pytest.ini
collected 10 items
tests/unit/branch_name_test.py ...... [ 60%]
tests/unit/drive_controls_test.py .. [ 80%]
tests/unit/import_test.py .. [100%]
========================== 10 passed in 2.73 seconds ===========================
When we run it on Travis, it fails. This is because it fails to find rospy. Our Travis setup installs ROS, runs catkin_make successfully, but fails at the unit tests.
The repo is public so if you're interesting in seeing the complete travis logs click here.
os: linux
dist: bionic
language: python
cache:
- apt
env:
global:
- REPO_ROOT=$(pwd)
- ARDUINO_IDE_VERSION="1.8.12"
- ARDUINO_PATH="$HOME/arduino-$ARDUINO_IDE_VERSION" # Required for CMake script
- TEENSYDUINO_VERSION="152"
- ROS_PYTHON_VERSION="3.6"
- ROS_VERSION="melodic"
- OS_VERSION="bionic"
install:
# Taken from this tutorial https://github.com/newdigate/teensy-blink
- mkdir -p robot/rover/build
- mkdir -p ~/Arduino/libraries
- wget --quiet https://downloads.arduino.cc/arduino-$ARDUINO_IDE_VERSION-linux64.tar.xz
- tar xf arduino-$ARDUINO_IDE_VERSION-linux64.tar.xz -C /home/$USER/
- curl -fsSL https://www.pjrc.com/teensy/td_$TEENSYDUINO_VERSION/TeensyduinoInstall.linux64 -o TeensyduinoInstall.linux64
- chmod +x TeensyduinoInstall.linux64
- /sbin/start-stop-daemon --start --quiet --pidfile /tmp/custom_xvfb_1.pid --make-pidfile --background --exec /usr/bin/Xvfb -- :1 -ac -screen 0 1280x1024x16
- sleep 3
- export DISPLAY=:1.0
- ./TeensyduinoInstall.linux64 --dir=$ARDUINO_PATH
# Install pip dependencies and setup robot python package
- pip install -r requirements.txt -r requirements-dev.txt
- pip install -e .
# Install ros-base and necessary packages
- sudo sh -c "echo \"deb http://packages.ros.org/ros/ubuntu $OS_VERSION main\" > /etc/apt/sources.list.d/ros-latest.list"
- sudo apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654
- sudo apt update -q
- sudo apt install -y ros-$ROS_VERSION-ros-base ros-$ROS_VERSION-cv-camera ros-$ROS_VERSION-web-video-server
- sudo apt install -y python-rosdep python-wstool
# Install ros package dependencies
- cd $REPO_ROOT/robot/rospackages
- sudo rosdep init
- rosdep update
- rosdep install -y -r --from-paths src --ignore-src --rosdistro $ROS_VERSION
script:
# Set up 'robot' python module and test python installation
- cd $REPO_ROOT
- python setup.py develop
- pytest --continue-on-collection-errors
# Set up and initialize catkin workspace for ros packages
- cd $REPO_ROOT/robot/rospackages
- source /opt/ros/$ROS_VERSION/setup.bash
- catkin_make
# Allow to compile arduino code in external IDEs or something
- cd $REPO_ROOT/robot/rover/build
- cmake ..
- make
notifications:
email:
recipients:
- (hidden for the purpose of the question)
on_success: change
on_failure: change
python:
- "3.6"
git:
submodules: true

It seems that you are calling the test before sourcing your ROS environment. You need change that around:
script:
# Set up and initialize catkin workspace for ros packages
- cd $REPO_ROOT/robot/rospackages
- source /opt/ros/$ROS_VERSION/setup.bash
- catkin_make
# Set up 'robot' python module and test python installation
- cd $REPO_ROOT
- python setup.py develop
- pytest --continue-on-collection-errors

Related

GitLab CI: UI Testing with Selenium

I have built some UI tests for my application using Selenium and PyTest. I want to automate these tests by integrating the in the GitLab CI. As the latter is docker-based and I am building by UI using a docker-compose script, I need docker in docker (docker:dind) capabilities besides Firefox, Selenium and Python.
My current gitlab-ci.yml looks like this:
ui_test:
stage: Build_and_Test
image:
name: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/docker/compose
entrypoint: [ '/bin/sh', '-c' ]
services:
- name: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/docker:dind
alias: docker
- selenium/standalone-firefox:latest
variables:
GET_SOURCES_ATTEMPTS: 1
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ''
PROXY_REGISTRY_PREFIX: '${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/'
before_script:
- apk add --update --no-cache bash git openssh gcc musl-dev g++ make build-base libffi-dev python3-dev libressl-dev python3 wget tar npm
- apk --no-cache add --repository http://dl-cdn.alpinelinux.org/alpine/edge/community firefox
- echo which firefox
- python3 -m ensurepip
- wget https://github.com/mozilla/geckodriver/releases/download/v0.31.0/geckodriver-v0.31.0-linux64.tar.gz
- tar -xvf geckodriver-v0.31.0-linux64.tar.gz
- chmod +x geckodriver
- export PATH=$PATH:/builds/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME
script:
- docker login -u "$CI_DEPENDENCY_PROXY_USER" -p "$CI_DEPENDENCY_PROXY_PASSWORD" "$CI_DEPENDENCY_PROXY_SERVER"
- alias python=python3
- echo 'alias python=python3' >> ~/.bashrc
- source ~/.bashrc
- python3 -m pip install --upgrade pip setuptools wheel
- python3 -m pip install --no-cache-dir --upgrade cryptography requests webdriver-manager
- python3 -m pip install --no-cache-dir --upgrade pytest
- python3 -m pip install --no-cache-dir --upgrade pytest-asyncio
- python3 -m pip install --no-cache-dir selenium==4.1.5
- mkdir -p ./test_output
- ./docker/build_hdssl_mock.sh
- ./docker/start_hdssl_mock.sh & cd ui & npm run serve & python3 -m pytest hdssl_control/tests/acceptance_tests
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH || $CI_PIPELINE_SOURCE == "merge_request_event" || $CI_COMMIT_BRANCH == "develop"
The problem here is that Firefox requires all kinds of dependencies that I do not know how to install in Alpine:
ERROR: unsatisfiable constraints:
so:libcjson.so.1 (missing):
required by: librist-0.2.6-r1[so:libcjson.so.1]
so:libmbedcrypto.so.7 (missing):
required by: librist-0.2.6-r1[so:libmbedcrypto.so.7]
so:libffi.so.8 (missing):
required by: firefox-101.0.1-r0[so:libffi.so.8]
so:libicui18n.so.71 (missing):
required by: firefox-101.0.1-r0[so:libicui18n.so.71]
so:libicuuc.so.71 (missing):
required by: firefox-101.0.1-r0[so:libicuuc.so.71]
Any help is greatly appreciated.

GitLab: How to reuse the applications installed in before_script stage into the next stages (Eg, test) without reinstalling

I have a testing project in GitLab which uses Python, Robot Framework, Chrome and Selenium. I am using the following gitlab-ci.yml:
image: python:3.9
variables:
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
cache:
paths:
- .cache/pip
- venv/
stages:
- test_API1
- test_API2
before_script:
- python -V # Print out python version for debugging
- wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
- echo 'deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main' | tee /etc/apt/sources.list.d/google-chrome.list
- apt update -y
- apt install -y google-chrome-stable
- pip install virtualenv
- virtualenv venv
- source venv/bin/activate
- pip install -r requirements.txt
API1:
stage: test_API1
when: manual
script:
- python3 -m robot.run -d Results1 Test/test_API1.robot
API2:
stage: test_API2
script:
- python3 -m robot.run -d Results2 Test/test_API2.robot
I specifically want the two Test Stages to be separate and in doing so, i have to go through the before_script stage two times, which slows down the overall pipeline.
Is there a better and faster way to have Python, Robot Framework, Chrome and Selenium installed in one stage and then reused in test stages? Can someone suggest any docker images which are already present to work for this scenario?

How to add bc to bitbucket CI?

I am adding pylint to Bitbucket CI. I decided to move with the custom scripts.
bitbucket-pipeline.yml is the following:
image: python:3.6.2
pipelines:
default:
- step:
caches:
- pip
script:
- set -e
- pip install -r requirements.txt
- pip install --upgrade urllib3
- nosetests project1/test
- nosetests project2/test
- pylint --rcfile=.pylintrc --output-format=text project1/report | tee pylint.txt
- score=$(sed -n 's/^Your code has been rated at \([-0-9.]*\)\/.*/\1/p' pylint.txt)
- apt-get install bc
- sh pylint_score.sh
The script for checking pylint score (pylint_score.sh) is below:
#!/usr/sh
score=$(sed -n 's/^Your code has been rated at \([-0-9.]*\)\/.*/\1/p' pylint.txt)
echo "Pylint score was $score"
threshhold=$(echo "$score>8.0" | bc)
echo "$threshhold"
if [ $threshhold -eq 0 ]
then
exit 1
fi
exit 0
The error I am getting:
E: Unable to locate package bc
How can I install bc into Bitbucket CI or there is any workaround to get it work without bc at all?
Initially, I looked on different solutions but it seems that there are neither out-of-the-box solution nor "best practice" guy that everyone implements. Therefore, I moved with my custom implementation.
To resolve my issue, I found Bitbucket ticket.
Inbitbucket-pipeline.yml:
apt-get install bc line should be modified to
apt-get update && apt-get install -y bc
or in some cases to
apt-get update && apt-get install -y --no-install-recommends bc

Travis-ci file for multi-module multi-language project

Is there a way to setup travis-ci to build multiple projects in different languages ?
Lets say my repo is
AwesomeRepoThatDoestOneThingInDifferentLanguages
-cpp
-csharp
-python2
-python3
-java
-go
-ruby
-lua
-objectivec
Now, I wish to apply ci to this.
If I just add .travis.yml file to my python2 project, travis-ci complains that no .travis.yml found, since its looking for one at the root of the repo.
So, is there a way where I can put a .travis.yml at the top, which then further invokes the travis.yml in each of my sub-modules ?
I was able to find a way to actually do this. This is not complete yet since some of the jobs are still failing and I need to fix them individually, but overall the idea works just fine.
I am using travis ci's matrix way. Here is my .travis.yml
Thought this might help if someone else had the same question or scenario.
---
matrix:
include:
-
before_script:
- "cd java"
jdk: oraclejdk8
language: java
script: "mvn clean package"
-
before_script:
- "cd python2"
- "pip install --upgrade setuptools"
- "pip install nose"
- "pip freeze > requirements.txt"
- "pip install -r requirements.txt"
- "cd tests"
cache: pip
language: python
python: 2.7
script: nosetests
-
before_script:
- "cd python3"
- "pip install --upgrade setuptools"
- "pip install nose"
- "pip freeze > requirements.txt"
- "pip install -r requirements.txt"
- "cd tests"
language: python
python: 3.6
script: nosetests
-
before_script:
- "cd objectivec"
language: objective-c
os: osx
osx_image: xcode10
script:
- "xcodebuild clean test -project DiffMatchPatch.xcodeproj -scheme DiffMatchPatch -destination 'platform=OS X,arch=x86_64'"
-
before_script:
- "sudo add-apt-repository --yes ppa:ubuntu-sdk-team/ppa"
- "sudo apt-get update -qq"
- "sudo apt-get install -qq qt5-qmake qtbase5-dev qtdeclarative5-dev libqt5webkit5-dev libsqlite3-dev"
language: cpp
script:
- "qmake -qt=qt5 -v"
- "qmake -qt=qt5"
- make
-
before_script:
- "cd csharp"
dotnet: "1.1.5"
env: DOTNETCORE=1
language: csharp
mono: latestscript
script:
- "dotnet restore"
solution: solution-name.sln
-
before_script:
- "cd dart"
dart: "1.15.0"
dart_task:
-
test: "--exclude-tags no-xvfb"
-
test: "--tags no-xvfb"
language: dart
with_content_shell: true
-
after_success:
- coveralls -b .. -r .. -i ./lua --dump c.report.json
- luacov-coveralls -j c.report.json -v
before_install:
- "cd lua"
- "pip install --upgrade pip"
- "pip install --upgrade setuptools"
- "pip freeze > requirements.txt"
- "pip install -r requirements.txt"
- "sudo wget https://luarocks.org/releases/luarocks-2.4.4.tar.gz"
- "sudo tar zxpf luarocks-2.4.4.tar.gz"
- "cd luarocks-2.4.4"
- "./configure --prefix=/usr/local/openresty/luajit \
--with-lua=/usr/local/openresty/luajit \
--lua-suffix=jit-2.1.0-beta2 \
--with-lua-include=/usr/local/openresty/luajit/include/luajit-2.1; sudo make bootstrap"
- "sudo luarocks install luasocket"
- "pip install cpp-coveralls"
- "luarocks install Lua-cURL --server=https://luarocks.org/dev"
- "luarocks install luacov-coveralls --server=https://luarocks.org/dev"
- "luarocks install lunitx"
- "luarocks install luabitop"
language: python
script:
- "lunit.sh tests/diff_match_patch_test.lua"
- "lunit.sh tests/speedtest.lua"
notifications:
email:
-
sudo: required

Run nvm with docker exec

I want to run nvm with docker exec
something like
docker run -d <image>
docker exec <container> nvm use v6.13.0 && npm install
but I have an error
OCI runtime exec failed: exec failed: container_linux.go:296: starting container process caused "exec: \"nvm\": executable file not found in $PATH": unknown
I know that I can do something like that which work
docker exec <container> /bin/bash -c 'source "$NVM_DIR"/nvm.sh && nvm use v6.13.0'
But I don't want. Why ? because the point is to create a docker container usable with all my project with different version of python and node and run the nvm use <version> && npm install directly from gitlab-ci using the .nvmrc file into my project
my gitlab-cy.yml run a makefile which basically run the nvm use and npm install
image: cracky5457/nvm-pyenv-yarn
stages:
- install
- test
variables:
GITLAB_CACHING: "true"
cache:
paths:
- pip-cache/
key: "python_2.7"
installing:
stage: install
script:
- make install
artifacts:
paths:
- venv/
- node_modules/
expire_in: 1 hour
tags:
- docker-runner
and I don't want to push /bin/bash -c into my makefile because the project will become docker dependent locally
This is my docker image with the instructions to run it ( you have to create a file base_dependencies.txt, node-versions.txt, python-versions.txt ) or you can just docker pull cracky5457/nvm-pyenv-yarn
https://hub.docker.com/r/cracky5457/nvm-pyenv-yarn/
FROM phusion/baseimage:0.10.0
# Make sure bash is the standard shell
RUN rm /bin/sh && ln -sf /bin/bash /bin/sh
ENV ENV ~/.profile
ENV PYENV_ROOT /root/.pyenv
ENV PATH $PYENV_ROOT/shims:$PYENV_ROOT/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$PATH
# Add yarn registry
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - && \
echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
# Install base system libraries.
ENV DEBIAN_FRONTEND=noninteractive
COPY base_dependencies.txt /base_dependencies.txt
RUN apt-get update && \
apt-get install -y $(cat /base_dependencies.txt)
# Install pyenv and default python version.
ENV PYTHONDONTWRITEBYTECODE true
RUN git clone https://github.com/yyuu/pyenv.git /root/.pyenv && \
cd /root/.pyenv && \
git checkout `git describe --abbrev=0 --tags` && \
eval "$(pyenv init -)"
# Install nvm and default node version.
ENV NVM_DIR /usr/local/nvm
RUN curl https://raw.githubusercontent.com/creationix/nvm/v0.33.8/install.sh | bash && \
echo 'source $NVM_DIR/nvm.sh' >> /etc/profile
# Clean up APT when done.
RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
# Install python and node versions
COPY python-versions.txt /python-versions.txt
RUN for version in $(cat python-versions.txt); do pyenv install $version; pyenv global $version; pip install virtualenv; done
COPY node-versions.txt /node-versions.txt
RUN for version in $(cat node-versions.txt); do source $NVM_DIR/nvm.sh; nvm install $version; done
# Use baseimage-docker's init system.
CMD ["/sbin/my_init"]
I didn't found a proper way.
You can create a bash file into /usr/bin/nvm with chmod +x /usr/bin/nvm
#!/bin/bash
export NVM_DIR="/usr/local/nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" # This loads nvm bash_completion
nvm "$#"
And then
docker exec <container> nvm use
But it's tricky and I can't add an other instruction in my exec, for exemple I can't docker exec <container> nvm use && npm install at the same time.
But I finally fixed my issue directly in gitlab-ci.yaml using
$(NVM_DIR)/nvm.sh && nvm use && npm install

Resources