I am adding pylint to Bitbucket CI. I decided to move with the custom scripts.
bitbucket-pipeline.yml is the following:
image: python:3.6.2
pipelines:
default:
- step:
caches:
- pip
script:
- set -e
- pip install -r requirements.txt
- pip install --upgrade urllib3
- nosetests project1/test
- nosetests project2/test
- pylint --rcfile=.pylintrc --output-format=text project1/report | tee pylint.txt
- score=$(sed -n 's/^Your code has been rated at \([-0-9.]*\)\/.*/\1/p' pylint.txt)
- apt-get install bc
- sh pylint_score.sh
The script for checking pylint score (pylint_score.sh) is below:
#!/usr/sh
score=$(sed -n 's/^Your code has been rated at \([-0-9.]*\)\/.*/\1/p' pylint.txt)
echo "Pylint score was $score"
threshhold=$(echo "$score>8.0" | bc)
echo "$threshhold"
if [ $threshhold -eq 0 ]
then
exit 1
fi
exit 0
The error I am getting:
E: Unable to locate package bc
How can I install bc into Bitbucket CI or there is any workaround to get it work without bc at all?
Initially, I looked on different solutions but it seems that there are neither out-of-the-box solution nor "best practice" guy that everyone implements. Therefore, I moved with my custom implementation.
To resolve my issue, I found Bitbucket ticket.
Inbitbucket-pipeline.yml:
apt-get install bc line should be modified to
apt-get update && apt-get install -y bc
or in some cases to
apt-get update && apt-get install -y --no-install-recommends bc
Related
I have a testing project in GitLab which uses Python, Robot Framework, Chrome and Selenium. I am using the following gitlab-ci.yml:
image: python:3.9
variables:
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
cache:
paths:
- .cache/pip
- venv/
stages:
- test_API1
- test_API2
before_script:
- python -V # Print out python version for debugging
- wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
- echo 'deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main' | tee /etc/apt/sources.list.d/google-chrome.list
- apt update -y
- apt install -y google-chrome-stable
- pip install virtualenv
- virtualenv venv
- source venv/bin/activate
- pip install -r requirements.txt
API1:
stage: test_API1
when: manual
script:
- python3 -m robot.run -d Results1 Test/test_API1.robot
API2:
stage: test_API2
script:
- python3 -m robot.run -d Results2 Test/test_API2.robot
I specifically want the two Test Stages to be separate and in doing so, i have to go through the before_script stage two times, which slows down the overall pipeline.
Is there a better and faster way to have Python, Robot Framework, Chrome and Selenium installed in one stage and then reused in test stages? Can someone suggest any docker images which are already present to work for this scenario?
I'm working on a project where code coverage needs to display in Sonarqube.
We are framing dockerfile.test where at ENTRYPOINT nosetests is provided. setup.cfg file is created putting all required nosetests plugins. Then sonar-project.properties file has been created for sonarqube reference. And a stage named "Test/Sonar" is created in jenkins ,
below has the script of all above mentioned files.
FROM python:3.7-slim
RUN apt-get update && apt-get install -y --no-install-recommends gcc gfortran perl libtest-utf8-perl \
libtest-file-perl libtest-files-perl cpanminus build-essential curl libcurl4-openssl-dev libssl-dev \
gfortran liblzma-dev libblas-dev liblapack-dev libpq-dev procps libldap2-dev libsasl2-dev ldap-utils git
RUN cpanm install File::Copy::Recursive
RUN pip install numpy && pip install scipy pandas scikit-learn
COPY requirements.txt /requirements.txt
RUN pip install --no-cache-dir -r /requirements.txt
COPY test_requirements.txt /test_requirements.txt
RUN pip install --no-cache-dir -r /test_requirements.txt
COPY . /app
WORKDIR /app
#Install Mim
RUN python mim-python-client/setup.py install
RUN mkdir -p /local/tmp/mim-cache
RUN chmod -R 777 /local/tmp/mim-cache
#Install Extract
WORKDIR /app/extract-rules
RUN bash --posix ./dev/build.sh
RUN cp ./bin/python_parser.py /opt/extract/bin/
WORKDIR /app/backend/worker
ENV PYTHONUNBUFFERED=1
ENTRYPOINT nosetests
[nosetests]
verbose=3
with-xunit=1
xunit-file=nosetests.xml
with-coverage=1
cover-xml=1
cover-package=.
exclude-dir=test
sonar.projectKey=IPA_NIP
sonar.projectName=IPA_NIP
sonar.projectVersion=1.0.0
sonar.language=py
sonar.sourceEncoding=UTF-8
sonar.python.coverage.reportPaths=coverage.xml
sonar.python.xunit.reportPaths=nosetests.xml
sonar.python.xunit.skipDetails=false
sonar.sources=backend/orch, backend/worker, webserver/pinap
sonar.typescript.lcov.reportPaths=coverage/IPA_NIP/lcov.info
sonar.python.coveragePlugin=cobertura
stage ('Test/Sonar') {
steps {
// This specific examples expects that you have a Dockerfile.test that you build, then run to generate test results.
// Since different projects can vary however, you should make sure you use the solution that works best for you.
sh "docker build . -f Dockerfile.test -t python-test:${BUILD_ID}"
sh '''docker run -u 50001255:25 -v "$(pwd)":/app python-test:${BUILD_ID}'''
sh '''sed -i 's|filename="|filename="'"$(pwd)"'/|' coverage.xml'''
sonarScan('Sonar')
}
// Make test results visible in Jenkins UI if the install step completed successfully
post {
success {
junit testResults: 'nosetests.xml', allowEmptyResults: true
}
}
}
I couldn't able to find why I'm getting 0% coverage at sonarqube.
dockerfile.test is executing without any error. I'm getting status about tests (Unit test) as OK. But I'm getting a warning
WARN: The report '/apps/dftjenkins/jenkins_node/workspace/v_AutomationAuthenticationScript/coverage.xml' seems to be empty, ignoring. '{}'
org.sonar.plugins.python.EmptyReportException: null
Could anyone help me out with my queries?
I have a dockerfile to upload some python code on Azure. It has been working for a few months, but today it suddenly stopped working.
The relevant commands in the Dockerfile are:
FROM python:3.9.5
:
:
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/ubuntu/20.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN exit
RUN apt-get update && ACCEPT_EULA=Y apt-get install msodbcsql17
The error message is that appeared today is:
Err:1 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 msodbcsql17 amd64 17.7.2.1-1
404 Not Found [IP: 104.214.230.139 443]
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/m/msodbcsql17/msodbcsql17_17.7.2.1-1_amd64.deb 404 Not Found [IP: 104.214.230.139 443]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
The command '/bin/sh -c apt-get update && ACCEPT_EULA=Y apt-get install msodbcsql17' returned a non-zero code: 100
2021/06/16 20:50:56 Container failed during run: build. No retries remaining.
failed to run step ID: build: exit status 100
I believe this might be related to the .deb files being moved - or that some computer at Microsoft is down?
A good workaround would maybe be to download the relevant msodbcsql17 package directly, but I was unable to find this package in the normal repos?
There seems to be some ongoing trouble with microsoft repos for some linux distributions (including ubuntu and debian). Not clear when this will be fixed.
https://github.com/dotnet/core/issues/6381
https://github.com/MicrosoftDocs/sql-docs/issues/6494
The answer might be related to this post:
https://github.com/dotnet/core/issues/6381
It seems that some Ubuntu repositories are broken.
Hopefully it will be fixed soon...
I Will keep an eye on the resolve, but I have the same issue using:
FROM python:3
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get update
RUN ACCEPT_EULA=Y apt-get install -y msodbcsql17
# optional: for bcp and sqlcmd
RUN ACCEPT_EULA=Y apt-get install -y mssql-tools
RUN echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bash_profile
RUN echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc
# optional: for unixODBC development headers
RUN apt-get install -y unixodbc-dev
We have unit tests that require rospy (One of the test uses geometry_msgs/Twist).
When we run the unit tests locally, everything is fine.
pytest
============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/cedric/Programming/robotics-prototype, inifile: pytest.ini
collected 10 items
tests/unit/branch_name_test.py ...... [ 60%]
tests/unit/drive_controls_test.py .. [ 80%]
tests/unit/import_test.py .. [100%]
========================== 10 passed in 2.73 seconds ===========================
When we run it on Travis, it fails. This is because it fails to find rospy. Our Travis setup installs ROS, runs catkin_make successfully, but fails at the unit tests.
The repo is public so if you're interesting in seeing the complete travis logs click here.
os: linux
dist: bionic
language: python
cache:
- apt
env:
global:
- REPO_ROOT=$(pwd)
- ARDUINO_IDE_VERSION="1.8.12"
- ARDUINO_PATH="$HOME/arduino-$ARDUINO_IDE_VERSION" # Required for CMake script
- TEENSYDUINO_VERSION="152"
- ROS_PYTHON_VERSION="3.6"
- ROS_VERSION="melodic"
- OS_VERSION="bionic"
install:
# Taken from this tutorial https://github.com/newdigate/teensy-blink
- mkdir -p robot/rover/build
- mkdir -p ~/Arduino/libraries
- wget --quiet https://downloads.arduino.cc/arduino-$ARDUINO_IDE_VERSION-linux64.tar.xz
- tar xf arduino-$ARDUINO_IDE_VERSION-linux64.tar.xz -C /home/$USER/
- curl -fsSL https://www.pjrc.com/teensy/td_$TEENSYDUINO_VERSION/TeensyduinoInstall.linux64 -o TeensyduinoInstall.linux64
- chmod +x TeensyduinoInstall.linux64
- /sbin/start-stop-daemon --start --quiet --pidfile /tmp/custom_xvfb_1.pid --make-pidfile --background --exec /usr/bin/Xvfb -- :1 -ac -screen 0 1280x1024x16
- sleep 3
- export DISPLAY=:1.0
- ./TeensyduinoInstall.linux64 --dir=$ARDUINO_PATH
# Install pip dependencies and setup robot python package
- pip install -r requirements.txt -r requirements-dev.txt
- pip install -e .
# Install ros-base and necessary packages
- sudo sh -c "echo \"deb http://packages.ros.org/ros/ubuntu $OS_VERSION main\" > /etc/apt/sources.list.d/ros-latest.list"
- sudo apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654
- sudo apt update -q
- sudo apt install -y ros-$ROS_VERSION-ros-base ros-$ROS_VERSION-cv-camera ros-$ROS_VERSION-web-video-server
- sudo apt install -y python-rosdep python-wstool
# Install ros package dependencies
- cd $REPO_ROOT/robot/rospackages
- sudo rosdep init
- rosdep update
- rosdep install -y -r --from-paths src --ignore-src --rosdistro $ROS_VERSION
script:
# Set up 'robot' python module and test python installation
- cd $REPO_ROOT
- python setup.py develop
- pytest --continue-on-collection-errors
# Set up and initialize catkin workspace for ros packages
- cd $REPO_ROOT/robot/rospackages
- source /opt/ros/$ROS_VERSION/setup.bash
- catkin_make
# Allow to compile arduino code in external IDEs or something
- cd $REPO_ROOT/robot/rover/build
- cmake ..
- make
notifications:
email:
recipients:
- (hidden for the purpose of the question)
on_success: change
on_failure: change
python:
- "3.6"
git:
submodules: true
It seems that you are calling the test before sourcing your ROS environment. You need change that around:
script:
# Set up and initialize catkin workspace for ros packages
- cd $REPO_ROOT/robot/rospackages
- source /opt/ros/$ROS_VERSION/setup.bash
- catkin_make
# Set up 'robot' python module and test python installation
- cd $REPO_ROOT
- python setup.py develop
- pytest --continue-on-collection-errors
months ago I made some tests on gitlab using codeceptjs/puppeteer. I was having 0 issues with the tests. this morning I woke up and every single test on every single app is failing before the apps even begin. none of the tests have been updated in 4+ weeks. they were running fine all on their own until now.
Now I get this error every single time before the first test even starts
`example -- user inserts text and html macro :heavy_multiplication_x: "before each" hook: codeceptjs.before for "user inserts text and html macro" in 13ms Error: Failed to launch the browser process!`
I have no idea what is causing this. I did some googling and all previous issues say that adding "args": ["--no-sandbox", "--disable-setuid-sandbox"] to the config will solve this problem but I already had that in my config file. it's been in my config file for months now. not sure what changed between last night and tonight. this is my gitlab-ci.yml file
`image: node:latest
all_tests:
script:
- apt-get update && apt-get install -yq libgconf-2-4
- apt-get update && apt-get install -y wget --no-install-recommends && wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - && sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' && apt-get update && apt-get install -y google-chrome-unstable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst ttf-freefont --no-install-recommends && rm -rf /var/lib/apt/lists/* && apt-get purge --auto-remove -y curl && rm -rf /src/*.deb
- npm install
- npm i codeceptjs-puppeteer
- npm i codeceptjs puppeteer
- ln -sf ./node_modules/.bin/codeceptjs /usr/local/bin/codeceptjs
- npm i --save arrify
- ./node_modules/.bin/codeceptjs run --steps
cache:
paths:
- ./node_modules/.bin/codeceptjs
after_script:
- echo "Cleaning up"
- rm -rf "%CACHE_PATH%/%CI_PIPELINE_ID%"
`
months ago I pulled that from a repo on gitlab that had working puppeteer tests since I couldn't figure out how to build a properly working one. codeceptjs/puppeteer has a config file here are the contents
`exports.config = {
tests: './tests/test_test.js',
output: './output',
helpers: {
Puppeteer: {
url: 'http://localhost',
show: false,
chrome: {
"args": ["--no-sandbox", "--disable-setuid-sandbox"]
}
}
},
include: {
I: './steps_file.js',
login: './pages/login.js',
confluence: './pages/confluence.js',
editor: './pages/editor.js',
},
bootstrap: null,
mocha: {},
name: 'cloud ME user'
};`
so yeah that config was working fine for me for months until last night when all of a sudden it failed. everything I read says I need to specify that chrome is headless but I already tried that. I tried that months ago. anyone have any idea what is causing this?
Your build config is using node:latest as the base image. Apparently, that has been changed yesterday (see the README.md of node docker repository from version 13 to 14.
You could try downgrading the node image back to version 13:
image: node:13
Or even something more specific according to your needs.