Why am I getting extra jobs with my Travis-CI matrix? - travis-ci

I don't understand the behaviour of Travis-CI with matrix.
My .travis.yml:
language: go
go:
- "1.10.x"
- "1.11.x"
env:
matrix:
- MONGO_SETTINGS=--auth
- MONGO_SETTINGS=
matrix:
include:
- env: MONGO_SETTINGS=--auth
before_script:
- mongorestore -h 127.0.0.1 --port 27017 -d data integration
- mongo data --eval 'db.createUser({user:"travis", pwd:"test", roles:["readWrite"]});'
- mongod --dbpath=data/db --shutdown
- sleep 10
- mongod --dbpath=data/db $MONGO_SETTINGS &
- sleep 3
- mongo data --username travis --password test --eval "db.getCollection('data').find({})"
script:
- go test ./... -tags=authentication
- env: MONGO_SETTINGS=
before_script:
- mongorestore -h 127.0.0.1 --port 27017 -d data integration
- mongo data --eval "db.getCollection('data').find({})"
script:
- go test ./...
install:
- wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.4.18.tgz
- tar xfz mongodb-linux-x86_64-3.4.18.tgz
- export PATH=`pwd`/mongodb-linux-x86_64-3.4.18/bin:$PATH
- mkdir -p data/db
- mongod --dbpath=data/db &
- sleep 3
The matrix returns 6 jobs
1.1 Go: 1.10.x MONGO_SETTINGS=--auth
1.2 Go: 1.10.x MONGO_SETTINGS=
1.3 Go: 1.11.x MONGO_SETTINGS=--auth
1.4 Go: 1.11.x MONGO_SETTINGS=
1.5 Go: 1.10.x MONGO_SETTINGS=--auth
1.6 Go: 1.10.x MONGO_SETTINGS=
Why there are 1.5 and 1.6 jobs ?
To my mind, 1.5 and 1.6 are equal to 1.1 and 1.2.
The expected matrix is :
1.1 Go: 1.10.x MONGO_SETTINGS=--auth
1.2 Go: 1.10.x MONGO_SETTINGS=
1.3 Go: 1.11.x MONGO_SETTINGS=--auth
1.4 Go: 1.11.x MONGO_SETTINGS=
Edit: Thanks to #banzaiman. My mistake was to use matrix.include which added two new jobs.
language: go
go:
- "1.10.x"
- "1.11.x"
env:
matrix:
- MONGO_SETTINGS=--auth
- MONGO_SETTINGS=
install:
- wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.4.18.tgz
- tar xfz mongodb-linux-x86_64-3.4.18.tgz
- export PATH=`pwd`/mongodb-linux-x86_64-3.4.18/bin:$PATH
- mkdir -p data/db
- mongod --dbpath=data/db &
- sleep 3
before_script:
- if [[ ${MONGO_SETTINGS} = "--auth" ]]; then
mongorestore -h 127.0.0.1 --port 27017 -d data integration;
mongo data --eval 'db.createUser({user:"travis", pwd:"test", roles:["readWrite"]})';
mongod --dbpath=data/db --shutdown;
sleep 10;
mongod --dbpath=data/db --fork --logpath mongodb.log "$MONGO_SETTINGS";
sleep 3;
mongo data --username travis --password test --eval "db.getCollection('data').find({})";
else
mongorestore -h 127.0.0.1 --port 27017 -d data integration;
mongo data --eval "db.getCollection('data').find({})";
fi
script:
- if [[ ${MONGO_SETTINGS} = "--auth" ]]; then
go test ./... -tags=authentication;
else
go test ./...;
fi

You have a 2x2 build matrix with:
go:
- "1.10.x"
- "1.11.x"
env:
matrix:
- MONGO_SETTINGS=--auth
- MONGO_SETTINGS=
and additional 2 jobs in matrix.include, for a grand total of 6 jobs.
Notice that your jobs 1.5 and 1.6 behave differently from jobs 1.1 and 1.2 with respect to before_script. I can't tell from your description which jobs you don't want, so I can't really advise you on how to modify this.

Related

Running CircleCI locally bitbucket & token error

Description
Running this:
$ circleci build --branch master --repo-url https://bitbucket.org/cityhive/city-hive-standalone-widget --job e2e
Getting these errors
Cloning into '.'...
Username for 'https://bitbucket.org': fatal: could not read Username for 'https://bitbucket.org': Success
...
# (from Jira orb)
/bin/bash: CIRCLE_TOKEN: Please provide a CircleCI API token for this orb to work!
If I'm trying to pass token I get this:
$ circleci build --token d14c4********* --branch master --repo-url https://bitbucket.org/cityhive/city-hive-standalone-widget --job e2e
Fetching latest build environment...
Docker image digest: sha256:7a080ca333******
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Error: unknown flag: --token
ERRO[0000] Exit with status 1 due to generic error error="unknown flag: --token"
Location
https://circleci.com/docs/2.0/local-cli/#run-a-job-in-a-container-on-your-machine
Steps to Reproduce
Run job that checks out the code and have the JIRA orb locally locally
Additional Context
running macOS Monterey 12.3.1 (21E258), Silicon apple chip
Docker:
$ docker version
Client:
Cloud integration: v1.0.20
Version: 20.10.10
API version: 1.41
Go version: go1.16.9
Git commit: b485636
Built: Mon Oct 25 07:43:15 2021
OS/Arch: darwin/arm64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.10
API version: 1.41 (minimum version 1.12)
Go version: go1.16.9
Git commit: e2f740d
Built: Mon Oct 25 07:41:10 2021
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.4.11
GitCommit: 5b46e404f6b9f661a205e28d59c982d3634148f8
runc:
Version: 1.0.2
GitCommit: v1.0.2-0-g52b36a2
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Config.yml
version: 2.1
orbs:
node: circleci/node#5.0.0
jira: circleci/jira#1.3.1
ruby: circleci/ruby#1.4.0
commands:
ch-checkout:
steps:
# Instead of the default `checkout` phase that doesn't support depth.
- run:
name: "Checkout Repository with --Depth 1"
command: |
SSH_CONFIG_DIR="/home/circleci/.ssh"
mkdir -p "$SSH_CONFIG_DIR"
chmod 0700 "$SSH_CONFIG_DIR"
echo "bitbucket.org ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAubiN81eDcafrgMeLzaFPsw2kNvEcqTKl/VqLat/MaB33pZy0y3rJZtnqwR2qOOvbwKZYKiEO1O6VqNEBxKvJJelCq0dTXWT5pbO2gDXC6h6QDXCaHo6pOHGPUy+YBaGQRGuSusMEASYiWunYN0vCAI8QaXnWMXNMdFP3jHAJH0eDsoiGnLPBlBp4TNm6rYI74nMzgz3B9IikW4WVK+dc8KZJZWYjAuORU3jc1c/NPskD2ASinf8v3xnfXeukU0sJ5N6m5E8VLjObPEO+mN2t/FZTMZLiFqPWc/ALSqnMnnhwrNi2rbfg/rd/IpL8Le3pSBne8+seeFVBoGqzHM9yXw==" >> "$SSH_CONFIG_DIR/known_hosts"
chmod 0600 "$SSH_CONFIG_DIR/known_hosts"
cd '/home/circleci/myapp'
git clone --depth 1 "$CIRCLE_REPOSITORY_URL" --branch "$CIRCLE_BRANCH" .
- run: ls -la ~/myapp/
ch-install-compass:
steps:
- run:
name: Install ruby
command: |
sudo apt update
sudo apt install ruby-dev -y
ruby -v
- run:
name: Install compass
command: sudo gem install compass
ch-ls:
steps:
- run:
name: ls
command: ls
ch-widget-npm:
steps:
- node/install-packages
ch-widget-bower:
steps:
- run:
name: Widget - Install Deps (bower)
command: npx bower install
ch-widget-build:
steps:
- run:
name: Widget - Build
command: npx grunt build:prod
ch-widget-test:
steps:
- run:
name: Widget - Test
command: |
npx grunt build:dev
# this only works by repeating it twice in the same step the test is invoked 🤷‍♀️
Xvfb -s "-nolisten tcp -nolisten unix" -ac :99 -screen 0 1280x1024x16 &
export DISPLAY=:99
Xvfb -s "-nolisten tcp -nolisten unix" -ac :99 -screen 0 1280x1024x16 &
export DISPLAY=:99
npx karma start --browsers Chrome --single-run --env=ci --reporters dots
npx karma start --browsers Chrome --single-run --loader-tests --env=ci --reporters dots
ch-elements-npm:
steps:
- node/install-packages:
app-dir: city-hive-custom-elements
ch-elements-build:
steps:
- run:
name: Custom Elements - Build
command: cd city-hive-custom-elements/; npm run build
ch-elements-test:
steps:
- run:
name: Custom Elements - Tests
command: |
# this only works by repeating it twice in the same step the test is invoked 🤷‍♀️
Xvfb -s "-nolisten tcp -nolisten unix" -ac :99 -screen 0 1280x1024x16 &
export DISPLAY=:99
Xvfb -s "-nolisten tcp -nolisten unix" -ac :99 -screen 0 1280x1024x16 &
export DISPLAY=:99
cd city-hive-custom-elements/; npm run test -- --watch=false
ch-install-chrome:
steps:
- run:
name: Install chrome deps
command: sudo apt update && sudo apt install xdg-utils fonts-liberation libasound2 libatk-bridge2.0-0 libatk1.0-0 libatspi2.0-0 libcairo2 libcups2 libgbm1 libgtk-3-0 libnspr4 libnss3 libpango-1.0-0 libxdamage1 libxkbcommon0
- run:
name: Install chrome
command: |
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo dpkg -i google-chrome-stable_current_amd64.deb
ch-karma-deps:
steps:
- run:
name: Karma - Deps
command: |
sudo apt-get install -y xvfb
ch-spin-widget-dev:
steps:
- run:
name: Spin Widget
command: npx grunt serve:dev
background: true
ch-spin-elements-dev:
steps:
- run:
name: Spin Custom Elements
command: cd city-hive-custom-elements/; npm run serve
background: true
ch-cypress-deps:
steps:
- run:
name: Install Cypress deps
command: sudo apt update && sudo apt install libgtk2.0-0 libgtk-3-0 libgbm-dev libnotify-dev libgconf-2-4 libnss3 libxss1 libasound2 libxtst6 xauth xvfb
ch-wait-for-servers:
steps:
- run:
name: Wait for local servers
command: |
npx wait-on http://localhost:4200
npx wait-on http://localhost:9003
ch-cypress:
steps:
- run:
name: Run Cypress tests
no_output_timeout: 10m
command: |
mkdir ~/test_reports
./bin/ci_e2e
- store_test_results:
path: ~/test_reports
- store_artifacts:
path: ~/test_reports
jobs:
build-widget:
working_directory: ~/myapp
docker:
- image: cimg/node:16.13.1
environment:
# Enable colors in the output
TERM: xterm
steps:
- ch-checkout
- ch-install-compass
- ch-widget-npm
- ch-widget-bower
- ch-widget-build
test-widget:
working_directory: ~/myapp
docker:
- image: cimg/node:16.13.1-browsers
environment:
# Enable colors in the output
TERM: xterm
steps:
- ch-checkout
- ch-install-compass
- ch-widget-npm
- ch-widget-bower
- ch-install-chrome
- ch-karma-deps
- ch-widget-test
build-elements:
working_directory: ~/myapp
docker:
- image: cimg/node:16.13.1
environment:
# Enable colors in the output
TERM: xterm
steps:
- ch-checkout
- ch-elements-npm
- ch-elements-build
test-elements:
working_directory: ~/myapp
docker:
- image: cimg/node:16.13.1-browsers
environment:
# Enable colors in the output
TERM: xterm
steps:
- ch-checkout
- ch-elements-npm
- ch-elements-build
- ch-install-chrome
- ch-karma-deps
- ch-elements-test
e2e:
working_directory: ~/myapp
docker:
- image: cimg/node:16.13.1-browsers
resource_class: large
environment:
# Enable colors in the output
TERM: xterm
parallelism: 31
steps:
- ch-checkout
- ch-install-compass
- ch-widget-npm
- ch-widget-bower
- ch-elements-npm
- ch-spin-widget-dev
- ch-spin-elements-dev
- ch-install-chrome
- ch-cypress-deps
- ch-wait-for-servers
- ch-cypress
workflows:
build-and-test:
jobs:
- build-widget:
post-steps:
- jira/notify
- test-widget:
post-steps:
- jira/notify
- build-elements:
post-steps:
- jira/notify
- test-elements:
post-steps:
- jira/notify
- e2e:
post-steps:
- jira/notify
Try using CIRCLECI_CLI_TOKEN environment variable instead:
CIRCLECI_CLI_TOKEN=d14c4********* \
circleci build \
--branch master \
--repo-url https://bitbucket.org/cityhive/city-hive-standalone-widget \
--job e2e
# goldylucks,
The --token flag is a global flag for the CLI setup:
--token string your token for using CircleCI, also CIRCLECI_CLI_TOKEN
For the Jira orb, the CircleCI API token is needed as an environment variable.
Looking at the output of circleci build -help, I see:
-e, --env -e VAR=VAL Set environment variables, e.g. -e VAR=VAL
This is the flag you'd need to use, as outlined in the CircleCI CLI documentation.

How to pass "yes" when pipeline asks an answer?

I'm trying to deploy a laravel app to my server. Here is my gitlab-ci.yml file:
image: edbizarro/gitlab-ci-pipeline-php:7.3
stages:
- build
- deploy
# Variables
variables:
MYSQL_ROOT_PASSWORD: root
MYSQL_USER: username
MYSQL_PASSWORD: password
MYSQL_DATABASE: database
DB_HOST: localhost:3306
PUBLIC_URL: https://example.com
build:
stage: build
tags:
- my-tag
- another-tag
script:
- echo "Building deploy package"
- echo "composer install"
- composer install --optimize-autoloader --no-dev
- mysql --version
- echo "Migrating database"
- php artisan migrate:fresh --seed
- echo "Dumping mysql database"
- mysqldump --host="${DB_HOST}" --user="${MYSQL_USER}" --password="${MYSQL_PASSWORD}" "${MYSQL_DATABASE}" > db.sql
- php artisan config:cache
- php artisan route:cache
- php artisan view:cache
- echo "Build successful"
artifacts:
expire_in: 1 hour
paths:
- build
deploy_production:
stage: deploy
tags:
- my-tag
- another-tag
script:
- echo "Current Directory:"
- pwd
- ls
- echo "Deploying to server"
- sudo cp -rv build/* /var/www/vhosts/example.com/
- echo "Deployed"
environment:
name: production
url: https://example.com
only:
- master
But it gives this error:
...
$ mysql --version
mysql Ver 15.1 Distrib 10.3.27-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
$ echo "Migrating database"
Migrating database
$ php artisan migrate:fresh --seed
**************************************
* Application In Production! *
**************************************
Do you really wish to run this command? (yes/no) [no]:
> Command Canceled!
Cleaning up file based variables
00:00
ERROR: Job failed: exit code 1
I'm using docker on gitlab-runner.
How can I handle this error?
php artisan has a --no-interaction flag you can use so it doesn't wait for a "yes" from an interactive user. E.g.:
php artisan migrate:fresh --seed --no-interaction

Circle CI - Can't connect to Redis or memcached using Docker Compose, but I can do so on my local machine

I'm developing a Node.js program that connects to both Redis and memcached. I am testing my Node.js program with Jest, and before running the test I run docker-compose up. My Node.js program connects to the Docker Redis and memcached Docker containers fine, and my tests pass fine on my local machine.
However, I want the tests to run on Circle CI so that every time I git push, the CI environment will verify the program is buildable and that tests are passing.
When I try to do the same on Circle CI, it seems that the Docker containers spin up fine, however the tests aren't able to connect to the Redis or memcached servers in the containers, despite it working fine on my local PC.
My config.yml for Circle CI:
version: 2
jobs:
build:
docker:
- image: circleci/node
steps:
- checkout
- setup_remote_docker
- run:
name: Install Docker Compose
command: |
curl -L https://github.com/docker/compose/releases/download/1.28.2/docker-compose-`uname -s`-`uname -m` > ~/docker-compose
chmod +x ~/docker-compose
sudo mv ~/docker-compose /usr/local/bin/docker-compose
- run:
name: Start Container
command: |
docker-compose up -d
docker-compose ps
- restore_cache:
key: npm-cache-v1-{{ checksum "package-lock.json" }}
- run:
name: Install Dependencies
command: npm ci
- save_cache:
key: npm-cache-v1-{{ checksum "package-lock.json" }}
paths:
- /home/circleci/.npm
- run:
name: Ensure Test Parity
command: |
chmod +x ./validateTestCases.sh
./validateTestCases.sh
- run:
name: Run Tests
command: npm test
My docker-compose.yml:
services:
redis:
image: redis
container_name: redis-container
ports:
- 6379:6379
memcached:
image: memcached
container_name: memcached-container
ports:
- 11211:11211
My build failing test log in Circle CI:
#!/bin/bash -eo pipefail
npm test
> easy-cache#1.0.0 test
> jest
FAIL memcached/memcached.test.js
● Test suite failed to run
Error: connect ECONNREFUSED 127.0.0.1:11211
FAIL redis/redis.test.js
● Test suite failed to run
Timeout - Async callback was not invoked within the 5000 ms timeout specified by jest.setTimeout.Error: Timeout - Async callback was not invoked within the 5000 ms timeout specified by jest.setTimeout.
at mapper (node_modules/jest-jasmine2/build/queueRunner.js:27:45)
Test Suites: 2 failed, 2 total
Tests: 0 total
Snapshots: 0 total
Time: 36.183 s
Ran all test suites.
npm ERR! code 1
npm ERR! path /home/circleci/project
npm ERR! command failed
npm ERR! command sh -c jest
npm ERR! A complete log of this run can be found in:
npm ERR! /home/circleci/.npm/_logs/2021-02-05T20_29_26_896Z-debug.log
Exited with code exit status 1
CircleCI received exit code 1
Link to my current source code
I am not sure what to try next. I have tried moving the npm test block right after docker-compose up -d but that had no effect.
It turns out that Docker Compose is not required for what I'm trying to do. Instead, you can include multiple Docker images in Circle CI.
Here's my updated Circle CI yaml file, where my tests run successfully (connection to Redis and memcached works like on my local PC using Docker Compose):
version: 2
jobs:
build:
docker:
- image: circleci/node
- image: redis
- image: memcached
steps:
- checkout
# - setup_remote_docker
# - run:
# name: Install Docker Compose
# command: |
# curl -L https://github.com/docker/compose/releases/download/1.28.2/docker-compose-`uname -s`-`uname -m` > ~/docker-compose
# chmod +x ~/docker-compose
# sudo mv ~/docker-compose /usr/local/bin/docker-compose
# - run:
# name: Start Container
# command: |
# docker-compose up -d
# docker-compose ps
- restore_cache:
key: npm-cache-v1-{{ checksum "package-lock.json" }}
- run:
name: Install Dependencies
command: npm ci
- save_cache:
key: npm-cache-v1-{{ checksum "package-lock.json" }}
paths:
- /home/circleci/.npm
- run:
name: Ensure Test Parity
command: |
chmod +x ./validateTestCases.sh
./validateTestCases.sh
- run:
name: Run Tests
command: npm test

Passing reports through docker in codecov gives error

I'm trying to setup codecov as code coverage tool in my repository. I referred to this link to pass reports through docker container -
Link - https://github.com/codecov/support/wiki/Testing-with-Docker
But travis ci fails and gives this error -
docker: Error parsing reference: "..." is not a valid repository/tag.
Here is my travis.yml
sudo: required
dist: trusty
language: node_js
node_js:
- 6
before_install:
- export CHROME_BIN=chromium-browser
- export DISPLAY=:99.0
- sh -e /etc/init.d/xvfb start
- docker run -v "$PWD/shared:/shared" ...
before_script:
- ng build
script:
- ng test --watch=false
- ng lint
- >
docker run -ti -v $(pwd):/app --workdir=/app coala/base coala --version
after_success:
- bash ./deploy.sh
- bash <(curl -s https://codecov.io/bash)
- mv -r coverage/ shared
cache:
bundler: true
directories:
- node_modules
- .coala-cache
services: docker
branches:
only:
- angular
How should I solve this? Thanks!
I assume you refer to Codecov Outside Docker. The current error message already tells you that the three dots ... need to be replaced with a real Docker repository name, e.g. node:6-alpine.
What you're still missing is the part of running the tests (including reports) inside the Docker container, so that you can mv the test reports to the shared folder. You could achieve that by adding a custom Dockerfile based on node, similar to the one below. I chose a more or less full base image including Chrome and other tools to make your use case work:
FROM markadams/chromium-xvfb-js:7
WORKDIR /proj
CMD npm install && \
node_modules/.bin/ng build && \
node_modules/.bin/ng test --watch=false && \
node_modules/.bin/ng lint && \
mkdir -p shared && \
mv coverage.txt shared
That custom image needs to be built and then run like this (assuming the Dockerfile to be in your project root directory):
docker build -t ci-build .
docker run --rm -v "$(pwd):/proj" ci-build
I suggest to change the .travis.yml like follows:
sudo: required
dist: trusty
language: node_js
node_js:
- 6
before_install:
- docker build -t ci-build .
script:
- >
docker run --rm -v $(pwd):/proj ci-build
- >
docker run -ti -v $(pwd):/app --workdir=/app coala/base coala --version
after_success:
- bash ./deploy.sh
- bash <(curl -s https://codecov.io/bash)
cache:
bundler: true
directories:
- node_modules
- .coala-cache
services: docker
branches:
only:
- angular
Another note: the coala/base image works similarly.

I am trying to use selenium in Capybara to locally test the rails app. I am using rake to start the task, but it gives following error

unable to connect to Mozilla geckodriver 127.0.0.1:4444 (Selenium::WebDriver::Error::WebDriverError)
addons:
firefox: 'latest'
before_script:
- bin/setup
- export PATH=$PATH:$HOME/travis_geckodriver/
- firefox --version
install:
- export DISPLAY=:99.0
- sh -e /etc/init.d/xvfb start
- sleep 3 # give xvfb some time to start
- mkdir $HOME/travis_geckodriver/
- cd $HOME/travis_geckodriver/
- curl -L https://github.com/mozilla/geckodriver/releases/download/v0.11.1/geckodriver-v0.11.1-linux64.tar.gz | tar xvz
- chmod +x geckodriver
- "cd -"
This allows you to use the most recent selenium / firefox combination. I struggled with this for hours yesterday and after pulling many leavers this is the combination that works for me. I hope it helps.

Resources