I added an additional container in circle ci and tried to run tests parallaly.
This is my circle.yml file.
machine:
ruby:
version: 2.3.0
database:
post:
- cp config/sunspot.ci.yml config/sunspot.yml
- bundle exec sunspot-solr start -p 8981
dependencies:
pre:
- sudo apt-get update; sudo apt-get -y install solr-tomcat
test:
override:
- rvm use 2.3.0 && bundle exec rspec --color --format progress:
environment:
RAILS_ENV: test
parallel: true
files:
- "spec/**/*_spec.rb"
but tests don't seem to be running in parallel.
What am I missing? Thanks in advance
For some reason removing rvm use 2.3.0 from test section fixed my problem.
machine:
ruby:
version: 2.3.0
database:
# After default database task finished
post:
# Circle CI already replace our original sunspot.yml, replace it with backup copy `config/sunspot.yml.ci`
- cp config/sunspot.ci.yml config/sunspot.yml
- bundle exec sunspot-solr start -p 8981
dependencies:
pre:
- sudo apt-get update; sudo apt-get -y install solr-tomcat
test:
override:
- bundle exec rspec --tag ~#flaky --color --profile 20 --format progress:
timeout: 240
environment:
RAILS_ENV: test
parallel: true
files:
- my files here
Related
I am running Cypress to test a Rails app on CircleCI. I have the tests running on CircleCI with the following config. But no video/screenshot assets are created in CircleCI artefacts if tests fail.
version: 2.1
orbs:
ruby: circleci/ruby#1.0.6
node: circleci/node#3.0.1
jobs:
build:
docker:
- image: cimg/ruby:2.7.2-node
steps:
- checkout # pull down our git code.
- ruby/install-deps # use the ruby orb to install dependencies
- node/install-packages:
pkg-manager: yarn
test:
parallelism: 3
docker:
- image: cimg/ruby:2.7.2-node # this is our primary docker image, where step commands run.
- image: circleci/postgres:12.3
environment: # add POSTGRES environment variables.
POSTGRES_USER: user
POSTGRES_PASSWORD: password
POSTGRES_DB: testdb
environment:
BUNDLE_JOBS: "3"
BUNDLE_RETRY: "3"
PGHOST: 127.0.0.1
PGUSER: user
PGPASSWORD: password
RAILS_ENV: test
steps:
- checkout
- ruby/install-deps
- node/install-packages:
pkg-manager: yarn
- run:
name: Wait for DB
command: dockerize -wait tcp://localhost:5432 -timeout 1m
- run:
name: Database setup
command: bundle exec rails db:schema:load --trace
- run:
name: Load test data
command: bundle exec rails db:seed --trace
- run:
name: Run Rails Server
background: true
command: CYPRESS=1 bundle exec rails s -p 5017
- run:
name: Wait for server
command: |
until $(curl --retry 10 --output /dev/null --silent --head --fail http://127.0.0.1:5017); do
printf '.'
sleep 5
done
- run: sudo apt-get update
- run: sudo apt-get install libgtk2.0-0 libgtk-3-0 libgbm-dev libnotify-dev libgconf-2-4 libnss3 libxss1 libasound2 libxtst6 xauth xvfb
- run: yarn cypress install --force
- run:
yarn: true
command: yarn run cypress run
- store_test_results:
path: test-reports/
So this is my config.yml
version: 2.1
orbs:
queue: eddiewebb/queue#1.5.0
executors:
node_postgres_redis:
docker:
- image: circleci/ruby:2.4.10-node-browsers
environment:
CC_TEST_REPORTER_ID: 7ceff1524bdc09dsd3e232321cf9daa531154170d590823232eddqw2330f3570
PGHOST: 127.0.0.1
RAILS_ENV: test
TEST_REPORT_PATH: "test/reports"
- image: circleci/postgres:9.6.2-alpine
environment:
POSTGRES_USER: circleci
POSTGRES_DB: circleci-jet-test
POSTGRES_PASSWORD: ""
- image: circleci/redis:3.2
ubuntu:
machine:
image: ubuntu-1604:202004-01
commands:
setup_and_run_test:
steps:
- checkout
# Restore bundle cache
- restore_cache:
key: jet-bundle-{{ .Branch }}-{{ checksum "Gemfile.lock" }}
# imagemagic
- run:
name: Setup Image Magick
command: |
sudo apt-get install libmagickwand-dev imagemagick imagemagick-6.q16 libmagickcore-dev
sudo ln -s /usr/lib/x86_64-linux-gnu/ImageMagick-6.9.7/bin-q16/Magick-config /usr/bin/Magick-config
- run:
name: Install Bundler
command: |
gem install bundler -v 1.17.3
# Install gem dependencies
- run: bundle check --path=vendor/bundle || bundle install --path=vendor/bundle
# Store bundle cache
- save_cache:
key: jet-bundle-{{ .Branch }}-{{ checksum "Gemfile.lock" }}
paths:
- vendor/bundle
# Database setup
- run:
name: Database Setup
command: |
bundle exec rake db:create
bundle exec rake db:migrate
bundle exec rake db:structure:load
Everythings runs properly until it tries to execute
bundle exec rake db:structure:load
Then it throws this error which is a bit weird
failed to execute:
psql -q -f /home/circleci/circleci-jet/db/structure.sql *****_test
Please check the output above for any errors and make sure that psql is installed in your PATH and has proper permissions.
I cannot understand this error and what needs to be done on my side.
Anyone with examples, please can show me direction.
Thanks
I'm trying to get a rails app to run on Circle CI. With the following configuration I can't seem to get anything to work at all. When I ssh in, I can't even run sudo.
Here's the error output:
install dependencies
$ #!/bin/bash -eo pipefail
bundler install --jobs=4 --retry=3 --path vendor/bundle
/bin/bash: bundler: command not found
Exited with code 127
I noticed there is nothing but assets/ under the vendor directory.
Here's my configuration file:
version: 2
jobs:
build:
working_directory: ~/repo
docker:
- image: circleci/postgres:9.6-alpine-postgis-ram
- image: circleci/ruby:2.3
environment:
PGHOST: 127.0.0.1
PGUSER: staging
RAILS_ENV: test
- image: circleci/postgres:10.5
environment:
POSTGRES_USER: staging
POSTGRES_DB: test
POSTGRES_PASSWORD: ""
steps:
- checkout
# Download and cache dependencies
- restore_cache:
keys:
- v1-dependencies-{{ checksum "Gemfile.lock" }}
# fallback to using the latest cache if no exact match is found
- v1-dependencies-
- run:
name: install dependencies
command: |
bundle install --jobs=4 --retry=3 --path vendor/bundle
- save_cache:
paths:
- ./vendor/bundle
key: v1-dependencies-{{ checksum "Gemfile.lock" }}
# Database setup
- run: bundle exec rake db:create
- run: bundle exec rake db:schema:load
# run tests!
- run:
name: run tests
command: |
mkdir /tmp/test-results
TEST_FILES="$(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings)"
bundle exec rspec --format progress \
--format RspecJunitFormatter \
--out /tmp/test-results/rspec.xml \
--format progress \
$TEST_FILES
# collect reports
- store_test_results:
path: /tmp/test-results
- store_artifacts:
path: /tmp/test-results
destination: test-results
All help is appreciated, thanks!
Your commands are running on the postgres container because it is listed first. If you make the ruby container first, then you will have access to bundle. And then when you SSH, you will be inside of the ruby container.
https://circleci.com/docs/2.0/configuration-reference/#docker--machine--macosexecutor
The first image listed in the file defines the primary container image where all steps will run.
When our test suite runs we are getting the following issue regarding redis-server. No matter what we have tried, nothing seems to get past this error. We have validated via dockerize that the containers are live by waiting as seen below, but this error still occurs.
Any thoughts would be greatly appreciated!
Resque Initializer
require 'resque'
require 'redis'
require 'yaml'
# Resque Plugins
require 'resque/plugins/retry'
require 'resque-retry'
require 'resque-retry/server'
require 'resque-lock-timeout'
require 'resque-scheduler'
require 'resque/failure/multiple'
require 'resque/failure/redis'
require 'resque-job-stats/server'
require 'resque/rollbar'
if AppUnsecure.settings[:active_db_services].include?('redis')
uri = URI.parse(ENV["REDIS_URL"])
config = {
host: uri.host,
port: uri.port,
password: uri.password
}
Resque::Failure::Multiple.classes = [ Resque::Failure::Redis, Resque::Failure::Rollbar ]
Resque::Failure.backend = Resque::Failure::Multiple
Resque.redis = Redis.new(config)
elsif AppUnsecure.settings[:active_db_services].include?('redis-continous-integration')
Resque::Failure::MultipleWithRetrySuppression.classes = [Resque::Failure::Redis]
Resque.redis = Redis.new(host: 'redis://localhost', port: 6391)
else
Resque::Failure::MultipleWithRetrySuppression.classes = [Resque::Failure::Redis]
Resque.redis = Redis.new
end
Resque.redis.namespace = 'resque:GathrlySmartforms'
# Ignores Resque when processing jobs if activated!
Resque.inline = true if AppUnsecure.settings[:process_redis_inline]
# Setup Scheduler
# https://github.com/resque/resque-scheduler/issues/118
# https://github.com/resque/resque-scheduler/issues/581
Resque::Scheduler.configure do |c|
c.quiet = false
c.verbose = false
c.logfile = File.join(Rails.root, 'log', "#{Rails.env}_resque_scheduler.log")
c.logformat = 'text'
end
Resque::Scheduler.dynamic = true
schedules = {}
global = YAML.load_file("#{Rails.root}/config/resque_schedule.yml")
schedules.merge!(global) if global
# http://stackoverflow.com/questions/12158226/how-do-i-skip-loading-of-rails-initializers-when-running-a-rake-task
unless defined?(is_running_migration?) && is_running_migration?
Resque.schedule = schedules if schedules.present?
end
Resque::Server.class_eval do
use Rack::Auth::Basic do |username, password|
[username, password] == [Rails.application.secrets.my_resque_username, Rails.application.secrets.my_resque_password]
end
end
Circle Configuration
version: 2
jobs:
build:
working_directory: ~/DIR_NAME
docker:
- image: circleci/ruby:2.4.1-node
environment:
RAILS_ENV: continous_integration
PGHOST: 127.0.0.1
PGUSER: rails_test_user
- image: circleci/postgres:9.6.3-alpine
environment:
POSTGRES_USER: rails_test_user
POSTGRES_PASSWORD: ""
POSTGRES_DB: continous_integration
- image: redis:4.0.6
steps:
- checkout
- run:
name: Dockerize v0.6.0
command: |
wget https://github.com/jwilder/dockerize/releases/download/v0.6.0/dockerize-linux-amd64-v0.6.0.tar.gz
sudo rm -rf /usr/local/bin/dockerize
sudo tar -C /usr/local/bin -xzvf dockerize-linux-amd64-v0.6.0.tar.gz
rm dockerize-linux-amd64-v0.6.0.tar.gz
- run:
name: Wait for PG
command: dockerize -wait tcp://localhost:5432 -timeout 2m
- run:
name: Wait for Redis
command: |
dockerize -wait tcp://localhost:6379 -timeout 2m
- restore_cache:
keys:
- DIR_NAME-{{ checksum "Gemfile.lock" }}
- DIR_NAME-
- save_cache:
key: rails-demo-{{ checksum "Gemfile.lock" }}
paths:
- vendor/bundle
- run:
name: Setup Bundler and Gems
command: |
gem install bundler
gem update bundler
gem install brakeman
gem install rubocop
gem install rubocop-rspec
gem install scss_lint
gem install eslint-rails
gem install execjs
bundle config without development:test
bundle check --path=vendor/bundle || bundle install --without development test --path=vendor/bundle --jobs 4 --retry 3
- run:
name: Install Phantom Js
command: |
sudo curl --output /tmp/phantomjs https://s3.amazonaws.com/circle-downloads/phantomjs-2.1.1
sudo chmod ugo+x /tmp/phantomjs
sudo ln -sf /tmp/phantomjs /usr/local/bin/phantomjs
- run:
name: Install Postgres Tools
command: |
sudo apt-get update
sudo apt-get install postgresql-client
- run:
name: Install Redis Tools
command: |
sudo apt-get install redis-tools ; while ! redis-cli ping 2>/dev/null ; do sleep 1 ; done
- run:
name: Build Rails Database Yaml
command: |
cp config/database_example.yml config/database.yml
- run:
name: Setup Rails Database
command: |
bundle exec rake db:drop
bundle exec rake db:setup
- run:
name: Run Rspec
timeout: 60
command: |
RAILS_ENV=continous_integration bundle exec rspec --format RspecJunitFormatter -o /tmp/test-results/rspec.xml
- run:
name: Run Brakeman
command: |
brakeman -z
- run:
name: Run Rubocop
command: |
bundle exec rubocop --format fuubar --require rubocop-rspec --config .rubocop.yml
- run:
name: Run the SCSS Linter
command: |
bundle exec scss-lint --config=config/scsslint.yml
- run:
name: Run the Eslint Linter for JS
command: |
bundle exec rake eslint:run_all
- store_test_results:
path: /tmp/test-results
UPDATE
On various test runs it may occasionally work...however, the error is still the same, this must be the cause...
I don't know what Postgres connects to redis fine means exactly. The postgres user like through redis-cli? Postgres isn't connecting directly is it? (foreign data wrapper -- just read up on this ... I didn't even know you could do that, wow). Well, nevermind, that's sort of a tangent ...
Some minor insight, I can't test this whole thing very easily ... the PID file not being that is probably because it's just dying. The kill: invalid argument Q is probably from something doing kill -QUIT. This could be in redis' scripts or something. I bet the process is already dead and kill is just erroring out poorly. I think kill really wants the PID to exist. Although I just tested this and this is NOT how this works. So maybe Circle has non-GNU coreutils installed or something?
I don't think POSTGRES_USER should be in the redis image. See this example: https://discuss.circleci.com/t/circleci-bug-for-builds-with-redis/13124 Looks like things changed with 2.0.
If the gist has been updated to a completely new config, update the error messages and problem too, unless it's the same.
Based on what we can tell from here, it seems that your configuration is referencing a redis process that does not exist in test.
Also you're bundling without test. Not sure why.
Can you post your Redis configuration file? <- Still need this.
You need to configure REDIS_URL to the docker image.
I have a Rails App hosted on gitlab.com, and I am configuring it to deploy to heroku following this guide: http://docs.gitlab.com/ce/ci/examples/test-and-deploy-ruby-application-to-heroku.html. It works fine.
My question is, how can I run migrations every time I deploy to heroku? When deploying via CLI I would usually do:
git push heroku master && heroku run rake db:migrate
but using gitlab-ci.yml I have no clue on how to do this...
If you want to be able to use the full power of the Heroku CLI in your GitLab CI process (including having the build fail if a migration fails for whichever reason), you can also try this approach which will install the Heroku CLI and deliver status codes of your Heroku commands back to GitLab, as well as, of course, the command line output. Using heroku run without credentials on the commandline require the HEROKU_API_KEY environment variable to be set to a key which has access to the app in question.
before_script:
- echo "deb http://toolbelt.heroku.com/ubuntu ./" > /etc/apt/sources.list.d/heroku.list
- wget -O- https://toolbelt.heroku.com/apt/release.key | apt-key add -
- apt-get update
- apt-get install -y heroku-toolbelt
- gem install dpl
stages:
- deploy
test_on_heroku:
type: deploy
script:
- dpl --provider=heroku --app=my_heroku_app --api-key=$HEROKU_API_KEY
- heroku run <your command here> --exit-code --app my_heroku_app
I actually run my tests on an Heroku instance to be sure, the environment is exactly the same. This is where this comes in real handy.
The information in this answer may be out of date. Please see both answers below, and remember to upvote the answers that are up to date to help future visitors.
here is a sample .yml I have that runs my tests then pushed to Heroku stage (for master branch pushes) or production (for tags pushes)
image: "ruby:2.3"
test:
script:
- apt-get update -qy
- apt-get install -y nodejs
- gem install bundler
- bundle install -j $(nproc) --without production
- bundle exec rails db:create RAILS_ENV=test
- bundle exec rails db:migrate RAILS_ENV=test
- bundle exec rails RAILS_ENV=test
staging:
type: deploy
environment: staging
script:
- gem install dpl
- dpl --provider=heroku --app=$HEROKU_STAGING_APP_NAME --api-key=$HEROKU_API_KEY
- "curl -n -X POST https://api.heroku.com/apps/$HEROKU_STAGING_APP_NAME/ps -H \"Accept: application/json\" -H \"Authorization: Bearer ${HEROKU_API_KEY}\" -d \"command=bundle exec rails db:migrate\""
only:
- master
production:
type: deploy
environment: production
script:
- gem install dpl
- dpl --provider=heroku --app=$HEROKU_PRODUCTION_APP_NAME --api-key=$HEROKU_API_KEY
- "curl -n -X POST https://api.heroku.com/apps/$HEROKU_PRODUCTION_APP_NAME/ps -H \"Accept: application/json\" -H \"Authorization: Bearer ${HEROKU_API_KEY}\" -d \"command=bundle exec rails db:migrate\""
only:
- tags
To update #huesforalice's answer, this would also work for the new Heroku CLI, which replaced Heroku Toolbelt in November 2016:
before_script:
- apt-get update
- apt-get install apt-transport-https
- echo "deb https://cli-assets.heroku.com/branches/stable/apt ./" > /etc/apt/sources.list.d/heroku.list
- wget -O- https://cli-assets.heroku.com/apt/release.key | apt-key add -
- apt-get update
- apt-get install -y heroku
- gem install dpl
staging:
type: deploy
variables:
HEROKU_API_KEY: $HEROKU_STAGING_API_KEY
script:
- dpl --provider=heroku --app=$HEROKU_STAGING_APP --api-key=$HEROKU_STAGING_API_KEY
- heroku run rails db:migrate --exit-code --app $HEROKU_STAGING_APP
only:
- master
production:
type: deploy
variables:
HEROKU_API_KEY: $HEROKU_PRODUCTION_API_KEY
script:
- dpl --provider=heroku --app=$HEROKU_PRODUCTION_APP --api-key=$HEROKU_PRODUCTION_API_KEY
- heroku run rails db:migrate --exit-code --app $HEROKU_PRODUCTION_APP
only:
- tags
To further improve #huesforalice and #Jimmy Bosse 's answers - If you want to
avoid putting Heroku CLI installation in the global before_script but only use it in the deployment stages
at the same time, avoid copying and pasting the installation snippet into different stages
You can do something like this using YAML anchors to DRY up
before_script:
# the global before_script
- gem install bundler --no-document
- bundle check || bundle install --jobs $(nproc)
.deployment_before_script: &deployment_before_script
before_script:
- echo "deb http://toolbelt.heroku.com/ubuntu ./" > /etc/apt/sources.list.d/heroku.list
- wget -O- https://toolbelt.heroku.com/apt/release.key | apt-key add -
- apt-get update
- apt-get install -y heroku-toolbelt
- gem install dpl
# other stages...
staging:
stage: deploy
<<: *deployment_before_script
script:
- dpl --provider=heroku --app=$HEROKU_APP_STAGING --api-key=$HEROKU_API_KEY_STAGING
- heroku run bundle exec rails db:migrate --exit-code --app $HEROKU_APP_STAGING
only:
- master
production:
stage: deploy
<<: *deployment_before_script
script:
- dpl --provider=heroku --app=$HEROKU_APP_PRODUCTION --api-key=$HEROKU_API_KEY_STAGING
- heroku run bundle exec rails db:migrate --exit-code --app $HEROKU_APP_PRODUCTION
when: manual
only:
- tags