I am working on a Ruby on Rails project where AWSCLI is used.
Recently, When I push my code to Staging, it tries to import newkeys, PrivateKey, PublicKey from rsa.key and throws an error.
The error message is as follows:
SyntaxError: Non-ASCII character '\xc3' in file /opt/circleci/.pyenv/versions/2.7.12/lib/python2.7/site-packages/rsa/key.py on line 1, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details
Error message Image
My circleci/config.yml file is as follows:
orbs:
aws-ecr: circleci/aws-ecr#7.0.0
version: 2.1
jobs:
build:
working_directory: ~/mfds
docker:
- image: circleci/ruby:2.3.4-node
environment:
RAILS_ENV: citest
LANG: C.UTF-8
- image: snowhork/mysql_jp:5.7
environment:
MYSQL_ROOT_PASSWORD: ***
MYSQL_DATABASE: myapp
MYSQL_USER: user
MYSQL_PASSWORD: ***
MYSQL_ROOT_HOST: "%"
steps:
- checkout
- restore_cache:
key: gemfiles-{{ checksum "mfds/Gemfile.lock" }}
# Bundle install dependencies
- run: cd mfds/ && bundle install --path vendor/bundle
# Store bundle cache
- save_cache:
key: gemfiles-{{ checksum "mfds/Gemfile.lock" }}
paths:
- /home/circleci/mfds/mfds/vendor/bundle
- run:
name: Wait for db
command: dockerize -wait tcp://localhost:3306 -timeout 1m
# Database setup
- run:
name: db setting
command: |
cd mfds
bundle exec rake db:create
bundle exec rake db:ridgepole:apply[citest]
- run:
name: rspec
command: |
cd mfds
bundle exec rspec
staging_deploy:
machine: true
steps:
- checkout
- run:
name: asset compile
command: |
cd mfds
npm install
./node_modules/.bin/webpack --optimize-minimize
- run:
name: gem install
command: |
gem install aws-sdk-ecs
pip install futures
pip install --upgrade awscli
- run:
name: push image
command: |
export AWS_ACCESS_KEY_ID=$XX_AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY=$XX_AWS_SECRET_ACCESS_KEY
bin/deployv2/docker-push staging
- run:
name: update service
command: |
export AWS_ACCESS_KEY_ID=$XX_AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY=$XX_AWS_SECRET_ACCESS_KEY
bin/deployv2/update-task staging
....
I don't understand how to resolve this problem.
Related
I am running Cypress to test a Rails app on CircleCI. I have the tests running on CircleCI with the following config. But no video/screenshot assets are created in CircleCI artefacts if tests fail.
version: 2.1
orbs:
ruby: circleci/ruby#1.0.6
node: circleci/node#3.0.1
jobs:
build:
docker:
- image: cimg/ruby:2.7.2-node
steps:
- checkout # pull down our git code.
- ruby/install-deps # use the ruby orb to install dependencies
- node/install-packages:
pkg-manager: yarn
test:
parallelism: 3
docker:
- image: cimg/ruby:2.7.2-node # this is our primary docker image, where step commands run.
- image: circleci/postgres:12.3
environment: # add POSTGRES environment variables.
POSTGRES_USER: user
POSTGRES_PASSWORD: password
POSTGRES_DB: testdb
environment:
BUNDLE_JOBS: "3"
BUNDLE_RETRY: "3"
PGHOST: 127.0.0.1
PGUSER: user
PGPASSWORD: password
RAILS_ENV: test
steps:
- checkout
- ruby/install-deps
- node/install-packages:
pkg-manager: yarn
- run:
name: Wait for DB
command: dockerize -wait tcp://localhost:5432 -timeout 1m
- run:
name: Database setup
command: bundle exec rails db:schema:load --trace
- run:
name: Load test data
command: bundle exec rails db:seed --trace
- run:
name: Run Rails Server
background: true
command: CYPRESS=1 bundle exec rails s -p 5017
- run:
name: Wait for server
command: |
until $(curl --retry 10 --output /dev/null --silent --head --fail http://127.0.0.1:5017); do
printf '.'
sleep 5
done
- run: sudo apt-get update
- run: sudo apt-get install libgtk2.0-0 libgtk-3-0 libgbm-dev libnotify-dev libgconf-2-4 libnss3 libxss1 libasound2 libxtst6 xauth xvfb
- run: yarn cypress install --force
- run:
yarn: true
command: yarn run cypress run
- store_test_results:
path: test-reports/
So this is my config.yml
version: 2.1
orbs:
queue: eddiewebb/queue#1.5.0
executors:
node_postgres_redis:
docker:
- image: circleci/ruby:2.4.10-node-browsers
environment:
CC_TEST_REPORTER_ID: 7ceff1524bdc09dsd3e232321cf9daa531154170d590823232eddqw2330f3570
PGHOST: 127.0.0.1
RAILS_ENV: test
TEST_REPORT_PATH: "test/reports"
- image: circleci/postgres:9.6.2-alpine
environment:
POSTGRES_USER: circleci
POSTGRES_DB: circleci-jet-test
POSTGRES_PASSWORD: ""
- image: circleci/redis:3.2
ubuntu:
machine:
image: ubuntu-1604:202004-01
commands:
setup_and_run_test:
steps:
- checkout
# Restore bundle cache
- restore_cache:
key: jet-bundle-{{ .Branch }}-{{ checksum "Gemfile.lock" }}
# imagemagic
- run:
name: Setup Image Magick
command: |
sudo apt-get install libmagickwand-dev imagemagick imagemagick-6.q16 libmagickcore-dev
sudo ln -s /usr/lib/x86_64-linux-gnu/ImageMagick-6.9.7/bin-q16/Magick-config /usr/bin/Magick-config
- run:
name: Install Bundler
command: |
gem install bundler -v 1.17.3
# Install gem dependencies
- run: bundle check --path=vendor/bundle || bundle install --path=vendor/bundle
# Store bundle cache
- save_cache:
key: jet-bundle-{{ .Branch }}-{{ checksum "Gemfile.lock" }}
paths:
- vendor/bundle
# Database setup
- run:
name: Database Setup
command: |
bundle exec rake db:create
bundle exec rake db:migrate
bundle exec rake db:structure:load
Everythings runs properly until it tries to execute
bundle exec rake db:structure:load
Then it throws this error which is a bit weird
failed to execute:
psql -q -f /home/circleci/circleci-jet/db/structure.sql *****_test
Please check the output above for any errors and make sure that psql is installed in your PATH and has proper permissions.
I cannot understand this error and what needs to be done on my side.
Anyone with examples, please can show me direction.
Thanks
I'm trying to get a rails app to run on Circle CI. With the following configuration I can't seem to get anything to work at all. When I ssh in, I can't even run sudo.
Here's the error output:
install dependencies
$ #!/bin/bash -eo pipefail
bundler install --jobs=4 --retry=3 --path vendor/bundle
/bin/bash: bundler: command not found
Exited with code 127
I noticed there is nothing but assets/ under the vendor directory.
Here's my configuration file:
version: 2
jobs:
build:
working_directory: ~/repo
docker:
- image: circleci/postgres:9.6-alpine-postgis-ram
- image: circleci/ruby:2.3
environment:
PGHOST: 127.0.0.1
PGUSER: staging
RAILS_ENV: test
- image: circleci/postgres:10.5
environment:
POSTGRES_USER: staging
POSTGRES_DB: test
POSTGRES_PASSWORD: ""
steps:
- checkout
# Download and cache dependencies
- restore_cache:
keys:
- v1-dependencies-{{ checksum "Gemfile.lock" }}
# fallback to using the latest cache if no exact match is found
- v1-dependencies-
- run:
name: install dependencies
command: |
bundle install --jobs=4 --retry=3 --path vendor/bundle
- save_cache:
paths:
- ./vendor/bundle
key: v1-dependencies-{{ checksum "Gemfile.lock" }}
# Database setup
- run: bundle exec rake db:create
- run: bundle exec rake db:schema:load
# run tests!
- run:
name: run tests
command: |
mkdir /tmp/test-results
TEST_FILES="$(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings)"
bundle exec rspec --format progress \
--format RspecJunitFormatter \
--out /tmp/test-results/rspec.xml \
--format progress \
$TEST_FILES
# collect reports
- store_test_results:
path: /tmp/test-results
- store_artifacts:
path: /tmp/test-results
destination: test-results
All help is appreciated, thanks!
Your commands are running on the postgres container because it is listed first. If you make the ruby container first, then you will have access to bundle. And then when you SSH, you will be inside of the ruby container.
https://circleci.com/docs/2.0/configuration-reference/#docker--machine--macosexecutor
The first image listed in the file defines the primary container image where all steps will run.
Are Instrumentation tests for Android Espresso available on CircleCI 2.0?
If yes, can anybody, please, help to configure config.yml file for me?
I’ve made thousand attempts and no luck. I can run unit tests, but not Instrumentation.
Thanks
The answer for this question is: yes. Instrumentation tests are possible for CircleCi. This is the configuration I have:
version: 2
jobs:
build:
working_directory: ~/code
docker:
- image: circleci/android:api-25-alpha
environment:
JVM_OPTS: -Xmx3200m
steps:
- checkout
- restore_cache:
key: jars-{{ checksum "build.gradle" }}-{{ checksum "app/build.gradle" }}
- run:
name: Chmod permissions #if permission for Gradlew Dependencies fail, use this.
command: sudo chmod +x ./gradlew
- run:
name: Download Dependencies
command: ./gradlew androidDependencies
- save_cache:
paths:
- ~/.gradle
key: jars-{{ checksum "build.gradle" }}-{{ checksum "app/build.gradle" }}
- run:
name: Setup emulator
command: sdkmanager "system-images;android-25;google_apis;armeabi-v7a" && echo "no" | avdmanager create avd -n test -k "system-images;android-25;google_apis;armeabi-v7a"
- run:
name: Launch emulator
command: export LD_LIBRARY_PATH=${ANDROID_HOME}/emulator/lib64:${ANDROID_HOME}/emulator/lib64/qt/lib && emulator64-arm -avd test -noaudio -no-boot-anim -no-window -accel on
background: true
- run:
name: Wait emulator
command: |
# wait for it to have booted
circle-android wait-for-boot
# unlock the emulator screen
sleep 30
adb shell input keyevent 82
- run:
name: Run Tests
command: ./gradlew connectedAndroidTest
- store_artifacts:
path: app/build/reports
destination: reports
- store_test_results:
path: app/build/test-results
The only problem with this configuration that it doesn't lead to successfull build because of not enough memory error. If somebody has better configuration, please, share.
I am running Android UI tests on CircleCI MacOS executor.
Here is my configuration:
version: 2
reference:
## Constants
gradle_cache_path: &gradle_cache_path
gradle_cache-{{ checksum "build.gradle" }}-{{ checksum "app/build.gradle" }}
workspace: &workspace
~/src
## Configurations
android_config: &android_config
working_directory: *workspace
macos:
xcode: "9.4.0"
shell: /bin/bash --login -eo pipefail
environment:
TERM: dumb
JVM_OPTS: -Xmx3200m
## Cache
restore_gradle_cache: &restore_gradle_cache
restore_cache:
key: *gradle_cache_path
save_gradle_cache: &save_gradle_cache
save_cache:
key: *gradle_cache_path
paths:
- ~/.gradle
## Dependency Downloads
download_android_dependencies: &download_android_dependencies
run:
name: Download Android Dependencies
command: ./gradlew androidDependencies
jobs:
ui_test:
<<: *android_config
steps:
- checkout
- run:
name: Setup environment variables
command: |
echo 'export PATH="$PATH:/usr/local/opt/node#8/bin:${HOME}/.yarn/bin:${HOME}/${CIRCLE_PROJECT_REPONAME}/node_modules/.bin:/usr/local/share/android-sdk/tools/bin"' >> $BASH_ENV
echo 'export ANDROID_HOME="/usr/local/share/android-sdk"' >> $BASH_ENV
echo 'export ANDROID_SDK_HOME="/usr/local/share/android-sdk"' >> $BASH_ENV
echo 'export ANDROID_SDK_ROOT="/usr/local/share/android-sdk"' >> $BASH_ENV
echo 'export QEMU_AUDIO_DRV=none' >> $BASH_ENV
echo 'export JAVA_HOME=/Library/Java/Home' >> $BASH_ENV
- run:
name: Install Android sdk
command: |
HOMEBREW_NO_AUTO_UPDATE=1 brew tap homebrew/cask
HOMEBREW_NO_AUTO_UPDATE=1 brew cask install android-sdk
- run:
name: Install emulator dependencies
command: (yes | sdkmanager "platform-tools" "platforms;android-26" "extras;intel;Hardware_Accelerated_Execution_Manager" "build-tools;26.0.0" "system-images;android-26;google_apis;x86" "emulator" --verbose) || true
- *restore_gradle_cache
- *download_android_dependencies
- *save_gradle_cache
- run: avdmanager create avd -n Pixel_2_API_26 -k "system-images;android-26;google_apis;x86" -g google_apis -d "Nexus 5"
- run:
name: Run emulator in background
command: /usr/local/share/android-sdk/tools/emulator #Pixel_2_API_26 -skin 1080x2066 -memory 2048 -noaudio
background: true
- run:
name: Run Tests
command: ./gradlew app:connectedAndroidTest
https://gist.github.com/DoguD/58b4b86a5d892130af84074078581b87
I hope it helps
When our test suite runs we are getting the following issue regarding redis-server. No matter what we have tried, nothing seems to get past this error. We have validated via dockerize that the containers are live by waiting as seen below, but this error still occurs.
Any thoughts would be greatly appreciated!
Resque Initializer
require 'resque'
require 'redis'
require 'yaml'
# Resque Plugins
require 'resque/plugins/retry'
require 'resque-retry'
require 'resque-retry/server'
require 'resque-lock-timeout'
require 'resque-scheduler'
require 'resque/failure/multiple'
require 'resque/failure/redis'
require 'resque-job-stats/server'
require 'resque/rollbar'
if AppUnsecure.settings[:active_db_services].include?('redis')
uri = URI.parse(ENV["REDIS_URL"])
config = {
host: uri.host,
port: uri.port,
password: uri.password
}
Resque::Failure::Multiple.classes = [ Resque::Failure::Redis, Resque::Failure::Rollbar ]
Resque::Failure.backend = Resque::Failure::Multiple
Resque.redis = Redis.new(config)
elsif AppUnsecure.settings[:active_db_services].include?('redis-continous-integration')
Resque::Failure::MultipleWithRetrySuppression.classes = [Resque::Failure::Redis]
Resque.redis = Redis.new(host: 'redis://localhost', port: 6391)
else
Resque::Failure::MultipleWithRetrySuppression.classes = [Resque::Failure::Redis]
Resque.redis = Redis.new
end
Resque.redis.namespace = 'resque:GathrlySmartforms'
# Ignores Resque when processing jobs if activated!
Resque.inline = true if AppUnsecure.settings[:process_redis_inline]
# Setup Scheduler
# https://github.com/resque/resque-scheduler/issues/118
# https://github.com/resque/resque-scheduler/issues/581
Resque::Scheduler.configure do |c|
c.quiet = false
c.verbose = false
c.logfile = File.join(Rails.root, 'log', "#{Rails.env}_resque_scheduler.log")
c.logformat = 'text'
end
Resque::Scheduler.dynamic = true
schedules = {}
global = YAML.load_file("#{Rails.root}/config/resque_schedule.yml")
schedules.merge!(global) if global
# http://stackoverflow.com/questions/12158226/how-do-i-skip-loading-of-rails-initializers-when-running-a-rake-task
unless defined?(is_running_migration?) && is_running_migration?
Resque.schedule = schedules if schedules.present?
end
Resque::Server.class_eval do
use Rack::Auth::Basic do |username, password|
[username, password] == [Rails.application.secrets.my_resque_username, Rails.application.secrets.my_resque_password]
end
end
Circle Configuration
version: 2
jobs:
build:
working_directory: ~/DIR_NAME
docker:
- image: circleci/ruby:2.4.1-node
environment:
RAILS_ENV: continous_integration
PGHOST: 127.0.0.1
PGUSER: rails_test_user
- image: circleci/postgres:9.6.3-alpine
environment:
POSTGRES_USER: rails_test_user
POSTGRES_PASSWORD: ""
POSTGRES_DB: continous_integration
- image: redis:4.0.6
steps:
- checkout
- run:
name: Dockerize v0.6.0
command: |
wget https://github.com/jwilder/dockerize/releases/download/v0.6.0/dockerize-linux-amd64-v0.6.0.tar.gz
sudo rm -rf /usr/local/bin/dockerize
sudo tar -C /usr/local/bin -xzvf dockerize-linux-amd64-v0.6.0.tar.gz
rm dockerize-linux-amd64-v0.6.0.tar.gz
- run:
name: Wait for PG
command: dockerize -wait tcp://localhost:5432 -timeout 2m
- run:
name: Wait for Redis
command: |
dockerize -wait tcp://localhost:6379 -timeout 2m
- restore_cache:
keys:
- DIR_NAME-{{ checksum "Gemfile.lock" }}
- DIR_NAME-
- save_cache:
key: rails-demo-{{ checksum "Gemfile.lock" }}
paths:
- vendor/bundle
- run:
name: Setup Bundler and Gems
command: |
gem install bundler
gem update bundler
gem install brakeman
gem install rubocop
gem install rubocop-rspec
gem install scss_lint
gem install eslint-rails
gem install execjs
bundle config without development:test
bundle check --path=vendor/bundle || bundle install --without development test --path=vendor/bundle --jobs 4 --retry 3
- run:
name: Install Phantom Js
command: |
sudo curl --output /tmp/phantomjs https://s3.amazonaws.com/circle-downloads/phantomjs-2.1.1
sudo chmod ugo+x /tmp/phantomjs
sudo ln -sf /tmp/phantomjs /usr/local/bin/phantomjs
- run:
name: Install Postgres Tools
command: |
sudo apt-get update
sudo apt-get install postgresql-client
- run:
name: Install Redis Tools
command: |
sudo apt-get install redis-tools ; while ! redis-cli ping 2>/dev/null ; do sleep 1 ; done
- run:
name: Build Rails Database Yaml
command: |
cp config/database_example.yml config/database.yml
- run:
name: Setup Rails Database
command: |
bundle exec rake db:drop
bundle exec rake db:setup
- run:
name: Run Rspec
timeout: 60
command: |
RAILS_ENV=continous_integration bundle exec rspec --format RspecJunitFormatter -o /tmp/test-results/rspec.xml
- run:
name: Run Brakeman
command: |
brakeman -z
- run:
name: Run Rubocop
command: |
bundle exec rubocop --format fuubar --require rubocop-rspec --config .rubocop.yml
- run:
name: Run the SCSS Linter
command: |
bundle exec scss-lint --config=config/scsslint.yml
- run:
name: Run the Eslint Linter for JS
command: |
bundle exec rake eslint:run_all
- store_test_results:
path: /tmp/test-results
UPDATE
On various test runs it may occasionally work...however, the error is still the same, this must be the cause...
I don't know what Postgres connects to redis fine means exactly. The postgres user like through redis-cli? Postgres isn't connecting directly is it? (foreign data wrapper -- just read up on this ... I didn't even know you could do that, wow). Well, nevermind, that's sort of a tangent ...
Some minor insight, I can't test this whole thing very easily ... the PID file not being that is probably because it's just dying. The kill: invalid argument Q is probably from something doing kill -QUIT. This could be in redis' scripts or something. I bet the process is already dead and kill is just erroring out poorly. I think kill really wants the PID to exist. Although I just tested this and this is NOT how this works. So maybe Circle has non-GNU coreutils installed or something?
I don't think POSTGRES_USER should be in the redis image. See this example: https://discuss.circleci.com/t/circleci-bug-for-builds-with-redis/13124 Looks like things changed with 2.0.
If the gist has been updated to a completely new config, update the error messages and problem too, unless it's the same.
Based on what we can tell from here, it seems that your configuration is referencing a redis process that does not exist in test.
Also you're bundling without test. Not sure why.
Can you post your Redis configuration file? <- Still need this.
You need to configure REDIS_URL to the docker image.