GitHub Actions - share services across jobs - ruby-on-rails

I would like to know how I can share service containers between jobs in GitHub Actions. With this workflow currently the containers get destroyed after the build step.
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
env:
RAILS_ENV: test
RACK_ENV: test
RAILS_MASTER_KEY: ${{ secrets.RAILS_MASTER_KEY }}
POSTGRES_PASSWORD: postgres15
POSTGRES_USERNAME: postgres
POSTGRES_HOST: localhost
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout#v3
- name: Initialize Ruby
uses: ruby/setup-ruby#v1
with:
bundler-cache: true
- name: Setup Rails
run: bin/setup
services:
postgres:
image: postgres:15.1-alpine
ports:
- 5432:5432
env:
POSTGRES_PASSWORD: ${{ env.POSTGRES_PASSWORD }}
POSTGRES_USER: ${{ env.POSTGRES_USERNAME }}
# needed because the postgres container does not provide a healthcheck
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
test:
runs-on: ubuntu-latest
needs: [ build ]
steps:
- name: Lint Ruby
run: bundle exec rubocop
- name: Run tests
run: bin/rails test:all
coverage:
runs-on: ubuntu-latest
needs: [ build, test ]
steps:
- uses: joshmfrankel/simplecov-check-action#main
with:
minimum_suite_coverage: 98
minimum_file_coverage: 90
github_token: ${{ secrets.GITHUB_TOKEN }}
check_job_name: coverage

As per the documentation, you cannot share service containers across jobs.
https://docs.github.com/en/actions/using-containerized-services/about-service-containers
You can configure service containers for each job in a workflow. GitHub creates a fresh Docker container for each service configured in the workflow, and destroys the service container when the job completes. Steps in a job can communicate with all service containers that are part of the same job. However, you cannot create and use service containers inside a composite action.

Related

Github Actions Application is being build twice

I have a problem. I am starting to work with Github Actions, and I got a working pipeline where I am trying to:
Build the docker application
Run the tests
Publish to docker repository
But the test running and publishing are 2 separate jobs, so I am building the image twice. Here is the workflow code:
name: Test & Publish Docker Image
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
jobs:
run_application_tests:
name: Run test suite
runs-on: ubuntu-latest
env:
COMPOSE_FILE: docker-compose.yml
DOCKER_USER: ${{ secrets.DOCKER_USER }}
DOCKER_PASS: ${{ secrets.DOCKER_PASS }}
steps:
- name: Checkout code
uses: actions/checkout#v3
- name: Login To Docker Hub
uses: docker/login-action#v2
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Build and run docker image
run: docker-compose up -d --build
- name: Run migrations
run: make migrate
- name: Run tests
run: make test
build_and_publish_docker_image:
needs: run_application_tests
name: Build & Publish Docker Images
runs-on: ubuntu-latest
steps:
- name: Checkout The Repo
uses: actions/checkout#v3
- name: Login To Docker Hub
uses: docker/login-action#v2
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Build & Push Docker Image
uses: docker/build-push-action#v3
with:
push: true
tags: |
me/image:latest
me/image:${{ github.sha }}
The docker image is a Ruby-On-Rails application so the bundle install takes very long. Each build takes about 2-3 minutes.
I also tried adding:
cache-from: type=gha
cache-to: type=gha,mode=max
To the docker/build-push-action#v3, but that resulted in:
Error: buildx failed with: ERROR: cache export feature is currently not supported for docker driver. Please switch to a different driver (eg. "docker buildx create --use")
What can I change to improve this pipeline time based?
in your job run_application_tests you have this code:
- name: Login To Docker Hub
uses: docker/login-action#v2
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Build and run docker image
run: docker-compose up -d --build
this code runs docker-compose why?
this is how your code should be if you want to run tests only:
jobs:
run_application_tests:
name: Run test suite
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout#v3
- name: Run migrations
run: make migrate
- name: Run tests
run: make test

Gem::RemoteFetcher::FetchError bad response Unauthorized 401 sidekiqpro github actions

Hi every one I have a problem when I executed my ci with github actions I received this problem.
Retrying download gem from https://gems.contribsys.com/ due to error (2/4): Gem::RemoteFetcher::FetchError bad response Unauthorized 401
this is my .yml file to github actions
jobs:
test:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:13-alpine
ports:
- "5432:5432"
options:
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
env:
POSTGRES_DB: rails_test
env:
RAILS_ENV: test
DATABASE_URL: "postgres://rails:password#localhost:5432/rails_test"
SIDEKIQ_PRO_USER: ${{ secrets.SIDEKIQ_PRO_USER }}
SIDEKIQ_PRO_PWD: ${{ secrets.SIDEKIQ_PRO_PWD }}
lint:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout#v3
- name: Install Ruby and gems
uses: ruby/setup-ruby#8f312efe1262fb463d906e9bf040319394c18d3e # v1.92
with:
bundler-cache: true
# Add or replace any other lints here
- name: Security audit dependencies
run: bin/bundler-audit --update |
bundle config gems.contribsys.com ${secrets.SIDEKIQ_PRO_USER}:${ secrets.SIDEKIQ_PRO_PWD }
- name: Security audit application code
run: bin/brakeman -q -w2
- name: Lint Ruby files
run: bin/rubocop --parallel
in my gemfile I put this
gem 'sidekiq-pro', '~> 5.3', source: 'https://gems.contribsys.com'
For other hands, how I can print my ENV in console of github actions , I am not sure if the variable that I set is filled , any ideas?

Cache created in `ubuntu-latest` cannot be restored in Docker container

Here's my workflow file:
name: Build Pipeline
on: push
env:
NODE_VERSION: 11
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: actions/setup-node#v2
with:
node-version: ${{ env.NODE_VERSION }}
- id: cache-node-modules
uses: actions/cache#v2
with:
path: ${{ github.workspace }}/node_modules
key: node_modules-${{ hashFiles('package-lock.json') }}
restore-keys: node_modules
- uses: actions/cache#v2
with:
path: ${{ github.workspace }}/build
key: build-${{ github.sha }}
restore-keys: build
- if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm install
- run: npm run build -- --incremental
npm-scripts:
needs: [build]
runs-on: ubuntu-latest
strategy:
matrix:
script: ['lint:pipeline', 'lint:exports', 'i18n:pipeline', 'schema:validate']
steps:
- uses: actions/checkout#v2
- uses: actions/setup-node#v2
with:
node-version: ${{ env.NODE_VERSION }}
- id: cache-node-modules
uses: actions/cache#v2
with:
path: ${{ github.workspace }}/node_modules
key: node_modules-${{ hashFiles('package-lock.json') }}
- if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: |
echo 'Expected to have a cache hit for "node_modules", since this job runs after the "build" job, which caches the latest version of "node_modules". Not having a cache hit means probably there is a bug with the workflow file.'
exit 1
- id: cache-build-output
uses: actions/cache#v2
with:
path: ${{ github.workspace }}/build
key: build-${{ github.sha }}
- if: steps.cache-build-output.outputs.cache-hit != 'true'
run: |
echo 'Expected to have a cache hit for the build output folder, since this job runs after the "build" job, which caches the latest version of the "build" folder. Not having a cache hit means probably there is a bug with the workflow file.'
exit 1
- run: npm run ${{ matrix.script }}
jest-tests:
needs: [build]
runs-on: ubuntu-latest
container: node:11
services:
postgres:
image: postgres
env:
POSTGRES_DB: localhost
POSTGRES_USER: postgres
POSTGRES_PASSWORD: password
redis:
image: redis
steps:
- uses: actions/checkout#v2
- id: cache-node-modules
uses: actions/cache#v2
with:
path: ${{ github.workspace }}/node_modules
key: node_modules-${{ hashFiles('package-lock.json') }}
- if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: |
echo 'Expected to have a cache hit for "node_modules", since this job runs after the "build" job, which caches the latest version of "node_modules". Not having a cache hit means probably there is a bug with the workflow file.'
exit 1
- id: cache-build-output
uses: actions/cache#v2
with:
path: ${{ github.workspace }}/build
key: build-${{ github.sha }}
- if: steps.cache-build-output.outputs.cache-hit != 'true'
run: |
echo 'Expected to have a cache hit for the build output folder, since this job runs after the "build" job, which caches the latest version of the "build" folder. Not having a cache hit means probably there is a bug with the workflow file.'
exit 1
- run: echo
node_modules and build folders are cached in the build job. These caches are able to be restored without a problem in the npm-scripts job. However, they are not able to be restored in the jest-tests job, where it gets a Cache not found for input keys error.
I don't know how this is possible, since the exact same cache keys are able to be restored without a problem in all of the npm-scripts jobs.
When I remove the:
container: node:11
services:
postgres:
image: postgres
env:
POSTGRES_DB: localhost
POSTGRES_USER: postgres
POSTGRES_PASSWORD: password
redis:
image: redis
part (and hence let the job run on ubuntu-latest, instead of a Docker container), the cache is able to be restored again properly. So not sure what's going on here.
It seems that the #actions/cache job silently fails if there is no zstd binary available in the PATH in the container that you are running in. This may be the case for your Node container.
I found this out by setting ACTIONS_STEP_DEBUG to true in the repository secrets. The debug log shows that the action tries to run zstd and can't, but it is instead reported as a cache miss. Once I figured that out, I found that there is a bug report open for it: https://github.com/actions/cache/issues/580
It is a weird bug. The workaround that I found is not running the jest-tests job in a container. That is, running the jest-tests job in a regular, ubuntu-latest machine, and mapping the service container ports like:
jest-tests:
needs: [build]
runs-on: ubuntu-latest
services:
postgres:
image: postgres
ports:
- 5432:5432
env:
POSTGRES_DB: localhost
POSTGRES_USER: postgres
POSTGRES_PASSWORD: password
redis:
image: redis
ports:
- 6379:6379
I'm using a custom image and just by adding zstd package to the image, it made the action/cache to work.

How to use parallel_tests in github actions

I'm trying to use parallel_tests in my github action to run my test suite but I was not able to find a proper solution.
The official docs has one but it is for gitlab:
https://github.com/grosser/parallel_tests/wiki/Distributed-Parallel-Tests-on-CI-systems
Any help would be appreciated thanks!
Here's a sample workflow you can drop into .github/workflows/tests.yml:
name: Rails Tests
on: push
env:
PGHOST: localhost
PGUSER: postgres
RAILS_ENV: test
jobs:
build:
runs-on: ubuntu-latest
strategy:
fail-fast: true
matrix:
# Set N number of parallel jobs you want to run
# Remember to update ci_node_index below to 0..N-1
ci_node_total: [6]
# set N-1 indexes for parallel jobs
# When you run 2 parallel jobs then first job will have index 0, the second job will have index 1 etc
ci_node_index: [0, 1, 2, 3, 4, 5]
services:
postgres:
image: postgres:11.5
ports: ["5432:5432"]
options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
redis:
image: redis:5
ports: ["6379:6379"]
steps:
- uses: actions/checkout#v1
- uses: ruby/setup-ruby#v1
with:
bundler-cache: true
- name: Set node version (from .tool-versions)
run: echo "NODE_VERSION=$(cat .tool-versions | grep nodejs | sed 's/^nodejs //')" >> $GITHUB_ENV
- uses: actions/setup-node#v2
with:
node-version: ${{ env.NODE_VERSION }}
- uses: bahmutov/npm-install#v1
- name: Install PostgreSQL client
run: |
sudo apt-get -yqq install libpq-dev postgresql-client
- name: Test Prep
env:
CI_NODE_INDEX: ${{ matrix.ci_node_index }}
run: |
bundle exec rake parallel:create["1"] parallel:load_schema["1"]
- name: Run tests
env:
RAILS_MASTER_KEY: ${{ secrets.RAILS_MASTER_KEY }}
CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
CI_NODE_INDEX: ${{ matrix.ci_node_index }}
run : |
bundle exec parallel_test spec/ -n $CI_NODE_TOTAL --only-group $CI_NODE_INDEX --type rspec
This was what I used to get it solved. Note: There are some other omitted parts such as the setup for ruby, postgresql, sqlite, etc.
name: "Lint and Test"
jobs:
test:
runs-on: ubuntu-latest
timeout-minutes: 30
services:
redis:
image: redis
ports: ["6379:6379"]
postgres:
ports: ["5432:5432"]
steps:
//omitted setups
- name: Setup Parallel Database
env:
RAILS_ENV: test
PGHOST: localhost
PGUSER: postgres
run: |
cp config/database.yml.example config/database.yml
bundle exec rake parallel:create
bundle exec rake parallel:rake[db:schema:load] || true
- name: Build and test with rspec
env:
RAILS_ENV: test
PGHOST: localhost
PGUSER: postgres
APP_REDIS_URL: redis://localhost:6379
MINIMUM_COVERAGE: 80
run: |
bundle exec parallel_rspec --verbose spec

Run deployment workflow if tests workflow is passed

I have the following two workflows:
Workflow to run test suite
Workflow to deploy the code, using https://github.com/miloserdow/capistrano-deploy
Now when I push my code, both workflows started. I want deployment Work to only start once Test Suite gets passed.
How can I do this?
Workflow that runs tests:
name: CI
on:
push:
branches: [setup_github]
jobs:
test:
runs-on: ubuntu-18.04
services:
postgres:
image: postgres:10
steps:
- name: Checkout
uses: actions/checkout#v1
- name: Set up Ruby
uses: ruby/setup-ruby#v1
with:
ruby-version: 2.5.3
- uses: borales/actions-yarn#v2.0.2
with:
cmd: install
- name: Install Dependencies
run: |
sudo apt-get -yqq install libpq-dev
- name: Install Gems
run: |
gem install bundler
- name: prepare Database
- name: RSpec
run: |
bundle exec rspec specs
Workflow that deploys:
name: Deploy on server
on:
push:
branches:
- setup_github
jobs:
deploy:
runs-on: ubuntu-18.04
steps:
- uses: actions/checkout#v1
- uses: ruby/setup-ruby#v1
with:
ruby-version: 2.5.3
bundler-cache: true
- uses: miloserdow/capistrano-deploy#master
with:
target: staging
deploy_key: ${{ secrets.DEPLOY_ENC_KEY }}
Your goal should be achievable by ensuring the following are true:
The CI workflow has run
The CI workflow was success
name: Deploy on server
on:
workflow_run:
workflows: [CI]
branches: [setup_github]
types:
- completed
jobs:
deploy:
if: ${{ github.event.workflow_run.conclusion == 'success' }}
runs-on: ubuntu-18.04
steps:
- uses: actions/checkout#v1
- uses: ruby/setup-ruby#v1
with:
ruby-version: 2.5.3
bundler-cache: true
- uses: miloserdow/capistrano-deploy#master
with:
target: staging
deploy_key: ${{ secrets.DEPLOY_ENC_KEY }}
This is described in the Github Actions docs on workflow_run.

Resources