I wanted to run django test cases inside container.
I am able to pull private image from docker hub. but when I ran command to test, It is failed to run.
Anyone tried running test cases inside the container.
jobs:
test:
container:
image: abcd
credentials:
username: "<username>"
password: "<password>"
steps:
- uses: actions/checkout#v2
- name: Display Python version
run: |
python -m pip install --upgrade pip
pip install -r requirements/dev.txt
- name: run test
run: |
python3 manage.py test
In my experience, I found out that using GitHub's container instruction causes more confusion than simply running whatever you want on the runner itself, as if you are running it on your own machine.
A big majority of the tests I am running on GitHub actions are running in containers, and some require private DockerHub images.
I always do this:
Create a docker-compose.yml for development use, so I can test things locally.
Usually in CI, you might want slightly different things in your docker-compose (for example, no volume mappings) - if this is the case, I am creating another docker-compose.yml in a .ci subfolder.
My docker-compose.yml contains a test service, that runs whatever test (or test suite) I want.
Here is a sample GitHub actions file I am using:
name: Test
on:
pull_request:
push: { branches: master }
jobs:
test:
name: Run test suite
runs-on: ubuntu-latest
env:
COMPOSE_FILE: .ci/docker-compose.yml
DOCKER_USER: ${{ secrets.DOCKER_USER }}
DOCKER_PASS: ${{ secrets.DOCKER_PASS }}
steps:
- name: Checkout code
uses: actions/checkout#v2
- name: Login to DockerHub
run: docker login -u $DOCKER_USER -p $DOCKER_PASS
- name: Build docker images
run: docker-compose build
- name: Run tests
run: docker-compose run test
Of course, this entails setting up the two mentioned secrets, but other than that, I found this method to be:
Reliable
Portable (I switched from Travis CI with the same approach easily)
Compatible with dev environment
Easy to understand and reproduce both locally and in CI
Related
I've developed a set of tests, which to run locally, all I need to do is to launch an Elasticsearch docker container, by going to the folder with the docker-compose.yml file, and on the bash do docker compose up -d. Then, I run the script with the tests.
I want to create a specific github workflow with these tests.
How would one go about it? I'm a complete novice when it comes to github.
Any help would be appreciated.
Here's my starting point(I haven't run it, because it's very likely not to run without errors)
name: local_db_testing
run-name: ${{ github.actor }} is testing on local ES db
on:
pull_request:
branches: [main, dev,ci_testing_branch]
types: [opened, reopened]
push:
paths-ignore:
- '**.md'
- './docs'
- 'Dockerfile'
jobs:
run_tests:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.8]
steps:
- name: Setup Python
uses: actions/setup-python#v4
with:
python-version: ${{ matrix.python-version }}
- name: Checkout branch
uses: actions/checkout#v3
- name: Install requirements
run: |
pip install -r requirements.txt
- name: Launching ES db container
run: |
cd ./elasticsearch_local_db
run docker compose up -d
- name: Execute tests
run: |
cd ./tests/local_ES_tests
python -m pytest test_set.py
I am using below config.yml file ( .circleci/config.yml ) to run the circle CI job for github and build and push docker image to repo:
orbs:
docker: circleci/docker#1.5.0
version: 2.1
executors:
docker-publisher:
environment:
IMAGE_NAME: johndocker/docker-node-app
docker: # Each job requires specifying an executor
# (either docker, macos, or machine), see
— image: circleci/golang:1.15.1
auth:
username: $DOCKERHUB_USERNAME
password: $DOCKERHUB_PASSWORD
jobs:
publishLatestToHub:
executor: docker-publisher
steps:
— checkout
— setup_remote_docker
— run
name: Publish Docker Image to Docker Hub
command: |
echo “$DOCKERHUB_PASSWORD” | docker login -u “$DOCKERHUB_USERNAME” — password-stdin
docker build -t $IMAGE_NAME .
docker push $IMAGE_NAME:latest
workflows:
version: 2
build-master:
jobs:
— publishLatestToHub
The config.yml is the magic that tells circleci what to do with our app, for this demo we want it to build a docker image.
In circleci *workflows* are simply orchestrators, they order how things should be done, *executors* defines or groups up task, *jobs* define the basic steps and commands to run.
But, it shows below error in Circle CI dashboard:
Unable to parse YAML, while scanning a simple key in 'string', line 21,
I checked using yml formatted also , but couldn't resolve the issue. Please help.
I have a GitHub workflow with job like this:
docker:
name: Docker image
needs: test-n-build
runs-on: ubuntu-20.04
steps:
- name: Checkout
uses: actions/checkout#v3
- name: Set up QEMU
uses: docker/setup-qemu-action#v1
- name: Set up Docker Buildx
uses: docker/setup-buildx-action#v1
- name: Set docker image tag
if: startsWith(github.ref, 'refs/tags/')
run: echo "DOCKER_TAG=:$(cargo read-manifest | sed 's/.*"version":"\{0,1\}\([^,"]*\)"\{0,1\}.*/\1/')" >> $GITHUB_ENV
- name: Login to DockerHub
uses: docker/login-action#v1
- name: Build and push
uses: docker/build-push-action#v2
with:
push: ${{ startsWith(github.ref, 'refs/tags/') }}
This workflow builds docker image and pushes it to registry if tag pushed to branch. As seen there are a Set docker image tag step with cargo command used, but when I'm copy-pasted things I forgot to add setup rust action. But that step successfully executed and no error like command cargo not found not appears.
Is it because that job needs test-n-build job where I actually setup Rust, or QEMU installs Rust? How it finds cargo command?
As you can see on the Ubuntu-20.04 virtual environment, it is provided with some Rust Tools installed:
### Rust Tools
Cargo 1.58.0
Rust 1.58.1
Rustdoc 1.58.1
Rustup 1.24.3
Therefore, you wouldn't need any extra setup to install them in that case.
You can check the available virtual environment used as runners and their configurations here.
I would like to run my CI on a Docker image. How should I write my .github/workflow/main.yml?
name: CI
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
jobs:
build:
name: build
runs:
using: 'docker'
image: '.devcontainer/Dockerfile'
steps:
- uses: actions/checkout#v2
- name: Build
run: make
I get the error:
The workflow is not valid. .github/workflows/main.yml
(Line: 11, Col: 5): Unexpected value 'runs'
I managed to make it work but with an ugly workaround:
build:
name: Build Project
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout#v1
- name: Build docker images
run: >
docker build . -t foobar
-f .devcontainer/Dockerfile
- name: Build exam
run: >
docker run -v
$GITHUB_WORKSPACE:/srv
-w/srv foobar make
Side question: where can I find the documentation about this? All I found is how to write actions.
If you want to use a container to run your actions, you can use something like this:
jobs:
build:
runs-on: ubuntu-latest
container:
image: docker://{host}/{image}:{tag}
steps:
...
Here is an example.
If you want more details about the jobs.<job_id>.container and its sub-fields, you can check the official documentation.
Note that you can also use docker images at the step level: Example.
I am reposting my answer to another question, in order to be sure to find it while Googling it.
The best solution is to build, publish and re-use a Docker image based on your Dockerfile.
I would advise to create a custom build-and-publish-docker.yml action following the Github documentation: Publishing Docker images.
Assuming your repository is public, you should be able to automatically upload your image to ghcr.io without any required configuration. As an alternative, it's also possible to publish the image to Docker Hub.
Once your image is built and published (based on the on event of the action previously created, which can be triggered manually also), you just need to update your main.yml action so it uses the custom Docker image. Again, here is a pretty good documentation page about the container option: Running jobs in a container.
As an example, I'm sharing what I used in a personal repository:
Dockerfile: the Docker image to be built on CI
docker.yml: the action to build the Docker image
lint.yml: the action using the built Docker image
I have this code at .github/workflows/main.yaml
# .github/workflows/main.yaml
name: CI Workflow
on: [push]
jobs:
rspec-job:
runs-on: ubuntu-latest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
container:
image: I-stucked-here
volumes:
- /vendor/bundle
steps:
- code omitted for brevity
The main idea of this job is to run all steps in container mode. Not in Linux host mode.
Under the same repository, I have a public Docker image named ruby-rimy-2.6.3. Since it's not publicly hosted on DockerHub, I can't find a way to programmatically authenticate myself to GitHub Packages/Registry.
I did try with different syntax (see code below) but it didn't work.
# .github/workflows/main.yaml
name: CI Workflow
on: [push]
jobs:
rspec-job:
runs-on: ubuntu-latest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
container:
image: docker://docker.pkg.github.com/zulhfreelancer/rimy/ruby-rimy-2.6.3:latest
volumes:
- /vendor/bundle
steps:
- code omitted for brevity
From the docs, GitHub says the GITHUB_TOKEN is available when the job is running. How do I use this GITHUB_TOKEN environment variable to run something like docker login on top of that
container: section so that the job is able to pull the image?
Using GitHub Personal Token is not an option for me because that repository is just my experiment repository before applying the same thing to my GitHub organization. I don't want to put my personal token under my organization's repository environment variables/secrets — that will simply exposes my personal token to my co-workers.
You do not need to use the container instruction to run tests in a container.
The GitHub Actions host comes with docker and docker-compose installed. The way I do it, is have a docker-compose.yml in my repository, which includes a "service" that runs tests. Then, your workflow needs to do docker login and simply run the docker-compose run test command.
Note that the beauty of this approach, is that your tests are executed exactly the same on your own machine and on the CI machine. Same exact steps.
Something along these lines:
name: Test
on:
pull_request:
push: { branches: master }
jobs:
test:
name: Run test suite
runs-on: ubuntu-latest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- name: Checkout code
uses: actions/checkout#v2
- name: Docker login
run: echo ${GITHUB_TOKEN} | docker login -u ${GITHUB_ACTOR} --password-stdin docker.pkg.github.com
- name: Build docker images
run: docker-compose build
- name: Run tests
run: docker-compose run test
I am doing the same with DockerHub, with great ease and success.
Of course, if you do not want to use docker-compose, you can still use any normal docker run ... commands after you login properly in the login step.
I am not sure that docker login command will work as is, see these for a deeper discussion:
https://github.com/actions/starter-workflows/issues/66
https://github.community/t5/GitHub-Actions/Github-Actions-Docker-login/td-p/29852/page/2