I have a complex transformation that I need to apply whenever a particular file is pushed to GitHub. The transfomation is written in Kotlin (Java) and containerized using Jib. That all works ok.
The problem is I don't know how to run the containerized java app from within a GitHub action. The GitHub action is defined as
# This is a workflow that transforms a data file into a json file
name: file-transform
# Controls when the workflow will run
on:
workflow_dispatch:
jobs:
container-test-job:
runs-on: ubuntu-latest
container:
image: docker.io/apigeneration/github-action-test
username: ${{ github.actor }}
password: ${{ secrets.github_token }}
volumes:
- /config:/config
- /data:/data
steps:
- name: Run docker application
run: ???
I have tried all the options I can think of for the run step but the action fails.
Part of the problem is that I'm not clear how Jib defines the app entry point and so how to define a java command as part of the run step (I've tried all the options I can think of based on the Jib documentation).
Just running the docker container automatically runs the java app so perhaps there's a better way to invoke it in the action though the container is is a private registry so I have to be able to pass in credentials.
Any help gratefully received.
Related
I'm trying to understand the difference between 2 practices when using Github Runner: copying code from a Dockerfile and recreating the environment on the Runner, or using containers.
Imagine that we have a Dockerfile with some app:
FROM ubuntu
...
It's not important what the app is, the important part is that it runs on Ubuntu like our workflow.
Is it better use this Dockerfile to create an image and push it
to a Docker registry and then use it like this:
on: push
jobs:
first-job:
runs-on: ubuntu-latest
container: container:latest
steps:
- name: checks-out repo
uses: actions/checkout#v3
- name: Greetings
run: echo Hello World
Or, rewrite the workflow so you don't need an image, instead
recreate this environment by youself:
on: push
jobs:
first-job:
runs-on: ubuntu-latest
steps:
- name: recreate container
run:|
....
- name: checks-out repo
uses: actions/checkout#v3
- name: Greetings
run: echo Hello World
I can ask this question in another way. I need to setup
latex, but on github.com/action we don't have anything
to do so ( uses: actions/set-up-latex..). But I already have an image pushed to the registry https://hub.docker.com/r/blang/latex.
Is it preferred to use this image or to recreate the environment on
the runner (VM)?
I appreciate if you explain this concept explicitly with details.
I would like to run my CI on a Docker image. How should I write my .github/workflow/main.yml?
name: CI
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
jobs:
build:
name: build
runs:
using: 'docker'
image: '.devcontainer/Dockerfile'
steps:
- uses: actions/checkout#v2
- name: Build
run: make
I get the error:
The workflow is not valid. .github/workflows/main.yml
(Line: 11, Col: 5): Unexpected value 'runs'
I managed to make it work but with an ugly workaround:
build:
name: Build Project
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout#v1
- name: Build docker images
run: >
docker build . -t foobar
-f .devcontainer/Dockerfile
- name: Build exam
run: >
docker run -v
$GITHUB_WORKSPACE:/srv
-w/srv foobar make
Side question: where can I find the documentation about this? All I found is how to write actions.
If you want to use a container to run your actions, you can use something like this:
jobs:
build:
runs-on: ubuntu-latest
container:
image: docker://{host}/{image}:{tag}
steps:
...
Here is an example.
If you want more details about the jobs.<job_id>.container and its sub-fields, you can check the official documentation.
Note that you can also use docker images at the step level: Example.
I am reposting my answer to another question, in order to be sure to find it while Googling it.
The best solution is to build, publish and re-use a Docker image based on your Dockerfile.
I would advise to create a custom build-and-publish-docker.yml action following the Github documentation: Publishing Docker images.
Assuming your repository is public, you should be able to automatically upload your image to ghcr.io without any required configuration. As an alternative, it's also possible to publish the image to Docker Hub.
Once your image is built and published (based on the on event of the action previously created, which can be triggered manually also), you just need to update your main.yml action so it uses the custom Docker image. Again, here is a pretty good documentation page about the container option: Running jobs in a container.
As an example, I'm sharing what I used in a personal repository:
Dockerfile: the Docker image to be built on CI
docker.yml: the action to build the Docker image
lint.yml: the action using the built Docker image
I wanted to run django test cases inside container.
I am able to pull private image from docker hub. but when I ran command to test, It is failed to run.
Anyone tried running test cases inside the container.
jobs:
test:
container:
image: abcd
credentials:
username: "<username>"
password: "<password>"
steps:
- uses: actions/checkout#v2
- name: Display Python version
run: |
python -m pip install --upgrade pip
pip install -r requirements/dev.txt
- name: run test
run: |
python3 manage.py test
In my experience, I found out that using GitHub's container instruction causes more confusion than simply running whatever you want on the runner itself, as if you are running it on your own machine.
A big majority of the tests I am running on GitHub actions are running in containers, and some require private DockerHub images.
I always do this:
Create a docker-compose.yml for development use, so I can test things locally.
Usually in CI, you might want slightly different things in your docker-compose (for example, no volume mappings) - if this is the case, I am creating another docker-compose.yml in a .ci subfolder.
My docker-compose.yml contains a test service, that runs whatever test (or test suite) I want.
Here is a sample GitHub actions file I am using:
name: Test
on:
pull_request:
push: { branches: master }
jobs:
test:
name: Run test suite
runs-on: ubuntu-latest
env:
COMPOSE_FILE: .ci/docker-compose.yml
DOCKER_USER: ${{ secrets.DOCKER_USER }}
DOCKER_PASS: ${{ secrets.DOCKER_PASS }}
steps:
- name: Checkout code
uses: actions/checkout#v2
- name: Login to DockerHub
run: docker login -u $DOCKER_USER -p $DOCKER_PASS
- name: Build docker images
run: docker-compose build
- name: Run tests
run: docker-compose run test
Of course, this entails setting up the two mentioned secrets, but other than that, I found this method to be:
Reliable
Portable (I switched from Travis CI with the same approach easily)
Compatible with dev environment
Easy to understand and reproduce both locally and in CI
I have this code at .github/workflows/main.yaml
# .github/workflows/main.yaml
name: CI Workflow
on: [push]
jobs:
rspec-job:
runs-on: ubuntu-latest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
container:
image: I-stucked-here
volumes:
- /vendor/bundle
steps:
- code omitted for brevity
The main idea of this job is to run all steps in container mode. Not in Linux host mode.
Under the same repository, I have a public Docker image named ruby-rimy-2.6.3. Since it's not publicly hosted on DockerHub, I can't find a way to programmatically authenticate myself to GitHub Packages/Registry.
I did try with different syntax (see code below) but it didn't work.
# .github/workflows/main.yaml
name: CI Workflow
on: [push]
jobs:
rspec-job:
runs-on: ubuntu-latest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
container:
image: docker://docker.pkg.github.com/zulhfreelancer/rimy/ruby-rimy-2.6.3:latest
volumes:
- /vendor/bundle
steps:
- code omitted for brevity
From the docs, GitHub says the GITHUB_TOKEN is available when the job is running. How do I use this GITHUB_TOKEN environment variable to run something like docker login on top of that
container: section so that the job is able to pull the image?
Using GitHub Personal Token is not an option for me because that repository is just my experiment repository before applying the same thing to my GitHub organization. I don't want to put my personal token under my organization's repository environment variables/secrets — that will simply exposes my personal token to my co-workers.
You do not need to use the container instruction to run tests in a container.
The GitHub Actions host comes with docker and docker-compose installed. The way I do it, is have a docker-compose.yml in my repository, which includes a "service" that runs tests. Then, your workflow needs to do docker login and simply run the docker-compose run test command.
Note that the beauty of this approach, is that your tests are executed exactly the same on your own machine and on the CI machine. Same exact steps.
Something along these lines:
name: Test
on:
pull_request:
push: { branches: master }
jobs:
test:
name: Run test suite
runs-on: ubuntu-latest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- name: Checkout code
uses: actions/checkout#v2
- name: Docker login
run: echo ${GITHUB_TOKEN} | docker login -u ${GITHUB_ACTOR} --password-stdin docker.pkg.github.com
- name: Build docker images
run: docker-compose build
- name: Run tests
run: docker-compose run test
I am doing the same with DockerHub, with great ease and success.
Of course, if you do not want to use docker-compose, you can still use any normal docker run ... commands after you login properly in the login step.
I am not sure that docker login command will work as is, see these for a deeper discussion:
https://github.com/actions/starter-workflows/issues/66
https://github.community/t5/GitHub-Actions/Github-Actions-Docker-login/td-p/29852/page/2
I am currently trying to build and deploy a dockerized Go project, pulled from a Git repo in using Concourse.
To give you some background about my current setup:
I got two AWS Lightsail instances set up, both of them using a Docker container to serve Concourse.
One of those instances is serving the web node, the other one is acting as a worker node, which connects to the web node.
My current pipeline looks like this:
resources:
- name: zsu-wasserlabor-api-repo
type: git
webhook_token: TOP_SECRET
source:
uri: git#github.com:lennartschoch/zsu-wasserlabor-api
branch: master
private_key: TOP_SECRET
jobs:
- name: build-api
plan:
- get: zsu-wasserlabor-api-repo
trigger: true
- task: build
config:
platform: linux
image_resource:
type: docker-image
source: {repository: alpine}
inputs:
- name: zsu-wasserlabor-api-repo
run:
path: sh
args:
- -c
- |
cd zsu-wasserlabor-api-repo
docker-compose build
The problem is that docker-compose is not installed.
I am feeling like I am doing something fundamentally wrong. Could anyone give me a hint?
Best,
Lennart
The pipeline described above specifies that it should use the alpine image, which doesn't have docker-compose on it. Thus, you will need to find an image that has docker-compose installed on it, but even then, there are additional steps you will need to take to make it work in Concourse (see this link for more details).
Fortunately, someone has made an image available that takes care of the additional steps, with a sample pipeline that you can find here: https://github.com/meAmidos/dcind
That being said, if you are simply trying to build a Docker image, you can use the docker-image-resource instead and just specify the Dockerfile.