I would like to run my CI on a Docker image. How should I write my .github/workflow/main.yml?
name: CI
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
jobs:
build:
name: build
runs:
using: 'docker'
image: '.devcontainer/Dockerfile'
steps:
- uses: actions/checkout#v2
- name: Build
run: make
I get the error:
The workflow is not valid. .github/workflows/main.yml
(Line: 11, Col: 5): Unexpected value 'runs'
I managed to make it work but with an ugly workaround:
build:
name: Build Project
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout#v1
- name: Build docker images
run: >
docker build . -t foobar
-f .devcontainer/Dockerfile
- name: Build exam
run: >
docker run -v
$GITHUB_WORKSPACE:/srv
-w/srv foobar make
Side question: where can I find the documentation about this? All I found is how to write actions.
If you want to use a container to run your actions, you can use something like this:
jobs:
build:
runs-on: ubuntu-latest
container:
image: docker://{host}/{image}:{tag}
steps:
...
Here is an example.
If you want more details about the jobs.<job_id>.container and its sub-fields, you can check the official documentation.
Note that you can also use docker images at the step level: Example.
I am reposting my answer to another question, in order to be sure to find it while Googling it.
The best solution is to build, publish and re-use a Docker image based on your Dockerfile.
I would advise to create a custom build-and-publish-docker.yml action following the Github documentation: Publishing Docker images.
Assuming your repository is public, you should be able to automatically upload your image to ghcr.io without any required configuration. As an alternative, it's also possible to publish the image to Docker Hub.
Once your image is built and published (based on the on event of the action previously created, which can be triggered manually also), you just need to update your main.yml action so it uses the custom Docker image. Again, here is a pretty good documentation page about the container option: Running jobs in a container.
As an example, I'm sharing what I used in a personal repository:
Dockerfile: the Docker image to be built on CI
docker.yml: the action to build the Docker image
lint.yml: the action using the built Docker image
Related
I'm trying to understand the difference between 2 practices when using Github Runner: copying code from a Dockerfile and recreating the environment on the Runner, or using containers.
Imagine that we have a Dockerfile with some app:
FROM ubuntu
...
It's not important what the app is, the important part is that it runs on Ubuntu like our workflow.
Is it better use this Dockerfile to create an image and push it
to a Docker registry and then use it like this:
on: push
jobs:
first-job:
runs-on: ubuntu-latest
container: container:latest
steps:
- name: checks-out repo
uses: actions/checkout#v3
- name: Greetings
run: echo Hello World
Or, rewrite the workflow so you don't need an image, instead
recreate this environment by youself:
on: push
jobs:
first-job:
runs-on: ubuntu-latest
steps:
- name: recreate container
run:|
....
- name: checks-out repo
uses: actions/checkout#v3
- name: Greetings
run: echo Hello World
I can ask this question in another way. I need to setup
latex, but on github.com/action we don't have anything
to do so ( uses: actions/set-up-latex..). But I already have an image pushed to the registry https://hub.docker.com/r/blang/latex.
Is it preferred to use this image or to recreate the environment on
the runner (VM)?
I appreciate if you explain this concept explicitly with details.
I am using below config.yml file ( .circleci/config.yml ) to run the circle CI job for github and build and push docker image to repo:
orbs:
docker: circleci/docker#1.5.0
version: 2.1
executors:
docker-publisher:
environment:
IMAGE_NAME: johndocker/docker-node-app
docker: # Each job requires specifying an executor
# (either docker, macos, or machine), see
— image: circleci/golang:1.15.1
auth:
username: $DOCKERHUB_USERNAME
password: $DOCKERHUB_PASSWORD
jobs:
publishLatestToHub:
executor: docker-publisher
steps:
— checkout
— setup_remote_docker
— run
name: Publish Docker Image to Docker Hub
command: |
echo “$DOCKERHUB_PASSWORD” | docker login -u “$DOCKERHUB_USERNAME” — password-stdin
docker build -t $IMAGE_NAME .
docker push $IMAGE_NAME:latest
workflows:
version: 2
build-master:
jobs:
— publishLatestToHub
The config.yml is the magic that tells circleci what to do with our app, for this demo we want it to build a docker image.
In circleci *workflows* are simply orchestrators, they order how things should be done, *executors* defines or groups up task, *jobs* define the basic steps and commands to run.
But, it shows below error in Circle CI dashboard:
Unable to parse YAML, while scanning a simple key in 'string', line 21,
I checked using yml formatted also , but couldn't resolve the issue. Please help.
What I need is a way to build a Dockerfile within the repository as an image and use this as the image for the next step(s).
I've tried the Bitbucket Pipeline configuration below but in the "Build" step it doesn't seem to have the image (which was built in the previous step) in its cache.
pipelines:
branches:
main:
- step:
name: Docker Image(s)
script:
- docker build -t foo/bar .docker/composer
services:
- docker
caches:
- docker
- step:
name: Build
image: foo/bar
script:
- echo "Hello, World"
- composer --version
services:
- docker
caches:
- docker
I've tried the answer on the StackOverflow question below but the context in that question is pushing the image in the following step. It's not about using the image which was built for the step itself.
Bitbucket pipeline use locally built image from previous step
There's a few conceptual mistakes in your current pipeline. Let me first first run through those before giving you some possible solutions.
Clarifications
Caching
Bitbucket Pipelines uses the cache keyword to persist data across multiple pipelines. Whilst it will also persist across steps, the primary use-case is for the data to be used on separate builds. The cache takes 7 days to expire, and thus will not be updated with new data during those 7 days. You can manually delete the cache on the main Pipelines page. If you want to carry data across steps in the same pipelines, you should use the artifacts keyword.
Docker service
You should only need to use the docker service whenever you want to have a docker daemon available to your build. Most commonly whenever you need to use a docker command in your script. In your second step, you do not need this. So it doesn't need the docker service.
Solution 1 - Combine the steps
Combine the steps, and run composer within the created image by using the docker run command.
pipelines:
branches:
main:
- step:
name: Docker image and build
script:
- docker build -t foo/bar .docker/composer
# Replace <destination> with the working directory of the foo/bar image.
- docker run -v $BITBUCKET_CLONE_DIR:<destination> foo/bar composer --version
services:
- docker
Solution 2 - Using two steps with DockerHub
This example keeps the two step approach. In this scenario, you will push your foo/bar image to a public repository in Dockerhub. Pipelines will then pull it to use in the subsequent step.
pipelines:
branches:
main:
- step:
name: Docker Image(s)
script:
- docker build -t foo/bar .docker/composer
- docker login -u $DOCKERHUB_USER -p $DOCKERHUB_PASSWORD
- docker push foo/bar
services:
- docker
- step:
name: Build
image: foo/bar
script:
- echo "Hello, World. I'm running insider of the previously pushed foo/bar container"
- composer --version
If you'd like to use a private repository instead, you can replace the second step with:
...
- step:
name: Build
image:
name: foo/bar
username: $DOCKERHUB_USERNAME
password: $DOCKERHUB_PASSWORD
email $DOCKERHUB_EMAIL
script:
- echo "Hello, World. I'm running insider of the previously pushed foo/bar container"
- composer --version
To expand on phod's answer. If you really want two steps, you can transfer the image from one step to another.
pipelines:
branches:
main:
- step:
name: Docker Image(s)
script:
- docker build -t foo/bar .docker/composer
- docker image save foo/bar -o foobar.tar.gz
services:
- docker
caches:
- docker
artifacts:
- foobar.tar.gz
- step:
name: Build
script:
- docker image load -i foobar.tar.gz
- docker run -v $BITBUCKET_CLONE_DIR:<destination> foo/bar composer --version
services:
- docker
Note that this will upload all the layers and dependencies for the image. It can take quite a while to execute and may therefor not be the best solution.
I have this code at .github/workflows/main.yaml
# .github/workflows/main.yaml
name: CI Workflow
on: [push]
jobs:
rspec-job:
runs-on: ubuntu-latest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
container:
image: I-stucked-here
volumes:
- /vendor/bundle
steps:
- code omitted for brevity
The main idea of this job is to run all steps in container mode. Not in Linux host mode.
Under the same repository, I have a public Docker image named ruby-rimy-2.6.3. Since it's not publicly hosted on DockerHub, I can't find a way to programmatically authenticate myself to GitHub Packages/Registry.
I did try with different syntax (see code below) but it didn't work.
# .github/workflows/main.yaml
name: CI Workflow
on: [push]
jobs:
rspec-job:
runs-on: ubuntu-latest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
container:
image: docker://docker.pkg.github.com/zulhfreelancer/rimy/ruby-rimy-2.6.3:latest
volumes:
- /vendor/bundle
steps:
- code omitted for brevity
From the docs, GitHub says the GITHUB_TOKEN is available when the job is running. How do I use this GITHUB_TOKEN environment variable to run something like docker login on top of that
container: section so that the job is able to pull the image?
Using GitHub Personal Token is not an option for me because that repository is just my experiment repository before applying the same thing to my GitHub organization. I don't want to put my personal token under my organization's repository environment variables/secrets — that will simply exposes my personal token to my co-workers.
You do not need to use the container instruction to run tests in a container.
The GitHub Actions host comes with docker and docker-compose installed. The way I do it, is have a docker-compose.yml in my repository, which includes a "service" that runs tests. Then, your workflow needs to do docker login and simply run the docker-compose run test command.
Note that the beauty of this approach, is that your tests are executed exactly the same on your own machine and on the CI machine. Same exact steps.
Something along these lines:
name: Test
on:
pull_request:
push: { branches: master }
jobs:
test:
name: Run test suite
runs-on: ubuntu-latest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- name: Checkout code
uses: actions/checkout#v2
- name: Docker login
run: echo ${GITHUB_TOKEN} | docker login -u ${GITHUB_ACTOR} --password-stdin docker.pkg.github.com
- name: Build docker images
run: docker-compose build
- name: Run tests
run: docker-compose run test
I am doing the same with DockerHub, with great ease and success.
Of course, if you do not want to use docker-compose, you can still use any normal docker run ... commands after you login properly in the login step.
I am not sure that docker login command will work as is, see these for a deeper discussion:
https://github.com/actions/starter-workflows/issues/66
https://github.community/t5/GitHub-Actions/Github-Actions-Docker-login/td-p/29852/page/2
Is it possible to pass docker images built in earlier job in circle ci
example
jobs:
build:
steps:
- checkout
// build image
deploy:
steps:
- deploy earlier image
i cant see how i can access the image without rebuilding it
Each job can run on a different host, so to share the image you would need to push it to a registry from the job that builds it.
To reference the same job that was pushed you'll need an identifier that is known ahead of time. A good example of this is the CIRCLE_SHA1 environment variable. You can use this variable as the image tag
jobs:
build:
machine: true
steps:
...
- run: |
docker build -t repo/app:$CIRCLE_SHA1 .
docker push repo/app:$CIRCLE_SHA1
test:
docker:
- image: repo/app:$CIRCLE_SHA1
steps:
...
I believe you can achieve this by persisting the image to a workspace and then attaching the workspace when you want to deploy it. See CircleCI's workspace documentation here: https://circleci.com/docs/workspaces