Github Actions Application is being build twice - docker

I have a problem. I am starting to work with Github Actions, and I got a working pipeline where I am trying to:
Build the docker application
Run the tests
Publish to docker repository
But the test running and publishing are 2 separate jobs, so I am building the image twice. Here is the workflow code:
name: Test & Publish Docker Image
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
jobs:
run_application_tests:
name: Run test suite
runs-on: ubuntu-latest
env:
COMPOSE_FILE: docker-compose.yml
DOCKER_USER: ${{ secrets.DOCKER_USER }}
DOCKER_PASS: ${{ secrets.DOCKER_PASS }}
steps:
- name: Checkout code
uses: actions/checkout#v3
- name: Login To Docker Hub
uses: docker/login-action#v2
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Build and run docker image
run: docker-compose up -d --build
- name: Run migrations
run: make migrate
- name: Run tests
run: make test
build_and_publish_docker_image:
needs: run_application_tests
name: Build & Publish Docker Images
runs-on: ubuntu-latest
steps:
- name: Checkout The Repo
uses: actions/checkout#v3
- name: Login To Docker Hub
uses: docker/login-action#v2
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Build & Push Docker Image
uses: docker/build-push-action#v3
with:
push: true
tags: |
me/image:latest
me/image:${{ github.sha }}
The docker image is a Ruby-On-Rails application so the bundle install takes very long. Each build takes about 2-3 minutes.
I also tried adding:
cache-from: type=gha
cache-to: type=gha,mode=max
To the docker/build-push-action#v3, but that resulted in:
Error: buildx failed with: ERROR: cache export feature is currently not supported for docker driver. Please switch to a different driver (eg. "docker buildx create --use")
What can I change to improve this pipeline time based?

in your job run_application_tests you have this code:
- name: Login To Docker Hub
uses: docker/login-action#v2
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Build and run docker image
run: docker-compose up -d --build
this code runs docker-compose why?
this is how your code should be if you want to run tests only:
jobs:
run_application_tests:
name: Run test suite
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout#v3
- name: Run migrations
run: make migrate
- name: Run tests
run: make test

Related

How to implement semantic versioning in GitHub Actions workflow?

I would like to semantic versioning my docker images which are built and pushed to GitHub Container Registry by the GitHub Action.
I found a satisfying solution here: https://stackoverflow.com/a/69059228/12877180
According to the solution I reproduced the following YAML.
name: Docker CI
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
env:
REGISTRY: ghcr.io
jobs:
build-push:
# needs: build-test
name: Buid and push Docker image to GitHub Container registry
runs-on: ubuntu-latest
permissions:
packages: write
contents: read
steps:
- name: Checkout the repository
uses: actions/checkout#v2
- name: Login to GitHub Container registry
uses: docker/login-action#v1
env:
USERNAME: ${{ github.actor }}
PASSWORD: ${{ secrets.GITHUB_TOKEN }}
with:
registry: ${{ env.REGISTRY }}
username: ${{ env.USERNAME }}
password: ${{ env.PASSWORD }}
- name: Get lowercase repository name
run: |
echo "IMAGE=${REPOSITORY,,}">>${GITHUB_ENV}
env:
REPOSITORY: ${{ env.REGISTRY }}/${{ github.repository }}
- name: Build and export the image to Docker
uses: docker/build-push-action#v2
with:
context: .
file: ./docker/Dockerfile
target: final
push: true
tags: |
${{ env.IMAGE }}:${{ secrets.MAJOR }}.${{ secrets.MINOR }}
build-args: |
ENVIRONMENT=production
- name: Update Patch version
uses: hmanzur/actions-set-secret#v2.0.0
with:
name: 'MINOR'
value: $((${{ secrets.MINOR }} + 1))
repository: ${{ github.repository }}
token: ${{ secrets.GH_PAT }}
Unfortunately this does not work.
The initial value of the MINOR secret is 0. If the build-push job is executed very first time, the docker image is perfectly pushed to the GHCR with the ghcr.io/my-org/my-repo:0.0 syntax.
The purpose of the build-push job is then increment the MINOR secret by 1.
If the action job build-push is executed again after new event, I get error while trying to build docker image using the incremented tag.
/usr/bin/docker buildx build --build-arg ENVIRONMENT=production --tag ghcr.io/my-org/my-repo:***.*** --target final --iidfile /tmp/docker-build-push-HgjJR7/iidfile --metadata-file /tmp/docker-build-push-HgjJR7/metadata-file --file ./docker/Dockerfile --push .
error: invalid tag "ghcr.io/my-org/my-repo:***.***": invalid reference format
Error: buildx failed with: error: invalid tag "ghcr.io/my-org/my-repo:***.***": invalid reference format
You need to increment the version in a bash command like this:
- name: Autoincrement a new patch version
run: |
echo "NEW_PATCH_VERSION=$((${{ env.PATCH_VERSION }}+1))" >> $GITHUB_ENV
- name: Update patch version
uses: hmanzur/actions-set-secret#v2.0.0
with:
name: 'PATCH_VERSION'
value: ${{ env.NEW_PATCH_VERSION }}
repository: ${{ github.repository }}
token: ${{ secrets.REPO_ACCESS_TOKEN }}

GitHub Actions: How to use matrix value inside a step

I have a simple pipeline in GitHub Actions. I'm trying to build and publish a Docker image to Docker Hub within the same. I would also like to build the image for different platforms (read: operating systems), hence I'm using strategy.matrix. This is how it looks like so far:
name: Build and Publish Docker image(s)
jobs:
publish:
runs-on: ubuntu-latest
steps:
- name: Check out repository
uses: actions/checkout#v2
- name: Log in to Docker Hub
uses: docker/login-action#v1
with:
password: ${{ secrets.DOCKERHUB_TOKEN }}
registry: docker.io
username: ${{ secrets.DOCKERHUB_USERNAME }}
- name: Build and push
uses: docker/build-push-action#v2
env:
platform: ${{ matrix.platforms }}
with:
context: ./${platform}/
push: true
tags: oscarotero/lume:1.3.0-${platform} # TODO: Parameterize version once this is finally working
strategy:
matrix:
platforms:
- alpine
- debian
on:
push:
branches:
- main
This is currently failing because it doesn't recognize platform when running the Build and push step. I have a similar one and it works with this approach.
Anyone more knowledgeable in GitHub Actions can shred some light here? All I can think of is that docker/build-push-action#v2 doesn't support that, but in that case, how can I approach this problem?
Removing the environment variable and using the matrix values directly solved the problem:
jobs:
publish:
runs-on: ubuntu-latest
steps:
- name: Check out repository
uses: actions/checkout#v2
- name: Log in to Docker Hub
uses: docker/login-action#v1
with:
password: ${{ secrets.DOCKERHUB_TOKEN }}
registry: docker.io
username: ${{ secrets.DOCKERHUB_USERNAME }}
- name: Build and push
uses: docker/build-push-action#v2
with:
context: ./${{ matrix.platform }}/
push: true
tags: oscarotero/lume:1.3.0-${{ matrix.platform }}
strategy:
matrix:
platform:
- alpine
- debian

Docker build and push using github: invalid reference format

I want to build my image and push it to dockerhub using github actions.This is my repo
And this is how I want to do my job:
name: ci
on:
push:
branches:
- 'master'
jobs:
docker:
runs-on: ubuntu-latest
steps:
-
name: Checkout
uses: actions/checkout#v2
-
name: Set up QEMU
uses: docker/setup-qemu-action#v1
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action#v1
-
name: Login to DockerHub
uses: docker/login-action#v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PASSWORD }}
-
name: Build and push
uses: docker/build-push-action#v2
with:
context: .
load: true
tags: shirzadi/ehsan:latest
-
name: Push it!
run: docker push ${{ env.IMAGE }}:${{ env.GIT_SHA }}
And this is the result:
You are using environment variables that don't exist. According to docker/build-push-action#v2 documentation it can build, tag and push your image in a single step:
jobs:
docker:
steps:
# ...
-
name: Build and push
uses: docker/build-push-action#v2
with:
context: .
push: true
tags: shirzadi/ehsan:latest,shirzadi/ehsan:${{ env.GITHUB_SHA }}
The tags key lists 2 tags and use the GITHUB_SHA environment variable. These tags will be pushed as the push key is set to true.
See:
Github Actions - Default environment variables
docker/build-push-action#v2 - customizing inputs
You are not having env.IMAGE data. Below piece of code should do the job.
- name: Build and push
run: docker build -t ${{ env.IMAGE }}:${{ github.sha }} .
env:
IMAGE: shirzadi/ehsan
- name: Push it!
run: docker push ${{ env.IMAGE }}:${{ github.sha }}
env:
IMAGE: shirzadi/ehsan

Deploy to kubernetes cluster with github workflow

I have setup a kubernetes cluster on AWS using kops.
I am trying to automate deployment with github actions.
name: Build and Deploy
on:
push:
branches:
- develop
jobs:
build_docker_image:
- uses: actions/checkout#v2
- name: Build the tagged Docker image
run: docker build --target dev -t org/customer-service-backend:la
push_docker_image_to_github-packages:
- uses: docker/build-push-action#v2
with:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
registry: docker.pkg.github.com
repository: org/customer-service-backend:latest
tag_with_ref: true
deploy_to_kubernetes_cluster:
... what to do here?
I am able to built the image and push to gihub packages.
I have created deployment.yml in the root directory of the repository.
How can I deploy to kubernetes cluster?
Also, I am tagging the images with latest. Is it fine or I need to use GITHUB_REF for tagging?
Update
I am able to configure all the things. I only need to get kubeconfig to authenticate to existing cluster.
name: Build and Deploy
on:
push:
branches:
- develop
jobs:
build_docker_image:
- uses: actions/checkout#v2
- name: Build the tagged Docker image
run: docker build --target dev -t org/customer-service-backend:${{ github.sha }}
push_docker_image_to_github_packages:
needs: build_docker_image
- uses: docker/build-push-action#v2
with:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
registry: docker.pkg.github.com
repository: org/customer-service-backend:${{ github.sha }}
tag_with_ref: true
deploy_to_kubernetes_cluster:
needs: push_docker_image_to_github_packages
name: Set Kubernetes Context
uses: azure/k8s-set-context#v1
with:
method: kubeconfig
kubeconfig: ${{ secrets.KUBE_CONFIG }} # Use secret (https://developer.github.com/actions/managing-workflows/storing-secrets/)
run: |
sed -i'' -e 's/IMAGE_LABEL/${{ github.sha }}/g' deployment.yml
kubectl apply -f deployment.yml
By looking your workflow config file, all the jobs running parallelly.
But, probably it's not what you want.
Pushing image needs a built image and deployment job needs an updated built image.
On Access kubernetes cluster, just access into your cluster and do,
cat $HOME/.kube/config
and copy the output.
Now, create a secret in github with KUBE_CONFIG as environment variable.
Notes - this is one method to access kubernetes cluster, there are other methods as well, choose one that suits your need
name: Build and Deploy
on:
push:
branches:
- develop
jobs:
build_docker_image:
name: Build Docker Image
runs-on: ubuntu-latest
steps:
- name: Checkout Repo
uses: actions/checkout#v2
- name: Build the tagged Docker image
run: docker build --target dev -t your_org/customer-service-backend:${{ github.sha }} .
push_docker_image_to_github_packages:
name: Push Docker Image to Github Packages
needs: build_docker_image
runs-on: ubuntu-latest
steps:
- name: Push Docker Image
uses: docker/build-push-action#v2
with:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
registry: docker.pkg.github.com
repository: your_org/customer-service-backend:${{ github.sha }}
deploy_to_kubernetes_cluster:
name: Deploy to Kubernetes Cluster
needs: push_docker_image_to_github_packages
runs-on: ubuntu-latest
steps:
- name: Checkout Repo
uses: actions/checkout#v2
- name: Set Kubernetes Context
uses: azure/k8s-set-context#v1
with:
method: kubeconfig
kubeconfig: ${{ secrets.KUBE_CONFIG }} # Use secret (https://developer.github.com/actions/managing-workflows/storing-secrets/)
- name: Deploy to Cluster
run: |
sed -i'' -e 's/IMAGE_LABEL/${{ github.sha }}/g' deployment.yml
kubectl apply -f deployment.yml

Docker buildx Error : rpc error: code = Unknown desc = server message: insufficient_scope: authorization failed

I'm new to Docker and trying to perform CI using GitHub Actions.
Here's my .yml file on GitHub.
name: CI to Docker Hub
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Check Out Repo
uses: actions/checkout#v2
- name: Login to Docker Hub
uses: docker/login-action#v1
with:
username: ${{ secrets.DOCKER_HUB_USERNAME }}
password: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }}
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action#v1
- name: Build and push
id: docker_build
uses: docker/build-push-action#v2
with:
context: .
file: ./Dockerfile
push: true
tags: your-order-backend:latest
- name: Image digest
run: echo ${{ steps.docker_build.outputs.digest }}
I have added Secrets in my Github too of Docker Hub.
I'm not sure why, but it is failing at > exporting to image:
It is resloved!
In my case, there was no repo in the Docker Hub created whose tag I have passed here in the yml file.
I created the repo and it worked
for me I fixed it by changing "push: true" to "load: true"
In my case, Id needed to create the repo in dockerhub as in the response of Sagar, but the problem still were there. I'd recognized I was passing the tags incorrectly, fix it and the problem was gone. (You can See the official examples)
Before:
tags: ${{ github.sha }}, latest
After (correctly):
tags: |
henriqueholtz/fullcycle-gitops:${{ github.sha }}
henriqueholtz/fullcycle-gitops:latest

Resources