in my actions I am using the gha export type like:
- uses: docker/setup-buildx-action#v1
- name: Build & Push
uses: docker/build-push-action#v2
with:
cache-from: type=gha,scope=${{ matrix.ecr_repo }}
cache-to: type=gha,mode=max,scope=${{ matrix.ecr_repo }}
push: true
tags: "${{ env.tags }}"
However when the action is building I sometimes have lines that should be cached suddenly invalidated.
e.g:
#20 [base 22/26] COPY src/go.sum ./
#20 CACHED
#30 [base 23/26] RUN go mod download
#30 sha256:fccecca06c0e928afeb80a372e92178cfe09778a2d41c53d690328cd9350920f 43.01kB / 43.01kB 0.1s done
#30 sha256:46e25f3fb1d9354adea3ee0ec11523a2d4bdb24c795548575f2a8a180f8ef285 2.53kB / 2.53kB 0.1s done
...
Dockerfile:
...
COPY src/go.mod ./
COPY src/go.sum ./
RUN go mod download
COPY src ./
RUN --mount=type=cache,target=/root/.cache/go-build \
go build -tags=static -o /cli && ls -al /cli
...
I don't understand why RUN go mod download is not cached when the COPY before is - is there any debugging I can do?
Related
I have following Dockerfile:
FROM openjdk:17
RUN mkdir -p /ctk
RUN mkdir -p /ctk/config/tnt/v2/
WORKDIR /ctk
COPY suitexmls/ /ctk/suitexmls/
COPY config/tnt/v2/EventSubscription.json /ctk/config/tnt/v2/
COPY .mvn/ .mvn
COPY mvnw pom.xml ./
RUN chmod +x ./mvnw
RUN ./mvnw dependency:go-offline
COPY src ./src
RUN ./mvnw clean package spring-boot:repackage
CMD ["./mvnw", "spring-boot:run"]
This docker build and excution works fine locally.
But it fails in github action workflow. It gives following error:-
#5 [ 2/11] RUN mkdir -p /ctk
#5 DONE 0.3s
#6 [ 3/11] RUN mkdir -p /ctk/config/tnt/v2/
#6 DONE 0.3s
#4 [ 1/11] FROM docker.io/library/openjdk:17#sha256:528707081fdb9562eb819128a9f85ae7fe000e2fbaeaf9f87662e7b3f38cb7d8
#4 ...
#7 [ 4/11] WORKDIR /ctk
#7 DONE 0.0s
#8 [ 5/11] COPY suitexmls/ /ctk/suitexmls/
#8 DONE 0.0s
#9 [ 6/11] COPY config/tnt/v2/EventSubscription.json /ctk/config/tnt/v2/
#9 DONE 0.0s
#10 [ 7/11] COPY .mvn/ .mvn
#10 DONE 0.0s
#11 [ 8/11] COPY mvnw pom.xml ./
#11 DONE 0.0s
#12 [ 9/11] RUN ./mvnw dependency:go-offline
#12 0.189 /bin/sh: ./mvnw: Permission denied
#12 ERROR: executor failed running [/bin/sh -c ./mvnw dependency:go-offline]: exit code: 126
#4 [ 1/11] FROM docker.io/library/openjdk:17#sha256:528707081fdb9562eb819128a9f85ae7fe000e2fbaeaf9f87662e7b3f38cb7d8
------
> [ 9/11] RUN ./mvnw dependency:go-offline:
#12 0.189 /bin/sh: ./mvnw: Permission denied
------
ERROR: failed to solve: executor failed running [/bin/sh -c ./mvnw dependency:go-offline]: exit code: 126
Error: buildx failed with: ERROR: failed to solve: executor failed running [/bin/sh -c ./mvnw dependency:go-offline]: exit code: 126
How to fix this error? I have to add more text to post this quesion.
Updated quesion:
docker-publish.yml code:-
name: Docker
on:
push:
branches: [ master ]
# Publish semver tags as releases.
tags: [ 'v*.*.*' ]
pull_request:
branches: [ master ]
jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout#v3
- name: Set up JDK 17
uses: actions/setup-java#v3
with:
java-version: 17
distribution: 'temurin'
- name: Cache local Maven repository
uses: actions/cache#v2
with:
path: ~/.m2/repository
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-
- name: build Toolkit
run: echo ${{ secrets.DCSA_USER }} && mvn -B package -DskipTests -X
- name: Extract Build tag
id: buildtag
run: echo "TAG=$(mvn org.apache.maven.plugins:maven-help-plugin:3.2.0:evaluate -Dexpression=project.version -q -DforceStdout)">> $GITHUB_ENV
- name: Login to Docker Hub
uses: docker/login-action#v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PASSWORD }}
# Build and push Docker image
- name: Build and push
uses: docker/build-push-action#v3
with:
push: true
tags: user/app:latest
Now Docker build is fine with suggested fix. But push is fails. Error:-
#17 pushing user/app:latest with docker
#17 pushing layer 5a9003ca1b2d
#17 pushing layer 5a9003ca1b2d 0.2s done
#17 pushing layer a22a34274397 0.2s done
#17 pushing layer c8dd97366670 0.3s done
#17 ERROR: denied: requested access to the resource is denied
------
> pushing user/app:latest with docker:
------
ERROR: denied: requested access to the resource is denied
Error: buildx failed with: ERROR: denied: requested access to the resource is denied
I setup dockerhub secret correclty. Still I can't access why? I have to add more text to post my updated question. It doesn't allow me to post it. I setup dockerhub secret correclty. Still I can't access why?
I have to add more text to post my updated question. It doesn't allow me to post it. I setup dockerhub secret correclty. Still I can't access why?
I have to add more text to post my updated question. It doesn't allow me to post it. Here it the place where I added secret:-
For your dockerfile add RUN chmod +x ./mvnw after copy mvnw to image as follows. This command give you the execution permission.
FROM openjdk:17
RUN mkdir -p /ctk
RUN mkdir -p /ctk/config/tnt/v2/
WORKDIR /ctk
COPY suitexmls/ /ctk/suitexmls/
COPY config/tnt/v2/EventSubscription.json /ctk/config/tnt/v2/
COPY .mvn/ .mvn
COPY mvnw pom.xml ./
RUN chmod +x ./mvnw
RUN ./mvnw dependency:go-offline
COPY src ./src
RUN ./mvnw clean package spring-boot:repackage
CMD ["./mvnw", "spring-boot:run"]
The docker image tag you are using cause the error. Because to push to the docker hub the tag must contain your name as follows.
tags: <your_user_name>/app:latest
I am working on a GitLab CI/CD project to build a NET6 application into a docker image, which is pushed to AWS ECR. My issue is manifested in the continuous failure of 'COPY . .' or any other command after 'restore' of my targeted project.
Repo structure:
repo root
- projectFolder
- projectAPI
- projectAPI.csproj
- DockerFile
- nuget.config
- referencedClassLibrary1
- referencedClassLibrary2
- .sln
- gitlab-ci.yml
- package.json
Dockerfile:
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build-env
EXPOSE 80
WORKDIR /app
ARG Nuget_CustomFeedUserName
ARG Nuget_CustomFeedPassword
COPY "TT.CS/TT.CS.API/nuget.config" ""
COPY "TT.CS/TT.CS.DAL/TT.CS.DAL.csproj" "TT.CS.DAL/"
COPY "TT.CS/TT.CS.BLL/TT.CS.BLL.csproj" "TT.CS.BLL/"
COPY "TT.CS/TT.CS.API/TT.CS.ConfigAPI.csproj" "TT.CS.API/"
RUN ls -al
RUN sed -i "s|</configuration>|<packageSourceCredentials><aws><add key=\"Username\" value=\"${Nuget_CustomFeedUserName}\" /><add key=\"ClearTextPassword\" value=\"${Nuget_CustomFeedPassword}\" /></aws></packageSourceCredentials></configuration>|" /app/nuget.config
RUN dotnet restore --configfile "/app/nuget.config" "TT.CS.API/TT.CS.ConfigAPI.csproj"
COPY . .
WORKDIR "/app/TT.CS.API"
RUN dotnet publish --no-restore -c Release -o /app/out "TT.CS.ConfigAPI.csproj"
FROM mcr.microsoft.com/dotnet/aspnet:6.0
WORKDIR /app
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "TT.CS.ConfigAPI.dll"]
Outcome:
#17 3.891 Restored /app/TT.CS.BLL/TT.CS.BLL.csproj (in 1.58 sec).
#17 3.891 Restored /app/TT.CS.DAL/TT.CS.DAL.csproj (in 1.58 sec).
#17 3.941 Restored /app/TT.CS.API/TT.CS.ConfigAPI.csproj (in 1.67 sec).
#17 DONE 4.1s
#18 [build-env 11/14] COPY . .
#18 sha256:23c9d357832b3a7c1a8c5d607316f97c2fa09b51818f473a97f709f9cfeb981f
#18 ERROR: failed to prepare i2owcagafc3j7shg0slffonae: invalid argument
------
> [build-env 11/14] COPY . .:
------
failed to prepare i2owcagafc3j7shg0slffonae: invalid argument
I've created a solution with a single folder sln/csproj without referencing the same solution projects, which works as expected but I'm still having issues with more complex solutions that combine multiple referenced projects with public and private nuget repos.
Does anyone see some obvious mistake? Thanks
So I'm having an issue with my docker build action with github actions and I definitely feel like the issue is obvious but I am failing to get it.
So here is my Github Action
name: Build / Publish
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
build-and-push-docker-image:
name: Build Pinsel Docker image and push to repos
runs-on: ubuntu-latest
steps:
-
name: Checkout codebase
uses: actions/checkout#v2
-
name: Docker meta
id: meta
uses: docker/metadata-action#v3
with:
images: name/app
tags: |
type=ref,event=branch
type=ref,event=pr
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
# Setup docker build
-
name: Set up Docker buildx
id: buildx
uses: docker/setup-buildx-action#v1
-
name: Login to Docker Hub
uses: docker/login-action#v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
-
name: Login into Github Packages
uses: docker/login-action#v1
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GHCR_PAT }}
-
name: Build image and push it to both registries
uses: docker/build-push-action#v2
with:
context: ../../
file: ./Dockerfile.prod
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
And here is my Dockerfile located in main directory
FROM node:16.4.2-alpine as build
RUN npm install -g pnpm#next-7
WORKDIR /app
ENV PATH /app/node_modules/.bin:$PATH
COPY /pnpm-lock.yaml ./
RUN pnpm fetch --prod
ADD . ./
RUN pnpm install -r --offline --prod
FROM nginx:1.21-alpine
COPY ./nginx/default.conf /etc/nginx/conf.d/default.conf
COPY --from=build /app/build /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]
and lastly the error
#11 [build 5/8] COPY /package.json ./
#11 ERROR: failed to calculate checksum of ref ypf6stpm3t9h2xl6ezvwo7876::kvjk90tcw8wfykmdkvu1uvzgs: "/package.json": not found
#12 [stage-1 2/3] COPY ./nginx/default.conf /etc/nginx/conf.d/default.conf
#12 CANCELED
Dockerfile.prod:13
--------------------
11 |
12 | FROM nginx:1.21-alpine
13 | >>> COPY ./nginx/default.conf /etc/nginx/conf.d/default.conf
14 | COPY --from=build /app/build /usr/share/nginx/html
15 | EXPOSE 80
--------------------
error: failed to solve: failed to compute cache key: failed to calculate checksum of ref ypf6stpm3t9h2xl6ezvwo7876::kvjk90tcw8wfykmdkvu1uvzgs: "/nginx/default.conf": not found
Error: buildx failed with: error: failed to solve: failed to compute cache key: failed to calculate checksum of ref ypf6stpm3t9h2xl6ezvwo7876::kvjk90tcw8wfykmdkvu1uvzgs: "/nginx/default.conf": not found
Everything goes well on github actions until it needs to copy files. It says it can't find pnpm-lock.yaml, nor can it find ./nginx both of which are there.
I feel like the answer is super simple and I am just mentally fried and it's not standing out. If I can get a nudge in the right direction, it'd be appreciated!
I have realized that github/checkout puts you at the root directory of the github project, meaning I didn't need to add ../../ for context, I made the mistake of thinking I was navigating from the github action workflow directory.
So for anyone else who ends up this issue: actions/checkout will put you in the root directory of the repo, so you only need to use navigate via context if your dockerfiles are somewhere other than the root folder (like in their own docker folder)
I'm creating web application (angular + nest.js) and I'm trying to make multi stage build - I want to reduce size of docker image.
Angular build works good.
Nest.js one builds successfully on my pc using docker compose build, but fails on github actions with following error: Status: COPY failed: stat usr/src/app/dist: file does not exist, Code: 1
What can i do to fix it?
Dockerfile:
FROM node:16.13.1-alpine AS build
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install --only=development
RUN npm run build
COPY . .
FROM node:16.13.1-alpine
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install --only=production
COPY . .
COPY --from=build /usr/src/app/dist ./dist
EXPOSE 7000
CMD ["node", "dist/main"]
previous single stage Dockerfile (builds successfully):
FROM node:16.13.1-alpine
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
RUN npm run build
COPY . .
EXPOSE 7000
CMD [ "npm", "start" ]
docker-compose.yml:
version: '3.8'
services:
backend:
container_name: nestjs
image: ghcr.io/<gh-nickname/repo-name>/backend
build:
context: backend
dockerfile: Dockerfile
ports:
- 7000:7000
frontend:
container_name: angular
image: ghcr.io/<gh-nickname/repo-name>/frontend
build:
context: frontend
dockerfile: Dockerfile
ports:
- 8000:80
github workflow file:
name: Build and publish to github packages
on:
push:
branches:
- master
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
jobs:
build-and-push-image:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- name: Checkout
uses: actions/checkout#v2
- name: Log in to the Container registry
uses: docker/login-action#v1
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action#v3
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
- name: Build image
run: docker compose build
- name: Publish image
run: docker compose push
Try with the following Dockerfile:
FROM node:16.13.1-alpine AS build
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install --only=development
RUN npm run build
COPY . .
FROM node:16.13.1-alpine
COPY --from=build /usr/src/app/dist ./dist
EXPOSE 7000
CMD ["node", "dist/main"]
Caching multistage builds on CI Servers is not that easy as locally. You have to tell it where to get the layers from. As schema:
docker build --target build-image --cache-from=you/repo/backend:build-stage --tag you/repo/backend:build-stage
docker build --target prod-image --cache-from=you/repo/backend:build-stage --cache-from=you/repo/backend:prod-stage --tag you/repo/backend:prod-stage
docker push
So for GH Actions you can use the internal caching of GH and do something like:
steps:
- name: Build base image
id: docker_build_base
uses: docker/build-push-action#v2
with:
context: .
file: Dockerfile
push: false
cache-from: |
type=gha,scope=base
cache-to: |
type=gha,scope=base,mode=max
target: base
tags: you/repo/backend:build-stage
- name: Build prod image
id: docker_build_prod
uses: docker/build-push-action#v2
with:
context: .
file: Dockerfile
push: true
cache-from: |
type=gha,scope=prod
type=gha,scope=base
cache-to: |
type=gha,scope=prod,mode=max
target: prod
tags: you/repo/backend:prod-stage
Remark: It's showing you the right direction and no copy&paste solution. Please adjust as you need..
I managed to solved it by moving COPY . . line before RUN npm run build
I am using the Github Action actions/cache#v2 to cache the docker layers. Following is the build.yml file:
name: Build
on:
push:
branches:
- '**'
jobs:
release:
runs-on: ubuntu-latest
steps:
- name: Checkout sources
uses: actions/checkout#v2
- name: Get Docker Tags
id: getDockerTag
run: |
echo ::set-output name=image_tag::${{github.sha}}
echo "Setting image tag as :: ${{github.sha}}"
# Set up buildx runner
- name: Set up Docker Buildx
uses: docker/setup-buildx-action#v1
- name: Cache Docker layers
uses: actions/cache#v2
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-${{ hashFiles('**/Dockerfile') }}
restore-keys: |
${{ runner.os }}-buildx-
- name: Build docker image 1
uses: docker/build-push-action#v2
with:
push: false
tags: go-docker-caching:${{ steps.getDockerTag.outputs.image_tag }}
cache-from: type=local,src=/tmp/.buildx-cache
cache-to: type=local,dest=/tmp/.buildx-cache,mode=max
Dockerfile:
FROM golang:latest as builder
# create a working directory
WORKDIR /main
COPY go.mod go.sum ./
RUN ls -a
# Download dependencies
RUN go mod tidy
RUN go mod download
COPY . .
## Build binary
#RUN CGO_ENABLED=0 GOARCH=amd64 GOOS=linux go build -a -installsuffix cgo -ldflags="-w -s" -o gin_test
RUN go build -o main
# use a minimal alpine image for deployment
FROM alpine:latest
# add ca-certificates in case you need them
RUN apk update && apk add ca-certificates && rm -rf /var/cache/apk/*
# set working directory
WORKDIR /root
# copy the binary from builder
COPY --from=builder /main .
RUN touch .main.yml
# Specify the PORT
EXPOSE 8080:8080
# run the binary
CMD ["./main"]
On local, all docker commands are getting cached, but on Github Actions, the following commands are not getting cached which leads to the docker build taking around 3 minutes to download the modules, even if nothing is changed.
RUN go mod download
RUN go build -o main
How to make sure all the commands are cached?