Dockerfile go build command not getting cached in Github Actions - docker

Question

I am using the Github Action actions/cache#v2 to cache the docker layers. Following is the build.yml file:
name: Build
on:
push:
branches:
- '**'
jobs:
release:
runs-on: ubuntu-latest
steps:
- name: Checkout sources
uses: actions/checkout#v2
- name: Get Docker Tags
id: getDockerTag
run: |
echo ::set-output name=image_tag::${{github.sha}}
echo "Setting image tag as :: ${{github.sha}}"
# Set up buildx runner
- name: Set up Docker Buildx
uses: docker/setup-buildx-action#v1
- name: Cache Docker layers
uses: actions/cache#v2
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-${{ hashFiles('**/Dockerfile') }}
restore-keys: |
${{ runner.os }}-buildx-
- name: Build docker image 1
uses: docker/build-push-action#v2
with:
push: false
tags: go-docker-caching:${{ steps.getDockerTag.outputs.image_tag }}
cache-from: type=local,src=/tmp/.buildx-cache
cache-to: type=local,dest=/tmp/.buildx-cache,mode=max
Dockerfile:
FROM golang:latest as builder
# create a working directory
WORKDIR /main
COPY go.mod go.sum ./
RUN ls -a
# Download dependencies
RUN go mod tidy
RUN go mod download
COPY . .
## Build binary
#RUN CGO_ENABLED=0 GOARCH=amd64 GOOS=linux go build -a -installsuffix cgo -ldflags="-w -s" -o gin_test
RUN go build -o main
# use a minimal alpine image for deployment
FROM alpine:latest
# add ca-certificates in case you need them
RUN apk update && apk add ca-certificates && rm -rf /var/cache/apk/*
# set working directory
WORKDIR /root
# copy the binary from builder
COPY --from=builder /main .
RUN touch .main.yml
# Specify the PORT
EXPOSE 8080:8080
# run the binary
CMD ["./main"]
On local, all docker commands are getting cached, but on Github Actions, the following commands are not getting cached which leads to the docker build taking around 3 minutes to download the modules, even if nothing is changed.
RUN go mod download
RUN go build -o main
How to make sure all the commands are cached?