Using Docker Buildkit on Google Cloud Build - docker

Question

I'm trying to use BuildKit with Docker on Google Cloud Build so that I can eventually use the --secret flag. I'm using Build Enhancements for Docker as a reference.
It works on my laptop when I use the following command: DOCKER_BUILDKIT=1 docker build -t hello-world:latest .
When I run it on Cloud Build, I get the error "docker.io/docker/dockerfile:experimental not found".
Any idea how to get this to work on Cloud Build?
Here is the setup (note: I'm not using the --secret flag yet):
Dockerfile:
#syntax=docker/dockerfile:experimental
FROM node:10.15.3-alpine
RUN mkdir -p /usr/src/app && \
apk add --no-cache tini
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install --production
COPY . .
RUN chown -R node:node .
USER node
EXPOSE 8080
ENTRYPOINT ["/sbin/tini", "--"]
CMD [ "node", "index.js" ]
cloudbuild.yaml:
steps:
- id: 'Build'
name: 'gcr.io/cloud-builders/docker'
args: [
'build',
'-t', 'gcr.io/$PROJECT_ID/hello-world:latest',
'.'
]
env:
- "DOCKER_BUILDKIT=1"
Cloud Build Log:
starting build "xxxx"
FETCHSOURCE
Fetching storage object: gs://xxxxx
Copying gs://xxxxx...
/ [0 files][ 0.0 B/ 15.3 KiB]
/ [1 files][ 15.3 KiB/ 15.3 KiB]
Operation completed over 1 objects/15.3 KiB.
BUILD
Already have image (with digest): gcr.io/cloud-builders/docker
#2 [internal] load .dockerignore
#2 digest: sha256:3ce0de94c925587ad30afb764af9bef89edeb62eb891b99694aedb086ee53f50
#2 name: "[internal] load .dockerignore"
#2 started: 2019-07-24 03:21:49.153855989 +0000 UTC
#2 completed: 2019-07-24 03:21:49.195969197 +0000 UTC
#2 duration: 42.113208ms
#2 transferring context: 230B done
#1 [internal] load build definition from Dockerfile
#1 digest: sha256:82b0dcd17330313705522448d60a78d4565304d55c86f55b903b18877d612601
#1 name: "[internal] load build definition from Dockerfile"
#1 started: 2019-07-24 03:21:49.150042849 +0000 UTC
#1 completed: 2019-07-24 03:21:49.189628322 +0000 UTC
#1 duration: 39.585473ms
#1 transferring dockerfile: 445B done
#3 resolve image config for docker.io/docker/dockerfile:experimental
#3 digest: sha256:401713457b113a88eb75a6554117f00c1e53f1a15beec44e932157069ae9a9a3
#3 name: "resolve image config for docker.io/docker/dockerfile:experimental"
#3 started: 2019-07-24 03:21:49.210803849 +0000 UTC
#3 completed: 2019-07-24 03:21:49.361743084 +0000 UTC
#3 duration: 150.939235ms
#3 error: "docker.io/docker/dockerfile:experimental not found"
docker.io/docker/dockerfile:experimental not found
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: exit status 1
Laptop Docker version:
Client: Docker Engine - Community
Version: 18.09.2
API version: 1.39
Go version: go1.10.8
Git commit: 6247962
Built: Sun Feb 10 04:12:39 2019
OS/Arch: darwin/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.2
API version: 1.39 (minimum version 1.12)
Go version: go1.10.6
Git commit: 6247962
Built: Sun Feb 10 04:13:06 2019
OS/Arch: linux/amd64
Experimental: false
Cloud Build Docker Version:
Step #0 - "Version": Client:
Step #0 - "Version": Version: 18.09.7
Step #0 - "Version": API version: 1.39
Step #0 - "Version": Go version: go1.10.8
Step #0 - "Version": Git commit: 2d0083d
Step #0 - "Version": Built: Thu Jun 27 17:56:17 2019
Step #0 - "Version": OS/Arch: linux/amd64
Step #0 - "Version": Experimental: false
Step #0 - "Version":
Step #0 - "Version": Server: Docker Engine - Community
Step #0 - "Version": Engine:
Step #0 - "Version": Version: 18.09.3
Step #0 - "Version": API version: 1.39 (minimum version 1.12)
Step #0 - "Version": Go version: go1.10.8
Step #0 - "Version": Git commit: 774a1f4
Step #0 - "Version": Built: Thu Feb 28 05:59:55 2019
Step #0 - "Version": OS/Arch: linux/amd64
Step #0 - "Version": Experimental: false
Update: I noticed that I was using #syntax=docker/dockerfile:experimental whereas the linked article has #syntax=docker/dockerfile:1.0-experimental. I get the same error when using 1.0-experimental.

Answer 1

There seems to be an issue when the "registry-mirrors" option is used in combination with buildkit, then the buildkit frontend images fail to fetch:
https://github.com/moby/moby/issues/39120
Pulling them before doing the build seems to resolve the issue:
- name: 'gcr.io/cloud-builders/docker'
args: ['pull', 'docker/dockerfile:experimental']
- name: 'gcr.io/cloud-builders/docker'
args: ['pull', 'docker/dockerfile:1.0-experimental']

Answer 2

I had a similar issue and managed to figure it out. It's not really possible to use docker buildkit with gcr.io/cloud-builders/docker, instead, you have to run a docker in docker daemon and run another docker build on the side with docker-compose.
Specifically, you'll need a docker-compose.yml that has:
docker (docker in docker daemon)
a docker build step that builds the image (with buildkit enabled)
a docker auth and push step that authorizes docker to push to gcr (you need to create creds.json w/ service role w/ gcs permission, see bottom for details)
In order to auth and push to gcr, one needs to do docker login with creds.json. See details: https://cloud.google.com/container-registry/docs/advanced-authentication
# deploy/app/docker-compose.yml
version: '3.7'
services:
docker:
image: "docker:18.09.9-dind"
privileged: true
volumes:
- docker-certs-client:/certs/client
- docker-certs-ca:/certs/ca
expose:
- 2376
environment:
- DOCKER_TLS_CERTDIR=/certs
networks:
- docker-in-docker-network
docker-build:
image: docker:18.09.9
working_dir: /project
command: build -t 'gcr.io/$PROJECT_ID/<image>:<tag>'
privileged: true
depends_on:
- docker
volumes:
- docker-certs-client:/certs/client:ro
- ../../:/project
environment:
- DOCKER_TLS_CERTDIR=/certs
- DOCKER_BUILDKIT=1
networks:
- docker-in-docker-network
docker-push:
image: docker:18.09.9
working_dir: /project
entrypoint: /bin/sh -c
command:
- |
cat creds.json | docker login -u _json_key --password-stdin https://gcr.io
docker push 'gcr.io/$PROJECT_ID/<image>:<tag>'
privileged: true
depends_on:
- docker
volumes:
- docker-certs-client:/certs/client:ro
- ../../:/project
environment:
- DOCKER_CERT_PATH=/certs/client
- DOCKER_HOST=tcp://docker:2376
- DOCKER_TLS_VERIFY=1
networks:
- docker-in-docker-network
volumes:
docker-certs-ca:
docker-certs-client:
networks:
docker-in-docker-network:
Then in your cloud-build.yaml:
you need to first decrypt a creds.json (must be created and encrypted first) -- for details: https://cloud.google.com/docs/authentication/getting-started
(The push step will use the key to authorize docker login to gcr.)
Run a docker daemon from docker-compose in daemon mode (so it doesn't block the build and push step)
Run the build step docker-compose
Run the auth and push step in docker-compose.
# cloud-build.yaml
steps:
# decrypt gcloud json secret
- name: gcr.io/cloud-builders/gcloud
args:
- kms
- decrypt
- --ciphertext-file=deploy/app/creds.json.enc
- --plaintext-file=creds.json
- --location=global
- --keyring=<...>
- --key=<...>
# run docker daemon
- name: 'docker/compose:1.24.1'
args: ['-f', 'deploy/app/docker-in-docker-compose.yml', 'up', '-d', 'docker']
env:
- 'PROJECT_ID=$PROJECT_ID'
# build image
- name: 'docker/compose:1.24.1'
args: ['-f', 'deploy/app/docker-in-docker-compose.yml', 'up', 'docker-build']
env:
- 'PROJECT_ID=$PROJECT_ID'
# docker auth and push to gcr
- name: 'docker/compose:1.24.1'
args: ['-f', 'deploy/app/docker-in-docker-compose.yml', 'up', 'docker-push']
env:
- 'PROJECT_ID=$PROJECT_ID'
timeout: 600s