I'm using Gitlab (not self hosted) for CI/CD and fail to push a docker image to Gitlab's container registry from the pipeline. While logging in and building the image appear to work, pushing it produces an error: denied: access forbidden. I don't understand why the access is forbidden.
My .gitlab-ci.yml looks like this:
Deploy backend:
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
image: docker:latest
script:
- docker build -t registry.gitlab.com/profilename/projectname/backend .
- docker push registry.gitlab.com/profilename/projectname/backend
services:
- docker:dind
stage: deploy
This is the log from the job:
[0KRunning with gitlab-runner 15.9.0~beta.115.g598a7c91 (598a7c91)[0;m
[0K on blue-5.shared.runners-manager.gitlab.com/default -AzERasQ, system ID: s_8a38c517a741[0;m
section_start:1676491269:prepare_executor
[0K[0K[36;1mPreparing the "docker+machine" executor[0;m[0;m
[0KUsing Docker executor with image docker:latest ...[0;m
[0KStarting service docker:dind ...[0;m
[0KPulling docker image docker:dind ...[0;m
[0KUsing docker image sha256:1278207f64426065791cb1a0a8967c69327c962f83273e57b63da0c55eb045ce for docker:dind with digest docker#sha256:44cd33fb5235eb32c853f17b6ae6dea3cc3fb9239b83f2e732c7de6f94829a72 ...[0;m
[0KWaiting for services to be up and running (timeout 30 seconds)...[0;m
[0KPulling docker image docker:latest ...[0;m
[0KUsing docker image sha256:1278207f64426065791cb1a0a8967c69327c962f83273e57b63da0c55eb045ce for docker:latest with digest docker#sha256:44cd33fb5235eb32c853f17b6ae6dea3cc3fb9239b83f2e732c7de6f94829a72 ...[0;m
section_end:1676491303:prepare_executor
[0Ksection_start:1676491303:prepare_script
[0K[0K[36;1mPreparing environment[0;m[0;m
Running on runner--azerasq-project-42230322-concurrent-0 via runner-azerasq-shared-1676491230-d30ae835...
section_end:1676491304:prepare_script
[0Ksection_start:1676491304:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m[0;m
[32;1m$ eval "$CI_PRE_CLONE_SCRIPT"[0;m
[32;1mFetching changes with git depth set to 20...[0;m
Initialized empty Git repository in /builds/profilename/projectname/.git/
[32;1mCreated fresh repository.[0;m
[32;1mChecking out a693f85a as detached HEAD (ref is 1-remove-login-barrier)...[0;m
[32;1mSkipping Git submodules setup[0;m
section_end:1676491307:get_sources
[0Ksection_start:1676491307:download_artifacts
[0K[0K[36;1mDownloading artifacts[0;m[0;m
[32;1mDownloading artifacts for Build backend (3778290090)...[0;m
Downloading artifacts from coordinator... ok [0;m host[0;m=storage.googleapis.com id[0;m=3778290090 responseStatus[0;m=200 OK token[0;m=64_EXTxB
section_end:1676491309:download_artifacts
[0Ksection_start:1676491309:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m[0;m
[0KUsing docker image sha256:1278207f64426065791cb1a0a8967c69327c962f83273e57b63da0c55eb045ce for docker:latest with digest docker#sha256:44cd33fb5235eb32c853f17b6ae6dea3cc3fb9239b83f2e732c7de6f94829a72 ...[0;m
[32;1m$ docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY[0;m
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[32;1m$ docker build -t registry.gitlab.com/profilename/projectname/backend .[0;m
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 121B done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 transferring context: 134B done
#2 DONE 0.0s
#3 [auth] library/golang:pull token for registry-1.docker.io
#3 DONE 0.0s
#4 [internal] load metadata for docker.io/library/golang:alpine
#4 DONE 0.5s
#5 [1/3] FROM docker.io/library/golang:alpine#sha256:48f336ef8366b9d6246293e3047259d0f614ee167db1869bdbc343d6e09aed8a
#5 resolve docker.io/library/golang:alpine#sha256:48f336ef8366b9d6246293e3047259d0f614ee167db1869bdbc343d6e09aed8a 0.0s done
#5 sha256:18da4399cedd9e383beb6b104d43aa1d48bd41167e312bb5306d72c51bd11548 1.16kB / 1.16kB done
#5 sha256:0b94e5e3eec1be96be80bab3ffc3186af109233342f79fb5051b45ba4beb6bd5 5.11kB / 5.11kB done
#5 sha256:63b65145d645c1250c391b2d16ebe53b3747c295ca8ba2fcb6b0cf064a4dc21c 3.37MB / 3.37MB 0.2s done
#5 sha256:a2d21d5440ebff5aaaaeb115a003f7a4a3897f1866a87de95bc4a21436fc563c 284.82kB / 284.82kB 0.1s done
#5 sha256:752c438cb1864d6b2151010a811031b48f0c3511c7aa49f540322590991c949d 0B / 100.63MB 0.2s
#5 sha256:48f336ef8366b9d6246293e3047259d0f614ee167db1869bdbc343d6e09aed8a 1.65kB / 1.65kB done
#5 sha256:07244a03b3147bcdf5c1256e62110d50e31af7af76ef53aae3bcc9da8410dcdc 0B / 155B 0.2s
#5 sha256:752c438cb1864d6b2151010a811031b48f0c3511c7aa49f540322590991c949d 5.24MB / 100.63MB 0.3s
#5 sha256:07244a03b3147bcdf5c1256e62110d50e31af7af76ef53aae3bcc9da8410dcdc 155B / 155B 0.3s done
#5 extracting sha256:63b65145d645c1250c391b2d16ebe53b3747c295ca8ba2fcb6b0cf064a4dc21c
#5 sha256:752c438cb1864d6b2151010a811031b48f0c3511c7aa49f540322590991c949d 13.41MB / 100.63MB 0.4s
#5 sha256:752c438cb1864d6b2151010a811031b48f0c3511c7aa49f540322590991c949d 19.92MB / 100.63MB 0.5s
#5 sha256:752c438cb1864d6b2151010a811031b48f0c3511c7aa49f540322590991c949d 27.26MB / 100.63MB 0.6s
#5 sha256:752c438cb1864d6b2151010a811031b48f0c3511c7aa49f540322590991c949d 47.19MB / 100.63MB 0.8s
#5 sha256:752c438cb1864d6b2151010a811031b48f0c3511c7aa49f540322590991c949d 56.62MB / 100.63MB 0.9s
#5 ...
#6 [internal] load build context
#6 transferring context: 22.68MB 1.0s done
#6 DONE 1.1s
#5 [1/3] FROM docker.io/library/golang:alpine#sha256:48f336ef8366b9d6246293e3047259d0f614ee167db1869bdbc343d6e09aed8a
#5 sha256:752c438cb1864d6b2151010a811031b48f0c3511c7aa49f540322590991c949d 67.11MB / 100.63MB 1.0s
#5 sha256:752c438cb1864d6b2151010a811031b48f0c3511c7aa49f540322590991c949d 73.40MB / 100.63MB 1.1s
#5 extracting sha256:63b65145d645c1250c391b2d16ebe53b3747c295ca8ba2fcb6b0cf064a4dc21c 0.9s done
#5 extracting sha256:a2d21d5440ebff5aaaaeb115a003f7a4a3897f1866a87de95bc4a21436fc563c
#5 sha256:752c438cb1864d6b2151010a811031b48f0c3511c7aa49f540322590991c949d 79.69MB / 100.63MB 1.2s
#5 sha256:752c438cb1864d6b2151010a811031b48f0c3511c7aa49f540322590991c949d 98.57MB / 100.63MB 1.4s
#5 extracting sha256:a2d21d5440ebff5aaaaeb115a003f7a4a3897f1866a87de95bc4a21436fc563c 0.2s done
#5 sha256:752c438cb1864d6b2151010a811031b48f0c3511c7aa49f540322590991c949d 100.63MB / 100.63MB 2.1s done
#5 extracting sha256:752c438cb1864d6b2151010a811031b48f0c3511c7aa49f540322590991c949d
#5 extracting sha256:752c438cb1864d6b2151010a811031b48f0c3511c7aa49f540322590991c949d 5.1s done
#5 extracting sha256:07244a03b3147bcdf5c1256e62110d50e31af7af76ef53aae3bcc9da8410dcdc
#5 extracting sha256:07244a03b3147bcdf5c1256e62110d50e31af7af76ef53aae3bcc9da8410dcdc done
#5 DONE 7.7s
#7 [2/3] COPY ./backend .
#7 DONE 2.3s
#8 [3/3] COPY ./.env .
#8 DONE 0.0s
#9 exporting to image
#9 exporting layers
#9 exporting layers 0.1s done
#9 writing image sha256:a68871721d60f549d798eeed6b0ee2cc341363fd92dc27b02226edb35715fdcd done
#9 naming to registry.gitlab.com/profilename/projectname/backend done
#9 DONE 0.1s
WARNING: buildx: git was not found in the system. Current commit information was not captured by the build
[32;1m$ docker push registry.gitlab.com/profilename/projectname/backend[0;m
Using default tag: latest
The push refers to repository [registry.gitlab.com/profilename/projectname/backend]
c7f4a40df92d: Preparing
ee97383dd371: Preparing
3ade35e5a1f0: Preparing
c6bcad44cf36: Preparing
d270ab11cf6e: Preparing
7cd52847ad77: Preparing
7cd52847ad77: Waiting
denied: access forbidden
section_end:1676491322:step_script
[0Ksection_start:1676491322:cleanup_file_variables
[0K[0K[36;1mCleaning up project directory and file based variables[0;m[0;m
section_end:1676491322:cleanup_file_variables
[0K[31;1mERROR: Job failed: exit code 1
[0;m
Building it locally and then uploading it to the Gitlab registry works perfectly. What am I missing here?
I went to Settings > CI/CD > Variables in Gitlab and deleted CI_REGISTRY_USER and CI_REGISTRY. The values of those variables looked ok to me, but apparently they get auto-populated if not defined with something else that works.
I'm building my docker images in GitLab CI and trying to cache the build so that if the app dependencies don't change, I don't need to re-build the entire image, but for some reason the Docker layer caching isn't working.
Dockerfile
FROM public.ecr.aws/lambda/python:3.9
RUN pip install poetry==1.3.2
WORKDIR /app
COPY ./pyproject.toml ./poetry.lock ./
RUN poetry config virtualenvs.create false && poetry install \
--no-interaction --no-ansi --without dev
ENV PYTHONPATH=/app
COPY . .
CMD ["app.handler"]
GitLab CI Log Output
$ docker build --cache-from "${CI_REGISTRY_IMAGE}:latest" -t $IMAGE_NAME .
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 321B done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 transferring context: 145B done
#2 DONE 0.0s
#3 [internal] load metadata for public.ecr.aws/lambda/python:3.9
#3 DONE 0.5s
#4 importing cache manifest from registry.gitlab.com/foo/bar:latest
#4 DONE 0.0s
#5 [1/6] FROM public.ecr.aws/lambda/python:3.9#sha256:7e3bcdc955c2c3cab1101bbbfd55849c0f56a6b5a21a149d50df91deacad6aac
#5 resolve public.ecr.aws/lambda/python:3.9#sha256:7e3bcdc955c2c3cab1101bbbfd55849c0f56a6b5a21a149d50df91deacad6aac 0.0s done
#5 sha256:ebf75d2390460a5a59d7c52cc4d6e7f6840c610b399c069b0ccf531792d77c7d 3.00kB / 3.00kB done
#5 sha256:7e3bcdc955c2c3cab1101bbbfd55849c0f56a6b5a21a149d50df91deacad6aac 772B / 772B done
#5 sha256:e3d5decbab4ddca757c7de36637b50826165efddecfe8b7ef76db761f51b790d 1.58kB / 1.58kB done
#5 DONE 0.3s
#6 [internal] load build context
#6 transferring context: 1.16MB 0.2s done
#6 DONE 0.3s
...
The pipeline is successfully picking up on the latest build as you can see from the importing cache manifest from registry.gitlab.com/foo/bar:latest line. However, it doesn't seem to be caching any of the build steps.
Can anyone see a reason why the cache might be missing in this case?
I am running into a problem with buildkit and I cannot figure out which is the reason.
I have one Dockerfile using as base image sles OS and it tries to do some package installation via zypper. Everytime this step is executed, not cached, it takes years to complete.
This is a dummy Dockerfile for verification of this issue.
# syntax=docker/dockerfile:1.3
FROM registry.suse.com/suse/sles12sp4
RUN zypper search iproute2
This is execution when I enable Buildkit:
docker build --no-cache --progress=plain --pull -t test_zypper .
#1 [internal] load build definition from Dockerfile
#1 sha256:1e8bc50247fba08161184996db9e2b6bca36c339623376a360765244d9d3ed8b
#1 transferring dockerfile: 202B done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 sha256:bfa4297d1f77b21d1d84347ff3f9c338cef560c9f5c8ef8f6843338b88a83178
#2 transferring context: 2B done
#2 DONE 0.0s
#3 resolve image config for docker.io/docker/dockerfile:1.3
#3 sha256:4fcd28d33487ad029eab28c03869fd56295f3902c713674c129a438f7a780653
#3 DONE 1.1s
#4 docker-image://docker.io/docker/dockerfile:1.3#sha256:42399d4635eddd7a9b8a24be879d2f9a930d0ed040a61324cfdf59ef1357b3b2
#4 sha256:7862c1373501a4a9cd96ccd04641bb1d96c86d034546e74fe74585e3dd12f952
#4 CACHED
#5 [internal] load build definition from Dockerfile
#5 sha256:adf8dd6b4b2604f820e4a4112252c8bfd5984ffa809d1fc7c5330e387575a53d
#5 DONE 0.0s
#6 [internal] load .dockerignore
#6 sha256:59c105584afe8ac8255febcea4650f6e8891b4b14fcdd7b93254039769df3828
#6 DONE 0.0s
#7 [internal] load metadata for registry.suse.com/suse/sles12sp4:latest
#7 sha256:30c143f62f5a593ad20fd34265d2933e13da97368f12f3e0c990b52851933dff
#7 DONE 0.5s
#8 [1/2] FROM registry.suse.com/suse/sles12sp4#sha256:06390bd3b9903f3d4bb1345deb7fc35e18af73de0263d0f4d5c619267bee2adf
#8 sha256:3d15a7aaf66ed6810de2347b0da9787e5a57b9c536d85ccc4b01e9eb5831bcc1
#8 CACHED
#9 [2/2] RUN zypper search iproute2
#9 sha256:17060fcd75740edd49881abc4d1b5a4f7de80f59cde5b2b6f32e97ff02bbc29d
#9 377.9 Refreshing service 'container-suseconnect-zypp'.
#9 556.7 Problem retrieving the repository index file for service 'container-suseconnect-zypp':
#9 556.7 [container-suseconnect-zypp|file:/usr/lib/zypp/plugins/services/container-suseconnect-zypp]
#9 556.7 Warning: Skipping service 'container-suseconnect-zypp' because of the above error.
#9 556.7 Loading repository data...
#9 556.7 Warning: No repositories defined. Operating only with the installed resolvables. Nothing can be installed.
#9 556.7 Reading installed packages...
#9 556.7 No matching items found.
#9 ERROR: executor failed running [/bin/sh -c zypper search iproute2]: exit code: 104
------
> [2/2] RUN zypper search iproute2:
------
executor failed running [/bin/sh -c zypper search iproute2]: exit code: 104
This is execution when I don't enable Buildkit:
time docker build --no-cache --progress=plain --pull -t test_zypper .
Sending build context to Docker daemon 678.5MB
Step 1/2 : FROM registry.suse.com/suse/sles12sp4
latest: Pulling from suse/sles12sp4
Digest: sha256:06390bd3b9903f3d4bb1345deb7fc35e18af73de0263d0f4d5c619267bee2adf
Status: Image is up to date for registry.suse.com/suse/sles12sp4:latest
---> 3126dff9c7fd
Step 2/2 : RUN zypper search iproute2
---> Running in 3efe8a741628
Refreshing service 'container-suseconnect-zypp'.
Problem retrieving the repository index file for service 'container-suseconnect-zypp':
[container-suseconnect-zypp|file:/usr/lib/zypp/plugins/services/container-suseconnect-zypp]
Warning: Skipping service 'container-suseconnect-zypp' because of the above error.
Loading repository data...
Warning: No repositories defined. Operating only with the installed resolvables. Nothing can be installed.
Reading installed packages...
No matching items found.
The command '/bin/sh -c zypper search iproute2' returned a non-zero code: 104
real 0m23.972s
user 0m1.987s
sys 0m2.161s
It is not a problem of not having repositories as in my original Dockerfile it is all defined and it eventually works, but taking 20min or more each zypper command.
Is something wrong in my way to use buildkit??
Thanks in advance!
I am trying to build a docker and it is giving me below exception. Any idea what could be wrong
First Day with the docker so my question could be bit out of sync
docker build .
[+] Building 1.2s (5/5) FINISHED
=> [internal] load build definition from Docker file 0.0s
=> => transferring dockerfile: 37B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.5s
=> CACHED [1/2] FROM docker.io/library/alpine#sha256:234cb88d3020898631af0ccbbcca9a66ae7306ecd30c9720690858c1b007d2a0 0.0s
=> ERROR [2/2] RUN apk add --update redis 0.6s
------
> [2/2] RUN apk add --update redis:
#5 0.171 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/aarch64/APKINDEX.tar.gz
#5 0.364 281472913525832:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1913:
#5 0.365 ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.14/main: Permission denied
#5 0.365 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/main: No such file or directory
#5 0.366 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/aarch64/APKINDEX.tar.gz
#5 0.536 281472913525832:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1913:
#5 0.538 ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.14/community: Permission denied
#5 0.538 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/community: No such file or directory
#5 0.538 ERROR: unable to select packages:
#5 0.541 redis (no such package):
#5 0.541 required by: world[redis]
# Use an exisiting docker image as a base
From alpine
# Download and install a dependency
Run apk add --update redis
#Tell the image what to do when it starts as a container
CMD ["redis-server"]
I am trying to take advantage of the caching/pulling system of BUILDKIT for Docker for my CI/CD process. But it does not work as expected.
I created a dummy local example (but the same happens also in my CI system - AWS CodePipeline, and for both DockerHub and AWS ECR).
The Dockerfile:
# base image
FROM python:3.7-slim
# set working directory
WORKDIR /usr/src/app
# add and install requirements
RUN pip install --upgrade pip
COPY ./requirements.txt /usr/src/app/requirements.txt
RUN pip $PIP_PROXY install --no-cache-dir --compile -r requirements.txt
RUN echo 123
# add app
COPY ./run_test.py /usr/src/app/run_test.py
# run server
CMD ["python", "run_test.py"]
run_test.py is actually not interesting, but here is the code just in case:
import requests
import time
while True:
time.sleep(1)
print(requests)
Also you need to create an empty requirements.txt file in the same folder.
In advance, I export two environment variables:
export DOCKER_BUILDKIT=1 # to activate buildkit
export DUMMY_IMAGE_URL=bi0max/test_docker
Then, to test I have the following command. First two commands remove local cache to resemble the CI environment, then build and push.
BE CAREFUL, CODE BELOW REMOVES LOCAL BUILD CACHE:
docker builder prune -a -f && \
(docker image rm $DUMMY_IMAGE_URL:latest || true) && \
docker build \
--cache-from $DUMMY_IMAGE_URL:latest \
--build-arg BUILDKIT_INLINE_CACHE=1 \
--tag $DUMMY_IMAGE_URL:latest "." && \
docker push $DUMMY_IMAGE_URL:latest
As expected, the first run just builds everything from scratch:
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 434B done
#2 DONE 0.0s
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.1s
#3 [internal] load metadata for docker.io/library/python:3.7-slim
#3 DONE 0.0s
#12 [1/7] FROM docker.io/library/python:3.7-slim
#12 DONE 0.0s
#7 [internal] load build context
#7 DONE 0.0s
#4 importing cache manifest from bi0max/test_docker:latest
#4 ERROR: docker.io/bi0max/test_docker:latest not found
#12 [1/7] FROM docker.io/library/python:3.7-slim
#12 resolve docker.io/library/python:3.7-slim done
#12 DONE 0.0s
#7 [internal] load build context
#7 transferring context: 204B done
#7 DONE 0.1s
#5 [2/7] WORKDIR /usr/src/app
#5 DONE 0.0s
#6 [3/7] RUN pip install --upgrade pip
#6 1.951 Requirement already up-to-date: pip in /usr/local/lib/python3.7/site-packages (20.1.1)
#6 DONE 2.3s
#8 [4/7] COPY ./requirements.txt /usr/src/app/requirements.txt
#8 DONE 0.0s
#9 [5/7] RUN pip $PIP_PROXY install --no-cache-dir --compile -r requirement...
#9 0.750 Collecting requests==2.22.0
#9 0.848 Downloading requests-2.22.0-py2.py3-none-any.whl (57 kB)
#9 0.932 Collecting idna<2.9,>=2.5
#9 0.948 Downloading idna-2.8-py2.py3-none-any.whl (58 kB)
#9 0.995 Collecting chardet<3.1.0,>=3.0.2
#9 1.011 Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
#9 1.135 Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
#9 1.153 Downloading urllib3-1.25.9-py2.py3-none-any.whl (126 kB)
#9 1.264 Collecting certifi>=2017.4.17
#9 1.282 Downloading certifi-2020.4.5.1-py2.py3-none-any.whl (157 kB)
#9 1.378 Installing collected packages: idna, chardet, urllib3, certifi, requests
#9 1.916 Successfully installed certifi-2020.4.5.1 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.9
#9 DONE 2.2s
#10 [6/7] RUN echo 123
#10 0.265 123
#10 DONE 0.3s
#11 [7/7] COPY ./run_test.py /usr/src/app/run_test.py
#11 DONE 0.0s
#13 exporting to image
#13 exporting layers done
#13 writing image sha256:f98327afae246096725f7e54742fe9b25079f1b779699b099e66c8def1e19052 done
#13 naming to docker.io/bi0max/test_docker:latest done
#13 DONE 0.0s
#14 exporting cache
#14 preparing build cache for export done
#14 DONE 0.0s
Then, I slightly adjust run_test.py file and the result is again as expected. All the layers until the last step ([7/7] COPY) are downloaded from repository and reused.
#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 434B done
#1 DONE 0.1s
#3 [internal] load metadata for docker.io/library/python:3.7-slim
#3 DONE 0.0s
#8 [internal] load build context
#8 DONE 0.0s
#4 [1/7] FROM docker.io/library/python:3.7-slim
#4 DONE 0.0s
#5 importing cache manifest from bi0max/test_docker:latest
#5 DONE 1.2s
#8 [internal] load build context
#8 transferring context: 193B done
#8 DONE 0.0s
#6 [2/7] WORKDIR /usr/src/app
#6 CACHED
#7 [3/7] RUN pip install --upgrade pip
#7 CACHED
#9 [4/7] COPY ./requirements.txt /usr/src/app/requirements.txt
#9 CACHED
#10 [5/7] RUN pip $PIP_PROXY install --no-cache-dir --compile -r requirement...
#10 CACHED
#11 [6/7] RUN echo 123
#11 pulling sha256:79fc69c08b391d082b4d2617faed489d220444fa0cf06953cdff55c667866bed
#11 pulling sha256:071624272167ab4e35a30eb1640cb3f15ced19c6cd10fa1c9d49763372e81c23
#11 pulling sha256:04ed4ecd76e1a110f468eb1a3173bbfa578c6b4c85a6dc82bf4a489ed8b8c54d
#11 pulling sha256:79fc69c08b391d082b4d2617faed489d220444fa0cf06953cdff55c667866bed 0.2s done
#11 pulling sha256:d6406c1ce2dc5e841233ebce164ee469388102cb98f1473adaeca15455d6d797
#11 pulling sha256:071624272167ab4e35a30eb1640cb3f15ced19c6cd10fa1c9d49763372e81c23 0.5s done
#11 pulling sha256:04ed4ecd76e1a110f468eb1a3173bbfa578c6b4c85a6dc82bf4a489ed8b8c54d 0.5s done
#11 pulling sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
#11 pulling sha256:d6406c1ce2dc5e841233ebce164ee469388102cb98f1473adaeca15455d6d797 0.3s done
#11 pulling sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 0.2s done
#11 CACHED
#12 [7/7] COPY ./run_test.py /usr/src/app/run_test.py
#12 DONE 0.0s
#13 exporting to image
#13 exporting layers done
#13 writing image sha256:f37692114f10b9a3646203569a0849af20774651f4aa0f5dc8d6f133fb7ff062 done
#13 naming to docker.io/bi0max/test_docker:latest done
#13 DONE 0.0s
#14 exporting cache
#14 preparing build cache for export done
#14 DONE 0.0s
Now, I change run_test.py again and I would expect docker to do the same thing as last time. But I get the following result, where it build everything from scratch:
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 434B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/python:3.7-slim
#3 DONE 0.0s
#5 [1/7] FROM docker.io/library/python:3.7-slim
#5 DONE 0.0s
#8 [internal] load build context
#8 DONE 0.0s
#4 importing cache manifest from bi0max/test_docker:latest
#4 DONE 1.7s
#8 [internal] load build context
#8 transferring context: 182B done
#8 DONE 0.0s
#5 [1/7] FROM docker.io/library/python:3.7-slim
#5 resolve docker.io/library/python:3.7-slim done
#5 DONE 0.1s
#6 [2/7] WORKDIR /usr/src/app
#6 DONE 0.0s
#7 [3/7] RUN pip install --upgrade pip
#7 1.774 Requirement already up-to-date: pip in /usr/local/lib/python3.7/site-packages (20.1.1)
#7 DONE 2.1s
#9 [4/7] COPY ./requirements.txt /usr/src/app/requirements.txt
#9 DONE 0.0s
#10 [5/7] RUN pip $PIP_PROXY install --no-cache-dir --compile -r requirement...
#10 0.805 Collecting requests==2.22.0
#10 0.905 Downloading requests-2.22.0-py2.py3-none-any.whl (57 kB)
#10 1.079 Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
#10 1.109 Downloading urllib3-1.25.9-py2.py3-none-any.whl (126 kB)
#10 1.242 Collecting certifi>=2017.4.17
#10 1.259 Downloading certifi-2020.4.5.1-py2.py3-none-any.whl (157 kB)
#10 1.336 Collecting idna<2.9,>=2.5
#10 1.353 Downloading idna-2.8-py2.py3-none-any.whl (58 kB)
#10 1.410 Collecting chardet<3.1.0,>=3.0.2
#10 1.428 Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
#10 1.545 Installing collected packages: urllib3, certifi, idna, chardet, requests
#10 2.102 Successfully installed certifi-2020.4.5.1 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.9
#10 DONE 2.4s
#11 [6/7] RUN echo 123
#11 0.259 123
#11 DONE 0.3s
#12 [7/7] COPY ./run_test.py /usr/src/app/run_test.py
#12 DONE 0.0s
#13 exporting to image
#13 exporting layers done
#13 writing image sha256:f4ffb0e84e334b4b35fe2504de11012e5dc1ca5978eace055932e9bbbe83c93e done
#13 naming to docker.io/bi0max/test_docker:latest done
#13 DONE 0.0s
#14 exporting cache
#14 preparing build cache for export done
#14 DONE 0.0s
But the strangest thing for me is, when I change run_test.py for the third time, it uses cached layers again. And it continues in the same way: fourth time - doesn't use, fifth time - uses, etc...
Do I miss something here?
If I pull the image each time before building, then it always uses cache, but it also works in the same way without the BUILDKIT.
This issue got fixed in newer docker versions, a simple upgrade resolves the issue.
Otherwise the solution described on GitHub can help to not rely on the systems docker version: https://github.com/moby/buildkit/issues/1981#issuecomment-785534131
I believe the inline cache image becomes invalid (or incomplete) if it was built while reusing the cache. It's either a limitation or a bug.
There is a workaround: you can tag a distinct cache image, that you'll only push to the registry when BuildKit has rebuilt the image. AFAIK there is no mean to know whether BuildKit used the cache or not, but we can see the log is filled with CACHED when it did, so we can reuse it. For example:
# enable buildkit:
$ export DOCKER_BUILDKIT=1
# build image trying to use cache image + build cache image:
$ docker build . \
--tag image:latest \
--tag image:build-cache \
--use-cache-from=image:build-cache \
--build-arg BUILDKIT_INLINE_CACHE = 1 \
| tee docker.log
# push new image to the registry:
docker push image:latest
# trick: only push cache image to the registry if it was rebuilt:
grep -q CACHED docker.log || docker push image:build-cache