The docker buildx build command accepts an option --cache-to type=registry,ref=my/repo:buildcache that pushes the build cache to the registry (some docs).
How can this cache be inspected?
docker buildx imagetools inspect my/repo:buildcache crashes from a null pointer dereference.
docker manifest inspect my/repo:buildcache triggers a 500 Internal Server Error.
The Docker Hub website provides no information about a tag for a pushed build cache, except that it exists.
If I run a build with the symmetric --cache-from type=registry,ref=my/repo:buildcache option, it does print importing cache manifest from my/repo:buildcache but I don't know what it has imported.
Related
Since BuildKit has been set as the default backend of docker, can we just build cache based on command docker build ... without the help of docker buildx build ... --to-cache?
I checked options list of docker build but I can not find the --to-cache option.
Can docker be connected to more than one registry at a time and how to figure out which registries it is currently connected too?
$ docker help | fgrep registr
login Log in to a Docker registry
logout Log out from a Docker registry
pull Pull an image or a repository from a registry
push Push an image or a repository to a registry
As you can see, there is no option to list the registries. I did find
a way by running:
$ docker system info | fgrep -i registr
Registry: https://index.docker.io/v1/
So... one regsitry at a time only? It is not like apt where one can point to more than one source? Anybody can point me to some good documentation about docker and registries?
Oddly, I search the web to no vail.
Aside from docker login, Docker isn't "connected to a registry" per se. Registry names are part of the image name, and Docker will connect to a registry server if it needs to pull an image.
As a specific example, the official Docker image for Elasticsearch is on a non-default registry run by Elastic. The example in that documentation is
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.17.0
# ^^^^^^^^^^^^^^^^^
# registry host name
You don't need to otherwise configure your system to connect to that registry, download an index, or anything else. In fact, you don't even need this docker pull command; if you directly docker run the image, Docker will download it if it doesn't have a copy locally.
The default registry is Docker Hub, docker.io, and this cannot be changed.
There are several alternate registries out there. The various public-cloud providers each have their own, and there are also several free-standing image registries. Each has its own instructions on how to set it up. You always need to include the registry name as part of the image name. The Google Container Registry has a simple name syntax, for example, so if you use GCR then you can
# build an image locally, labeled to be stored in GCR
# (this step does not contact or use GCR at all)
docker build gcr.io/my-name/my-image:tag
# authenticate to the registry
# (normally GCR has a Google-specific login sequence)
docker login https://gcr.io
# push the image
docker push gcr.io/my-name/my-image:tag
# run the image, pulling it if not present
docker run ... gcr.io/my-name/my-image:tag
Suppose I am building an image using Docker Buildkit. My image is from a multistage Dockerfile, like so:
FROM node:12 AS some-expensive-base-image
...
FROM some-expensive-base-image AS my-app
...
I am now trying to build both images. Suppose that I push these to Docker Hub. If I were to use Docker Buildkit's external caching feature, then I would want to try to save build time on my CI pipeline by pulling in the remote some-expensive-base-image:latest image as the cache when building the some-expensive-base-image target. And, I would want to pull in both the just-built some-expensive-base-image image and the remote my-app:latest image as the caches for the latter image. I believe that I need both in order to prevent requiring the steps of some-expensive-base-image from needing to be rebuilt, since...well...they are expensive.
This is what my build script looks like:
export DOCKER_BUILDKIT=1
docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from some-expensive-base-image:latest --target some-expensive-base-image -t some-expensive-base-image:edge .
docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from some-expensive-base-image:edge --cache-from my-app:latest --target my-app -t my-app:edge .
My question: Does the order of the --cache-from arguments matter for the second docker build?
I have been getting inconsistent results on my CI pipeline for this build. There are cache misses that are happening when building that latter image, even though there hasn't been any code changes that would have caused cache busting. The Cache Minefest can be pulled without issue. There are times when the cache image is pulled, but other times when all steps of that latter target need to be rerun. I don't know why.
By chance, should I instead try to docker pull both images before running the docker build commands in my script?
Also, I know that I referred to Docker Hub in my example, but in real life, my application uses AWS ECR for its remote Docker repository. Would that matter for proper Buildkit functionality?
Yes, the order of --cache-from matters!
See the explanation on Github from the person who implemented the feature, quoting here:
When using multiple --cache-from they are checked for a cache hit in the order that user specified. If one of the images produces a cache hit for a command only that image is used for the rest of the build.
I've had similar problems in the past, you might find useful to check ths answer, where I've shared about using Docker cache in the CI.
I have a Java application in VSTS for which a build definition has been created to generate a number of build artifacts which include an ear file and a server configuration file. All of these build artifacts are zipped up in a final build definition task.
We now wish to create a Docker file which encapsulates the above build artifacts in another VSTS Docker build task. This will be done via a build definition commandline task and it is worth pointing out that our target docker registry is a corporate registry, not Azure.
The challenge I am now facing is how to generate the required docker image from the zipped artifact (or its contents if possible). Any ideas on how this could be achieved?
To generate a docker image from zip/tar file you can use docker load command:
docker load < test.tar.gz
Loaded image: test:latest
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
test latest 769b9341d937 7 weeks ago 2.489 MB
After that you can push the image to your private registry:
docker login <REGISTRY_HOST>:<REGISTRY_PORT>
docker tag <IMAGE_ID> <REGISTRY_HOST>:<REGISTRY_PORT>/<APPNAME>:<APPVERSION>
docker push <REGISTRY_HOST>:<REGISTRY_PORT>/<APPNAME>:<APPVERSION>
Example :
docker login repo.company.com:3456
docker tag 769b9341d937 repo.company.com:3456/test:0.1
docker push repo.company.com:3456/test:0.1
So at the end of your build pipeline add a Command Line task and run the above commands (change the values to your zip file location, username, password, registry url etc.).
I'm currently faced with the task of signing our internally created docker images stored in an Artifactory docker repository on premise.
We have a target environment which (currently) has no access to the internet nor to our internal docker registry.
I've learned so far, that by enabling docker content trust with
export DOCKER_CONTENT_TRUST=1
on the machine building the images is mandatory. As far as I understood the documentation the procedure is:
Enable docker content trust on the build client
use docker push which will generate the root and targets key
Store the key(s) in a save location
Upload the image to artifactory
Is it correct, that with 2. the official notary server is/must be used to verify, that the image is indeed signed by our company?
I'm just wondering if our current deployment scenario can use docker content trust:
Store image as myDockerImage.tar.gz (i.e. docker save <IMAGE_NAME>)
copy tar.gz file to target machine
use docker load -i <FILENAME>.tar.gz to import image to local registry on the target machine
docker run (< Must fail if image is not signed by our key)
As already stated the target machine neither has access to our infrastructure nor to the internet. Is it advisable to use docker content trust for this "offline" scenario? Is there a keyfile that can be put on the target machine instead of having a connection to the notary server?
After digging for a while I came to the conclusion, that using
shasum256 <FILE(S)> deployment-files.sha256
in combination with a signature using which signs the deployment-files.sha256 via openssl is my best option.
For totally offline you can't rely on docker trust
You can associate the image repo digest (the sha256 you can use during docker pull) with the digest of a saved image file:
#!/bin/sh
set -e
IMAGE_TAG=alpine:3.14
IMAGE_HASH=sha256:a573d30bfc94d672abd141b3bf320b356e731e3b1a7d79a8ab46ba65c11d79e1
docker pull ${IMAGE_TAG}#${IMAGE_HASH}
# save the image id so the recipient can find the image after docker load
IMAGE_ID=$(docker image inspect ${IMAGE_TAG}#${IMAGE_HASH} -f {{.Id}})
echo ${IMAGE_ID} > ${IMAGE_TAG}.id
docker save -o ${IMAGE_TAG} ${IMAGE_TAG}#${IMAGE_HASH}
sha256sum ${IMAGE_TAG} ${IMAGE_TAG}.id > ${IMAGE_TAG}.sha256
You now have an image that can be used with docker load, and a digest for that image that you can verify at the receiver (assuming you trust how you convey the digest file to them - ie, you can sign it)
The recipient of the image files can then do something like:
sha256sum -c alpine:3.14.sha256
docker load -i alpine:3.14
docker inspect $(cat alpine:3.14.id)