I am relatively new to docker and saw in other repositories that we can push multiple digests under same tag with different OS/ARCH in docker. For example:
How can I achieve the same? Right now whenever I do docker push [REPO_LINK] from different architectures, it replaces the last pushed one with it's architecture. Thanks!
You might be looking for fat manifest aka manifest list.
It enables building images with multiple architectures under same tag. You need to use docker manifest command, when using multiple machines.
Once you have pushed images from different machines, you have to finally combine the manifests of these images into single one (called as manifest list). See more from official docs.
This blog post was already mentioned in one comment, but you can still use that docker manifest example to combine manifests to single file, even if you are not working only on one machine.
Related question: Is it possible to push docker images for different architectures separately?
There are two options I know of.
First, you can have buildx run builds on multiple nodes, one for each platform, rather than using qemu. For that, you would use docker buildx create --append to add the additional nodes to the builder instance. The downside of this is you'll need the nodes accessible from the node running docker buildx which typically doesn't apply to ephemeral cloud build environments.
The second option is to use the experimental docker manifest command. Each builder would push a separate tag. And at the end of all those, you would use docker manifest create to build a manifest list and docker manifest push to push that to a registry. Since this is an experimental feature, you'll want to export DOCKER_CLI_EXPERIMENTAL=enabled to see it in the command line. (You can also modify ~/.docker/config.json to have an "experimental": "enabled" entry.)
Related
I've gone thru multiple questions posted on the forum but didn't get a clarity regarding my requirement.
I'm building a docker image after every successful CI build, there will be hardly 1 to 2 lines of changes in Dockerfile for every successful build.
Docker Build Command:
$(docker_registry)/$(Build.Repository.Name):azul
Docker Push Command
$(docker_registry)/$(Build.Repository.Name):azul
I wanted to overwrite the current docker image with the latest one(from the latest CI build changes) but retain the same tag - azul. Does docker support this ?
Yes, docker supports it. Every line you execute results in a new layer in image, that contains the changes compared to the previous layer. After modifying the Dockerfile, new layers will be created and the same preceding layers will be reused.
If you want to clean build the whole image with no cached layers, you can use the —no-cache parameter.
Mechanically this works. The new image will replace the old one with that name. The old image will still be physically present on the build system but if you look at the docker images output it will say <none> for its name; commands like docker system prune can clean these up.
The problems with this approach are on the consumer end. If I docker run registry.example.com/image:azul, Docker will automatically pull the image only if it's not already present. This can result in you running an older version of the image that happens to be on a consumer's system. This is especially a problem in cluster environments like Kubernetes, where you need a change in the text of the image name in a Kubernetes deployment specification to trigger an update.
In a CI system especially, I'd recommend assigning some sort of unique tag to every build. This could be based on the source control commit ID, or the branch name and bind number, or the current date, or something else. You can create a fixed tag like this as a convenience to developers (an image is allowed to have multiple tags) but I'd plan to not use this for actual deployments.
From what I know docker buildx build --push will overwrite existing image architectures with the one you specified in --platform parameter. As I understand you have to build and push for all architectures at the same time when using buildx. However, I know that official docker images use arm64 build farm to build linux/arm64 images. How is it possible? Do they just use docker push without buildx? If so, does it mean docker push doesn't overwrite existing architectures unlike buildx? What's the best way to do that if I want to build and push multiple architectures on separate machines?
You can build and push with separate commands on different hosts in a cluster, each sending to a different tag. And then after all tags for each platform have been pushed, you can use docker manifest to create a multiplatform manifest that points to all images with a single tag. This tool currently requires experimental support to be enabled.
Further details on docker manifest can be found in the docs: https://docs.docker.com/engine/reference/commandline/manifest/
It seems like there are two ways to add documentation to a docker image:
You can add a readme.md in the root folder (where your docker file is located) and this is meant to be parsed by the dockerhub automated build system.
The second way is by using the manifest
https://docs.docker.com/docker-hub/publish/publish/#prepare-your-image-manifest-materials
But the documentation doesn't really explain well how to annotate the manifest file for an image. Also it looks like the manifest command is considered experimental.
What is the recommended way of documenting a docker image?
Personally i prefer not having to add documentation when the container is being built, i would much rather a file in the source control. However the md file method seems to have minimal support.
Most modern container registries (like Dockerhub, Quay, Harbor) have a webinterface that can render and display documentation in Markdown format. When you do automatic builds on Dockerhub from a Github repo, the git repo's README.md can get automatically synced to the repo on Docker Hub. If you build your images locally (or via a CI runner) and push them to Docker Hub you could also push the README file using the docker-pushrm tool. It also supports other container registries than Dockerhub.
Is a Docker Hub Repository only for a single Docker File or multiple Docker Files ?
I am unclear, in my case in my have two repositories one for an Intel build (using Automated Build), and another for an Arm build of the same application that I had to build locally and push to Docker Hub.
Is that how you are meant to do it ?
With multi architecture & manifest it's possible to have many images for many architecture sharing the same tag.
I already answer one of your other post with this link : https://blog.slucas.fr/blog/docker-multiarch-manifest-hub-2/
Check this docker image, you can do a docker pull seblucas/alpine-homeassistant:latest for armhf, arm64 and amd64 without any problem (and each architecture will get its own image). The same is true for many other images provided by docker (alpine for example).
Yes, you can have multiple Dockerfiles in a repository, by using tags. Each tag corresponds to a Dockerfile, so you could two tags one called :intel and another one called :arm in the same repository.
I have dockerized a nodejs app on github. My Dockerfile is based on the offical nodejs images. The offical node-repo supports multiple architectures (x86, amd64, arm) seamlessly. This means I can build the exact same Dockerfile on different machines resulting in different images for the respective architecture.
So I am trying to offer the same architectures seamlessly for my app, too. But how?
My goal is automate it as much as possible.
I know I need in theory to create a docker-manifest, which acts as a docker-repo and redirects the end-users-docker-clients to their suitable images.
Docker-Hub itself can monitor a github repo and kick off an automated build. Thats would take care of the amd64 image. But what about the remaining architectures?
There is also the service called 'TravisCI' which I guess could take care of the arm-build with the help of qemu.
Then I think both repos could then be referenced statically by the manifest-repo. But this still leaves a couple architectures unfulfilled.
But using multiple services/ways of building the same app feels wrong. Does anyone know a better and more complete solution to this problem?
It's basically running the same dockerfile through a couple machines and recording them in a manifest.
Starting with Docker 18.02 CLI you can create multi-arch manifests and push them to the docker registries if you enabled client-side experimental features. I was able to use VSTS and create a custom build task for multi-arch tags after the build. I followed this pattern.
docker manifest create --amend {multi-arch-tag} {os-specific-tag-1} {os-specific-tag-2}
docker manifest annotate {multi-arch-tag} {os-specific-tag-1} --os {os-1} --arch {arch-1}
docker manifest annotate {multi-arch-tag} {os-specific-tag-2} --os {os-2} --arch {arch-2}
docker manifest push --purge {multi-arch-tag}
On a side note, I packaged the 18.02 docker CLI for Windows and Linux in my custom VSTS task so no install of docker was required. The manifest command does not appear to need the docker daemon to function correctly.