I have dockerized a nodejs app on github. My Dockerfile is based on the offical nodejs images. The offical node-repo supports multiple architectures (x86, amd64, arm) seamlessly. This means I can build the exact same Dockerfile on different machines resulting in different images for the respective architecture.
So I am trying to offer the same architectures seamlessly for my app, too. But how?
My goal is automate it as much as possible.
I know I need in theory to create a docker-manifest, which acts as a docker-repo and redirects the end-users-docker-clients to their suitable images.
Docker-Hub itself can monitor a github repo and kick off an automated build. Thats would take care of the amd64 image. But what about the remaining architectures?
There is also the service called 'TravisCI' which I guess could take care of the arm-build with the help of qemu.
Then I think both repos could then be referenced statically by the manifest-repo. But this still leaves a couple architectures unfulfilled.
But using multiple services/ways of building the same app feels wrong. Does anyone know a better and more complete solution to this problem?
It's basically running the same dockerfile through a couple machines and recording them in a manifest.
Starting with Docker 18.02 CLI you can create multi-arch manifests and push them to the docker registries if you enabled client-side experimental features. I was able to use VSTS and create a custom build task for multi-arch tags after the build. I followed this pattern.
docker manifest create --amend {multi-arch-tag} {os-specific-tag-1} {os-specific-tag-2}
docker manifest annotate {multi-arch-tag} {os-specific-tag-1} --os {os-1} --arch {arch-1}
docker manifest annotate {multi-arch-tag} {os-specific-tag-2} --os {os-2} --arch {arch-2}
docker manifest push --purge {multi-arch-tag}
On a side note, I packaged the 18.02 docker CLI for Windows and Linux in my custom VSTS task so no install of docker was required. The manifest command does not appear to need the docker daemon to function correctly.
Related
I've been building some multiarch images for Java apps, based off someone else's multiarch base image which works fine, because the base image includes the right native Java version for the architecture. All I have to do is add my platform-agnostic JAR file and I have a multiarch image that works on my Windows laptop and on my rPi Kubernetes cluster.
Now though, I am experimenting with GraalVm and I would like to create a multiarch base image of my own that my projects will all use. This means I need to work out how to create a multiarch Docker image where different architectures require different installs. The amd64 Dockerfile needs the amd64 version of GraalVm to be installed and the arm Dockerfile needs the arm version of GraalVM.
I have used Dockerfile ARGs with --build-arg in the past to work out what URI to download for the GraalVm installer, but I am stuck on how to do this with 'buildx'.
It seems that every single demo/tutorial I watch uses a simple example where the files they add are not specific to the architecture. So they can just add them regardless of the platform being built for. How do people solve this problem? Is buildx not suitable for this problem?
The manifest approach seems much more appropriate for this problem (albeit more verbose) in that I can create an image for each architecture and then combine them, thus giving me the opportunity to build them differently. Is this just a constraint of the tooling? I'd rather do it with buildx if it can be done.
I am just starting to learn about docker. Is docker repository (like Docker Hub) useful? I see the docker image as a package of source code and environment configurations (dockerfile) for deploying my application. Well if it's just a package, why can't I just share my source code with the dockerfile (via GitHub for example)? Then the user just downloads it all and uses docker build and docker run. And there is no need to push the docker image to the repository.
There are two good reasons to prefer pushing an image somewhere:
As a downstream user, you can just docker run an image from a repository, without additional steps of checking it out or building it.
If you're using a compiled language (C, Java, Go, Rust, Haskell, ...) then the image will just contain the compiled artifacts and not the source code.
Think of this like any other software: for most open-source things you can download its source from the Internet and compile it yourself, or you can apt-get install or brew install a built package using a package manager.
By the same analogy, many open-source things are distributed primarily as source code, and people who aren't the primary developer package and redistribute binaries. In this context, that's the same as adding a Dockerfile to the root of your application's GitHub repository, but not publishing an image yourself. If you don't want to set up a Docker Hub account or CI automation to push built images, but still want to have your source code and instructions to build the image be public, that's a reasonable decision.
That is how it works. You need to put the configuration files in your code, i.e,
Dockerfile and docker-compose.yml.
Is a Docker Hub Repository only for a single Docker File or multiple Docker Files ?
I am unclear, in my case in my have two repositories one for an Intel build (using Automated Build), and another for an Arm build of the same application that I had to build locally and push to Docker Hub.
Is that how you are meant to do it ?
With multi architecture & manifest it's possible to have many images for many architecture sharing the same tag.
I already answer one of your other post with this link : https://blog.slucas.fr/blog/docker-multiarch-manifest-hub-2/
Check this docker image, you can do a docker pull seblucas/alpine-homeassistant:latest for armhf, arm64 and amd64 without any problem (and each architecture will get its own image). The same is true for many other images provided by docker (alpine for example).
Yes, you can have multiple Dockerfiles in a repository, by using tags. Each tag corresponds to a Dockerfile, so you could two tags one called :intel and another one called :arm in the same repository.
I am facing "theoritical" compatility issues when using distroless-based containers with kubernetess 1.10.
Actually, distroless requires docker 17.5 (https://github.com/GoogleContainerTools/distroless) whereas kubernetes does support version 17.03 only (https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.10.md#external-dependencies)
is it possible to run distroless containers within kubernetes 1.10
clusters w/o any issue?
is it possible to build distroless based
images on a build server running docker 17.05 then deploying it on a
kubernetes 1.10 cluster (docker 17.03)?
The requirement for 17.05 is only to build a "distroless" image with docker build using multistage Dockerfile. When you have an image built, there is nothing stopping it from running on older Docker / containerd versions.
Docker has supported images with no distribution for ages now by using FROM: scratch and leaving it to the image author to populate whatever the software needs, which in some cases like fully static binaries might be only the binary of the software and nothing more :)
It seems that you might need Docker 17.05+ only for building images using multi-stage files.
After you build an image with the multi-stage Dockerfile, it will be the same image in the registry like if you build it in an old-fashioned way.
Taken from Use multi-stage builds:
With multi-stage builds, you use multiple FROM statements in your Dockerfile. Each FROM instruction can use a different base, and each of them begins a new stage of the build. You can selectively copy artifacts from one stage to another, leaving behind everything you don’t want in the final image.
The end result is the same tiny production image as before, with a significant reduction in complexity.
Kubernetes does not use Dockerfiles for creating pods. It uses ready to run images from the Docker registry instead.
That's why I believe that you can use such images in Kubernetes Pods without any issues.
But anyway, to create and push your images, you have to use a build machine with Docker 17.05+ that can consume new multi-stage syntax in the Dockerfile.
(I understand this question is somewhat out of scope for stack overflow, because contains more problems and somewhat vague. Suggestions to ask it in the proper ways are welcome.)
I have some open source projects depending in each other.
The code resides in github, the builds happen in shippable, using docker images which in turn are built on docker hub.
I have set up an artifact repo and a debian repository where shippable builds put the packages, and docker builds use them.
The build chain looks like this in terms of deliverables:
pre-zenta docker image
zenta docker image (two steps of docker build because it would time out otherwise)
zenta debian package
zenta-tools docker image
zenta-tools debian package
xslt docker image
adadocs artifacts
Currently I am triggering the builds by pushing to github and sometimes rerunning failed builds on shippable after the docker build ran.
I am looking for solutions for the following problems:
Where to put Dockerfiles? Now they are in the repo of the package needing the resulting docker image for build. This way all information to build the package are in one place, but sometimes I have to trigger an extra build to have the package actually built.
How to trigger build automatically?
..., in a way supporting git-flow? For example if I change the code in zenta develop branch, I want to make sure that zenta-tools will build and test with the development version of it, before merging with master.
Are there a tool with which I can overview the health of the whole build chain?
Since your question is related to Shippable, I've created a support issue for you here - https://github.com/Shippable/support/issues/2662. If you are interested in discussing the best way to handle your scenario, you can also send me an email at support#shippable.com You can set up your entire flow, including building the docker images, using Shippable.