I am running a Jenkins Alpine Linux AMD64 docker image, which I'm building myself and want to add linux/arm64 docker buildx support to it, in order to generate multi-platform images and I do not know how it supposed to work.
When I check the supported platform I get:
+ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS PLATFORMS
default * docker
default default running linux/amd64, linux/386
since I'm within an AMD64 image. I read that I need to install qemu for this, but I have no clue how buildx will recognize that.
The documentation is relatively bare on this at: https://docs.docker.com/buildx/working-with-buildx/
Anyone an idea how to add linux/arm64 build capability within a linux/amd64 image?
The only solution I see right now is to build an image on an actual arm64 system.
To use buildx, make sure your Docker runtime is at least version 19.03. buildx actually comes bundled with Docker by default, but needs to be enabled by setting the environment variable DOCKER_CLI_EXPERIMENTAL.
export DOCKER_CLI_EXPERIMENTAL=enabled
If you're on Linux, you need to set up binfmt_misc. This is pretty easy in most distributions, but is even easier now that you can just run a privileged Docker container to set it up for you.
docker run --rm --privileged docker/binfmt:66f9012c56a8316f9244ffd7622d7c21c1f6f28d
Create a new builder which gives access to the new multi-architecture features:
docker buildx create --use --name multi-arch-builder
Then you'll be able to build the containers with:
docker buildx build --platform=[your target platform] ...
This is the setup I use on my Jenkins pipeline.
Relevant documentation:
https://docs.docker.com/desktop/multi-arch/
In-depth tutorial: https://medium.com/#artur.klauser/building-multi-architecture-docker-images-with-buildx-27d80f7e2408
Related
When loading an image from a docker hub in dockerFile command FROM, if the platform is not specified, is it loaded based on the current server architecture?
I mean, when I build dockerfile on amazon linux 2 x86, is docker engine get docker image which can running on linux/x86-64?
Thank you!!
The docker build defaults to pulling base images with the platform of the target platform you are building. And that target platform defaults to the platform of your docker host. So if your docker engine is running on linux/amd64, and you do not pass --platform to either docker build or the FROM line, it will attempt to pull linux/amd64 images from a multi platform base image.
You can check your docker host by running:
docker system info --format '{{.OSType}}/{{.Architecture}}'
Just got done compiling a buildx multi-arch docker container that includes amd64, arm64 and armv7. It took about two hours to build -- with the armv7 portion being particularly slow. It only takes about 15 minutes to build natively on a RaspberryPi3.
I've setup a Docker swarm with 4 nodes, including each of the architectures I want to build for, but I haven't been able to work out how to setup the buildx cluster to use these native nodes.
I'd appreciate some guidance from anyone that's been down this particular road!
Edit:
I've made some progress with this, and here's what I know so far:
buildx clusters have nothing to do with docker swarm -- it's a separate thing
ssh public key authentication needs to be setup between the ssh "client" and "server" with your platform-specific target build machines being the servers in this case, and here's a good link for doing that:
https://www.ssh.com/academy/ssh/copy-id
test that ssh authentication is working from the machine initiating your cluster, with:
docker -H ssh://username#hostname info
you should see docker client and server info returned
create a new builder on the machine you'll be running the buildx from in the form:
docker buildx create --name <your_cluster_name> --node <your_node_name> --platform <your_desired_build_platforms> --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=10000000 --driver-opt env.BUILDKIT_STEP_LOG_MAX_SPEED=10000000
create additional target build platforms just like the above, but add --append to the options
docker buildx ls
is used to confirm the builder, nodes and platforms (the ones with an * should be what you specified and will override defaults.
docker buildx use <your_cluster_name>
will change the default builder
docker buildx inspect --bootstrap
will initiate your cluster multi-arch multi-platform builder, including downloading the buildkit image on each platform.
I had issues here, with one of my platforms requiring multiple attempts with docker buildx inspect --bootstrap. Ultimately, for my modest armv7 machine, I downloaded the:
moby/buildkit:buildx-stable-1
image on that machine first, and then ran buildx inspect from the machine I'm using to initiate the build. With the image already present, the container was spun-up without further issue.
It all looks to be setup correctly:
bnhf_cluster * docker-container
bnhf_cluster0 unix:///var/run/docker.sock running linux/amd64*, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
rpi6 ssh://pi#raspberrypi6 running linux/armv7*, linux/arm/v7
rpi5 ssh://pi#raspberrypi5 running linux/arm64*, linux/arm/v7, linux/arm/v6
Next, I'll find something new to try out a mutli-arch multi-platform build, as my primary project is all cached by buildx now -- so wouldn't be a good test.
I found this post to be the most helpful in getting multi-arch multi-platform buildx configured:
https://medium.com/#spurin/using-docker-and-multiple-buildx-nodes-for-simultaneous-cross-platform-builds-cee0f797d939
If you use portainer, you should see something like this in your container list on each of your platform specific targets, after the cluster is created:
Edit #2:
Wow, did that work well! I decided to build my primary application with the --no-cache option. The same build that took over 2 hours (with some layers already cached) on an 8th Gen i7 (4 cores, 8 threads, 32GB RAM) running Debian 11 in a WSL2 instance, took 742 seconds.
Sending the arm64 and arm/v7 jobs even to lowly Raspberry Pis is a huge improvement over the emulation offered with Docker Desktop.
I am building docker images locally on a M1 Macbook using docker-compose and run it on google compute engine.
The third image ...flask-nginx_1 works fine - probably as docker indicates it being a amd64 build.
The next_js_1 image however crashes saying 'standard_init_linux.go:228: exec user process caused: exec format error'
Is there a way to specify this in the dockerfile to create an amd64 image?
Most Docker commands support a --platform parameter, which sets the platform for the current invocation:
docker build --platform amd64 .
If you want a Dockerfile that always builds for a specific platform, you may specify the platform for an image in your Dockerfile using FROM --platform:
FROM --platform=amd64 debian
...
This overrides any command-line platform specification you might have.
In Compose, you can also specify the platform:
services:
my-svc:
platform: linux/amd64
image: my-svc-image
It's quite possible to mix images from different platforms in the same compose file.
Why are some docker images incompatible with platforms like Raspberry Pi (linux/arm/v7)?
Furthermore, can you modify a Dockerfile or another config file such that it is compatible?
Thanks for any suggestions!
So far I've installed docker and docker-compose then followed the puckel/docker-airflow readme, skipping the optional build, then tried to run the container by:
docker run -d -p 8080:8080 puckel/docker-airflow webserver
got this warning:
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested
found this issue and ran:
docker run -d -p 8080:8080 --platform linux/arm/v7 puckel/docker-airflow:latest webserver
then, this error:
docker: Error response from daemon: image with reference puckel/docker-airflow:latest was found but does not match the specified platform: wanted linux/arm/v7, actual: linux/amd64.
See 'docker run --help'.
Executable files, i.e. binary files, are dependent on the computer's architecture (amd64, arm...). Docker's image contains binary files. That is, the docker image is computer architecture dependent.
Therefore, if you look at the registry of docker, the OS and Architecture of the image are specified. Refer to the dockerhub/puckel/docker-airflow you used, linux/amd64 You can see it only supports. In other words, it doesn't work in arm architecture.
If you want to run this arm architecture there will be several ways, but the point is one. It is to build the result of building the origin code with arm, not amd64, as docker image.
In github.com/puckel/docker-airflow, guidelines for building are well stated.
First, if you look at the Dockerfile provided by the github, it starts from the image FROM python:3.7-slim-buster. For the corresponding python:3.7-slim-buster, it supports linux/arm/v5, linux/arm/v7, linux/arm/v5, linux/arm64/v8. dockerhub/python/3.7-slim-buster
In other words, you can build to arm architecture
I have experience creating images for multiple architectures through the docker buildx command. Of course, other methods exist, but I will only briefly introduce the commands below.
dockerhub/buildx
docker buildx is an experimental feature, and it is still recommended Experimental features must not be used in production environments.
docker buildx build --platform linux/arm/v5,linux/arm/v7 .
Recent versions of Docker include an overhauled build system called BuildKit that can be used with export DOCKER_BUILDKIT=1. I've noticed when it is running there is no trace of the builds in docker images or docker ps. Why not?
BuildKit runs under runc and containerd rather than directly in docker. This gives it more portability to run in other environments that do not want the full docker daemon installed and running. Because of this architecture, you will only see the resulting image that is exported from BuildKit, and not each of the individual steps as untagged images.
If you're looking to clean up the BuildKit cache, there is docker builder prune.
For more details on BuildKit, including how to run it as a standalone container or process, see their github repo: https://github.com/moby/buildkit/