I need to build a docker image for arm64 and amd64, and I've already used buildx.
But the build process for the two platform is lightly different and I would need to use two different Dockerfiles. How should I do?
The detail: the image use s6-overlay and it has two different tarballs depending on the arch
ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-x86_64.tar.xz /tmp
ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-aarch64.tar.xz /tmp
You probably want to create an image that the two dockerfiles can share and do a multi-stage build process. See here in the docker docs
solution: https://github.com/just-containers/s6-overlay/issues/512#issuecomment-1421339335
ARG S6_OVERLAY_VERSION="v3.1.3.0"
RUN wget "https://github.com/just-containers/s6-overlay/releases/download/${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz" -O "/tmp/s6-overlay-noarch.tar.xz" && \
tar -C / -Jxpf "/tmp/s6-overlay-noarch.tar.xz" && \
rm -f "/tmp/s6-overlay-noarch.tar.xz"
RUN [ "${TARGETARCH}" == "arm64" ] && FILE="s6-overlay-aarch64.tar.xz" || FILE="s6-overlay-x86_64.tar.xz"; \
wget "https://github.com/just-containers/s6-overlay/releases/download/${S6_OVERLAY_VERSION}/${FILE}" -O "/tmp/${FILE}" && \
tar -C / -Jxpf "/tmp/${FILE}" && \
rm -f "/tmp/${FILE}"
Related
Given a monorepo of ~35 services using a Gitlab-CI with k8s runners.
The images are built using Kaniko, utilizing <job>.extends of a prototype template, and life is great.
However, lately, we wanted to save a key on consul and change a gitlab-ci env-var after a successful build - which requires curl, and preferably jq.
I've been trying to create the following image to serve as image for image-building jobs:
FROM gcr.io/kaniko-project/executor:debug
RUN mkdir -p /workspace \
&& wget -qO /workspace/curl https://github.com/moparisthebest/static-curl/releases/download/v7.86.0/curl-amd64 \
&& chmod +x /workspace/curl \
&& wget -qO /workspace/jq https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 \
&& chmod +x /workspace/jq
ENV PATH "$PATH:/workspace"
The build of which appears to succeed.
However - de-facto, when used in a pipeline job, given the following script:
.build-with-kaniko:
script:
- mkdir -p /kaniko/.docker;
echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":..... > /kaniko/.docker/config.json
- which jq || log no jq;
which curl || log no curl;
- >-
/kaniko/executor
--context $PROJECT_PATH
--dockerfile $DOCKERFILE
--destination ${CI_REGISTRY}/${DOCKER_REPO}:${TAG}
- which jq || log no jq;
which curl || log no curl;
Before running the executor - the curl and jq are found.
But after running the executor - they are gone!! <tam-tam-taaaaaaAAAMM!!!> :o
I tried placing them in few different folders: /busibox, /kaniko, /workspace or even a custom dir /misc- and could not get it to work...
I thought maybe it packs them to the target image - but no, they are not there.
I also noted that after building with --no-push they are still there
(but then I do not get my image on the registry...).
What is going on? is there a post-push cleanup mechanism I should instruct to leave these two files?
Help?
What must I do to help kaniko understand I need these two utilities?
OMG. :facepalm:
I knew I'll find the answer only after I post the question... :shrug:
Here's what worked:
Declare it as a new volume:
FROM gcr.io/kaniko-project/executor:debug
RUN mkdir -p /misc \
&& wget -qO /misc/curl https://github.com/moparisthebest/static-curl/releases/download/v7.86.0/curl-amd64 \
&& chmod +x /misc/curl \
&& wget -qO /misc/jq https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 \
&& chmod +x /misc/jq
VOLUME /misc
ENV PATH "$PATH:/misc"
I got the clue from the current Dockerfile of the kaniko:debug image itself (at the time of this writing).
The image is recommended to be used as the base image for gitlab-ci jobs that use kaniko - and it includes /busybox.
I still don't understand why putting the tools in /busybox dir did not work, but I got a working solution now, and no time to dig deeper :sad: :shrug:
I've the following Dockerfile:
FROM php:7.4 as base
RUN apt-get install -y libzip-dev && \
docker-php-ext-install zip
# install some other things...
FROM base as intermediate
COPY upgrade.sh /usr/local/bin
FROM base as final
COPY start-app.sh /usr/local/bin
As you can see, I've 3 stages:
base
intermedia
final
At first, I'm building the base container and then both "derived" containers. My problem is that I need to push the intermediate and the final container to my (gitlab) registry. The containers are built using the following gitlab-ci.yml:
.build_container: &build_container
stage: build
image:
name: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/moby/buildkit:rootless
entrypoint: ["sh", "-c"]
variables:
BUILDKITD_FLAGS: --oci-worker-no-process-sandbox
script:
- mkdir ~/.docker
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64)\"}}}" > ~/.docker/config.json
- |
buildctl-daemonless.sh build \
--frontend dockerfile.v0 \
--local context=${CI_PROJECT_DIR} \
--local dockerfile=${CI_PROJECT_DIR} \
--opt filename=./${DOCKERFILE} \
--import-cache type=registry,ref=${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG} \
--export-cache type=inline \
--output type=image,name=${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG},push=true
Build:Container:
<<: *build_container
variables:
DOCKERFILE: "Dockerfile"
Ok, I can simply add another "buildctl-daemonless.sh"-command and use a separate file, but I want to make sure that both containers (intermediate and final) are build successfully before pushing them. So I'm looking for a solution that builds the intermediate and the final containers at first and then pushing both to the registry, e.g. something like this:
buildctl-daemonless.sh build \
--frontend dockerfile.v0 \
--local context=${CI_PROJECT_DIR} \
--local dockerfile=${CI_PROJECT_DIR} \
--opt filename=./${DOCKERFILE} \
--import-cache type=registry,ref=${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG} \
--export-cache type=inline
buildctl-daemonless.sh push intermediate final
Unfortunately there's no "push" command in buildctl-daemonless.sh as far as I know. So es anyone having an idea how I can to this?
Directly from buildkit, I don't think there's a separate push command. The output of a multi-platform image is usually directly to a registry, but could also be an OCI Layout tar file. To write to that tar file, you would add the following option:
--output type=oci,dest=path/to/output.tar
With that tar file, there are various standalone tools that can help. Pretty sure RedHat's skopeo and Google's go-containerregistry/crane both have options. My own tool is regclient which includes the following regctl command:
regctl image import ${image_name_tag} path/to/output.tar
There's a regclient/regctl:alpine image that's made to be embedded in CI pipelines like GitLab, and it will read registry auths from the same ~/.docker/config.json.
We have our Flask API in a docker image, we push this docker to a bitbucket repository, then a bitbucket pipeline start deploying.
Everything works as expected, but the compilation of OpenCV is taking in average 15 min.
I would like to know if is there any way to avoid this compilation every time we push to bitbucket. Something like caching.
I have read about cache on bitbucket pipelines but it did not work as I expected.
This is part of my Dockerfile I would like to improve:
RUN mkdir /opt && cd /opt && \
wget -q https://github.com/opencv/opencv/archive/${OPENCV_VERSION}.zip && \
unzip ${OPENCV_VERSION}.zip && \
rm -rf ${OPENCV_VERSION}.zip && \
mkdir -p /opt/opencv-${OPENCV_VERSION}/build && \
cd /opt/opencv-${OPENCV_VERSION}/build && \
CXX=/usr/bin/clang++ CC=/usr/bin/clang cmake \
-D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D WITH_FFMPEG=NO \
-D WITH_IPP=NO \
-D WITH_OPENEXR=NO \
-D WITH_TBB=YES \
-D BUILD_EXAMPLES=NO \
-D BUILD_ANDROID_EXAMPLES=NO \
-D INSTALL_PYTHON_EXAMPLES=NO \
-D BUILD_DOCS=NO \
-D BUILD_opencv_python2=NO \
-D BUILD_opencv_python3=ON \
-D ENABLE_PYTHON3=ON \
-D PYTHON3_EXECUTABLE=/usr/bin/python3 \
.. && \
make VERBOSE=1 -j8 && \
make && \
make install && \
rm -rf /opt/opencv-${OPENCV_VERSION}
I expect some solution like just pointing a pre-compiled version of the OpenCV Api.
I have recently faced this problem and agree that cache doesn't seem to work as expected. However without looking at your entire Dockerfile, it's hard to say. ADD's and COPY's will invalidate the cache so i'd suggest you move this section up to the top if you can before adding any files.
A better solution (if there is no pre-compiled version), is to use the concept of a base image which is what I have done to cut my build time down in half. Basically you build a base image flask-api-base which will install all your packages and compile OpenCV and then your actual final image will pull FROM flask-api-base:latest and build your application specific code. Just remember if the base image changes, you may need to wipe your Bitbucket cache.
I'm unfamiliar with OpenCV but assume that, if there is a binary that you can use, that would be the ideal option.
I'm curious as to why this layer (RUN ...) isn't being cached between builds. It appears that you're cleanly separating the make of OpenCV from other statements in your Dockerfile and so, this RUN should generate a distinct layer that's stable and thus reused across builds.
Does this statement occur after earlier e.g. RUN statements that do change? If so, you may want to reorder this statement and place it earlier in the Dockerfile so that this layer becomes constant. See best practices for the Dockerfile statements that generate layers.
Alternatively, you could make a separate image containing OpenCV and then FROM this image in your code builds. You may do this either using distinct Dockerfiles or multi-stage builds. This way, this image containing the OpenCV build would only be built on (your) demand and reused across subsequent builds.
The solution I used was to create my own image, upload it to Docker hub, and create a new one based on that.
So the first docker image should contain all the basic libraries my system uses.
The second has the environmental variables and the api itself.
I am using automated builds on Docker cloud to compile a C++ app and provide it in an image.
Compilation is quite long (range 2-3 hours) and commits on github are frequent (~10 to 30 per day).
Is there a way to keep the building cache (using ccache) somehow?
As far as I understand it, docker caching is useless since the compilation layer producing the ccache will not be used due to the source code changes.
Or can we tweak to bring some data back to first layer?
Any other solution? Pushing it somewhere?
Here is the Dockerfile:
# CACHE_TAG is provided by Docker cloud
# see https://docs.docker.com/docker-cloud/builds/advanced/
# using ARG in FROM requires min v17.05.0-ce
ARG CACHE_TAG=latest
FROM qgis/qgis3-build-deps:${CACHE_TAG}
MAINTAINER Denis Rouzaud <denis.rouzaud#gmail.com>
ENV CC=/usr/lib/ccache/clang
ENV CXX=/usr/lib/ccache/clang++
ENV QT_SELECT=5
COPY . /usr/src/QGIS
WORKDIR /usr/src/QGIS/build
RUN cmake \
-GNinja \
-DCMAKE_INSTALL_PREFIX=/usr \
-DBINDINGS_GLOBAL_INSTALL=ON \
-DWITH_STAGED_PLUGINS=ON \
-DWITH_GRASS=ON \
-DSUPPRESS_QT_WARNINGS=ON \
-DENABLE_TESTS=OFF \
-DWITH_QSPATIALITE=ON \
-DWITH_QWTPOLAR=OFF \
-DWITH_APIDOC=OFF \
-DWITH_ASTYLE=OFF \
-DWITH_DESKTOP=ON \
-DWITH_BINDINGS=ON \
-DDISABLE_DEPRECATED=ON \
.. \
&& ninja install \
&& rm -rf /usr/src/QGIS
WORKDIR /
You should try saving and restoring your cache data from a third party service:
- an online object storage like Amazon S3
- a simple FTP server
- an Internet available machine with ssh to make a scp
I'm assuming that your cache data is stored inside the ´~/.ccache´ directory
Using Docker multistage build
From some time, Docker supports Multi-stage builds and you can try using it to implement the solution with a single Dockerfile:
Warning: I've not tested it
# STAGE 1 - YOUR ORIGINAL DOCKER FILE CUSTOMIZED
# CACHE_TAG is provided by Docker cloud
# see https://docs.docker.com/docker-cloud/builds/advanced/
# using ARG in FROM requires min v17.05.0-ce
ARG CACHE_TAG=latest
FROM qgis/qgis3-build-deps:${CACHE_TAG} as builder
MAINTAINER Denis Rouzaud <denis.rouzaud#gmail.com>
ENV CC=/usr/lib/ccache/clang
ENV CXX=/usr/lib/ccache/clang++
ENV QT_SELECT=5
COPY . /usr/src/QGIS
WORKDIR /usr/src/QGIS/build
# restore cache
RUN curl -o ccache.tar.bz2 http://my-object-storage/ccache.tar.bz2
RUN tar -xjvf ccache.tar.bz2
COPY --from=downloader /.ccache ~/.ccache
RUN cmake \
-GNinja \
-DCMAKE_INSTALL_PREFIX=/usr \
-DBINDINGS_GLOBAL_INSTALL=ON \
-DWITH_STAGED_PLUGINS=ON \
-DWITH_GRASS=ON \
-DSUPPRESS_QT_WARNINGS=ON \
-DENABLE_TESTS=OFF \
-DWITH_QSPATIALITE=ON \
-DWITH_QWTPOLAR=OFF \
-DWITH_APIDOC=OFF \
-DWITH_ASTYLE=OFF \
-DWITH_DESKTOP=ON \
-DWITH_BINDINGS=ON \
-DDISABLE_DEPRECATED=ON \
.. \
&& ninja install
# save the current cache online
WORKDIR ~/
RUN tar -cvjSf ccache.tar.bz2 .ccache
RUN curl -T ccache.tar.bz2 -X PUT http://my-object-storage/ccache.tar.bz2
# STAGE 2
FROM alpine:latest
# YOUR CUSTOM LOGIC TO CREATE THE FINAL IMAGE WITH ONLY REQUIRED BINARIES
# USE THE FROM IMAGE YOU NEED, this is only an example
# E.g.:
# COPY --from=builder /usr/src/QGIS/build/YOUR_EXECUTABLE /usr/bin
# ...
In the stage 2 you will build the final image that will be pushed to your repository.
Using Docker cloud hooks
Another, but less clear, approach could be using a Docker Cloud pre_build hook file to download cache data:
#!/bin/bash
echo "=> Downloading build cache data"
curl -o ccache.tar.bz2 http://my-object-storage/ccache.tar.bz2 # e.g. Amazon S3 like service
cd /
tar -xjvf ccache.tar.bz2
Obviously you can use dedicate docker images to run curl or tar mounting the local directory as a volume in this script.
Then, copy the .ccache extracted folder inside your container during the build, using a COPY command before your cmake call:
WORKDIR /usr/src/QGIS/build
COPY /.ccache ~/.ccache
RUN cmake ...
In order to make this you should find a way to upload your cache data after the build and you could make this easily using a post_build hook file:
#!/bin/bash
echo "=> Uploading build cache data"
tar -cvjSf ccache.tar.bz2 ~/.ccache
curl -T ccache.tar.bz2 -X PUT http://my-object-storage/ccache.tar.bz2
But your compilation data aren't available from the outside, because they live inside the container. So you should upload the cache after the cmake command inside your main Dockerfile:
RUN cmake...
&& tar ...
&& curl ...
&& ninja ...
&& rm ...
If curl or tar aren't available, just add them to your container using the package manager (qgis/qgis3-build-deps is based on Ubuntu 16.04, so they should be available).
This could be maybe a trivial question but reading docs for ARG and ENV doesn't put things clear to me.
I am building a PHP-FPM container and I want to give the ability for enable/disable some extensions on user needs.
Would be great if this could be done in the Dockerfile by adding conditionals and passing flags on the build command perhaps but AFAIK is not supported.
In my case and my personal approach is to run a small script when container starts, something like the following:
#!/bin/sh
set -e
RESTART="false"
# This script will be placed in /config/init/ and run when container starts.
if [ "$INSTALL_XDEBUG" == "true" ]; then
printf "\nInstalling Xdebug ...\n"
yum install -y php71-php-pecl-xdebug
RESTART="true"
fi
...
if [ "$RESTART" == "true" ]; then
printf "\nRestarting php-fpm ...\n"
supervisorctl restart php-fpm
fi
exec "$#"
This is how my Dockerfile looks like:
FROM reynierpm/centos7-supervisor
ENV TERM=xterm \
PATH="/root/.composer/vendor/bin:${PATH}" \
INSTALL_COMPOSER="false" \
COMPOSER_ALLOW_SUPERUSER=1 \
COMPOSER_ALLOW_XDEBUG=1 \
COMPOSER_DISABLE_XDEBUG_WARN=1 \
COMPOSER_HOME="/root/.composer" \
COMPOSER_CACHE_DIR="/root/.composer/cache" \
SYMFONY_INSTALLER="false" \
SYMFONY_PROJECT="false" \
INSTALL_XDEBUG="false" \
INSTALL_MONGO="false" \
INSTALL_REDIS="false" \
INSTALL_HTTP_REQUEST="false" \
INSTALL_UPLOAD_PROGRESS="false" \
INSTALL_XATTR="false"
RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm \
https://rpms.remirepo.net/enterprise/remi-release-7.rpm
RUN yum install -y \
yum-utils \
git \
zip \
unzip \
nano \
wget \
php71-php-fpm \
php71-php-cli \
php71-php-common \
php71-php-gd \
php71-php-intl \
php71-php-json \
php71-php-mbstring \
php71-php-mcrypt \
php71-php-mysqlnd \
php71-php-pdo \
php71-php-pear \
php71-php-xml \
php71-pecl-apcu \
php71-php-pecl-apfd \
php71-php-pecl-memcache \
php71-php-pecl-memcached \
php71-php-pecl-zip && \
yum clean all && rm -rf /tmp/yum*
RUN ln -sfF /opt/remi/php71/enable /etc/profile.d/php71-paths.sh && \
ln -sfF /opt/remi/php71/root/usr/bin/{pear,pecl,phar,php,php-cgi,phpize} /usr/local/bin/. && \
mv -f /etc/opt/remi/php71/php.ini /etc/php.ini && \
ln -s /etc/php.ini /etc/opt/remi/php71/php.ini && \
rm -rf /etc/php.d && \
mv /etc/opt/remi/php71/php.d /etc/. && \
ln -s /etc/php.d /etc/opt/remi/php71/php.d
COPY container-files /
RUN chmod +x /config/bootstrap.sh
WORKDIR /data/www
EXPOSE 9001
Currently this is working but ... If I want to add let's say 20 (a random number) of extensions or any other feature that can be enable|disable then I will end with 20 non necessary ENV (because Dockerfile doesn't support .env files) definition whose only purpose would be set this flag for let the script knows what to do then ...
Is this the right way to do it?
Should I use ENV for this purpose?
I am open to ideas if you have a different approach for achieve this please let me know about it
From Dockerfile reference:
The ARG instruction defines a variable that users can pass at build-time to the builder with the docker build command using the --build-arg <varname>=<value> flag.
The ENV instruction sets the environment variable <key> to the value <value>.
The environment variables set using ENV will persist when a container is run from the resulting image.
So if you need build-time customization, ARG is your best choice.
If you need run-time customization (to run the same image with different settings), ENV is well-suited.
If I want to add let's say 20 (a random number) of extensions or any other feature that can be enable|disable
Given the number of combinations involved, using ENV to set those features at runtime is best here.
But you can combine both by:
building an image with a specific ARG
using that ARG as an ENV
That is, with a Dockerfile including:
ARG var
ENV var=${var}
You can then either build an image with a specific var value at build-time (docker build --build-arg var=xxx), or run a container with a specific runtime value (docker run -e var=yyy)
So if want to set the value of an environment variable to something different for every build then we can pass these values during build time and we don't need to change our docker file every time.
While ENV, once set cannot be overwritten through command line values. So, if we want to have our environment variable to have different values for different builds then we could use ARG and set default values in our docker file. And when we want to overwrite these values then we can do so using --build-args at every build without changing our docker file.
For more details, you can refer this.
Why to use ARG or ENV ?
Let's say we have a jar file and we want to make a docker image of it. So, we can ship it to any docker engine.
We can write a Dockerfile.
Dockerfile
FROM eclipse-temurin:17-jdk-alpine
VOLUME /tmp
ARG JAR_FILE
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
Now, if we want to build the docker image using Maven, we can pass the JAR_FILE using the --build-arg as target/*.jar
docker build --build-arg JAR_FILE=target/*.jar -t myorg/myapp
However, if we are using Gradle; the above command doesn't work and we've to pass a different path: build/libs/
docker build --build-arg JAR_FILE=build/libs/*.jar -t myorg/myapp .
Once you have chosen a build system, we don’t need the ARG. We can hard code the JAR location.
For Maven, that would be as follows:
Dockerfile
FROM eclipse-temurin:17-jdk-alpine
VOLUME /tmp
COPY target/*.jar app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
here, we can build an image with the following command:
docker build -t image:tag .
When to use `ENV`?
If we want to set some values at running containers and reflect that to the image like the Port Number that your application can run/listen on. We can set that using the ENV.
Both ARG and ENV seem very similar. Both can be accessed from within our Dockerfile commands in the same manner.
Example:
ARG VAR_A 5
ENV VAR_B 6
RUN echo $VAR_A
RUN echo $VAR_B
Personal Option!
There is a tradeoff between choosing ARG over ENV. If you choose ARG you can't change it later during the run. However, if you chose ENV you can modify the value at the container.
I personally prefer ARG over ENV wherever I can, like,
In the above Example:
I have used ARG as the build system maven or Gradle impacts during build rather than runtime. It thus encapsulates a lot of details and provided a minimum set of arguments for the runtime.
For more details, you can refer to this.