I have a Dockerfile and I need to include different lines depending on whether I'm running it on my development environment or a raspberry pi.
Can I add in some sort of architecture dependent IF statement around the only lines that vary?
# x64 version (shortened)
FROM node:10
COPY Gemfile* /usr/src/app/
WORKDIR /usr/src/app
RUN apt-get -y update
RUN apt-get -y install build-essential g++
RUN echo 'deb http://deb.debian.org/debian stretch main' > /etc/apt/sources.list
RUN apt-get -y update
RUN apt-get -y install ruby2.3 ruby2.3-dev
The apt source between architectures varies.
# ARM / Raspbian version. (shortened)
FROM node:10
COPY Gemfile* /usr/src/app/
WORKDIR /usr/src/app
RUN apt-get -y update
RUN apt-get -y install build-essential g++
RUN wget https://archive.raspbian.org/raspbian.public.key -O - | apt-key add -
RUN echo 'deb http://archive.raspbian.org/raspbian/ stretch main' > /etc/apt/sources.list
RUN apt-get -y update
RUN apt-get -y install ruby2.3 ruby2.3-dev
In docker file you can use ARG do define parameter for your build process like:
FROM node:10
ARG platform=x64
documentation for it
you can call it like this to change default value
docker build --build-arg platform=arm
and inside your docker file it behaves like any other variable so you can if on it:
RUN if [ "$platform" = "arm" ]; then ... else ... fi
Related
I have a dockerfile which basically does :
FROM debian
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y build-essentials automake autoconf libtool
WORKDIR /opt/build
RUN git clone https://somestuff . && make install clean
RUN rm -rf /opt/build && apt-get remove -y build-essentials automake autoconf libtool
When done, I have a 900MB image, whit only (as shown by df in the container) 40GB used
How can I improve my build strategy in order to keep the image a reasonable size ?
Regards
Xavier
Split dockerfile to multi-stage builds image.
REF https://docs.docker.com/build/building/multi-stage/
Dockerfile:
#####
# base (init base image)
#####
FROM debian AS base
ENV DEBIAN_FRONTEND=noninteractive
#####
# build-somestuff (temp build image)
#####
FROM base AS build-somestuff
WORKDIR /opt/build
RUN apt-get update && apt-get install -y build-essentials automake autoconf libtool
RUN git clone https://somestuff . && make install clean
# RUN rm -rf /opt/build && apt-get remove -y build-essentials automake autoconf libtool
#####
# main (final image)
#####
FROM base AS main
COPY --from=build-somestuff /opt/build/somestuff /bin/somestuff
WORKDIR /documents
VOLUME /documents
CMD ["/bin/bash"]
final image is main, not temp build image (build-somestuff)
main use COPY copy build make file from build-somestuff
Please reference https://hub.docker.com/r/asciidoctor/docker-asciidoctor/dockerfile for more details.
Following is my Dockerfile :-
FROM ubuntu:18.04 AS builder
RUN apt update -y
RUN apt install python3.8 -y && apt install python3-pip -y
RUN apt install build-essential automake pkg-config libtool libffi-dev libgmp-dev -y
RUN apt install libsecp256k1-dev -y
RUN apt install openjdk-8-jre -y
RUN apt install git -y
RUN apt install libkrb5-dev -y
RUN apt install vim -y
RUN mkdir /opt/app
RUN chown -R root:root /opt/app
COPY ["requirements.txt","/opt/app/requirements.txt"]
SHELL ["/bin/bash", "-c"]
WORKDIR /opt/app
RUN pip3 install -r requirements.txt && apt-get -y clean all
RUN mkdir /opt/app/
RUN chown -R root:root /opt/app/
RUN cd /opt/app/
RUN git clone -b master https://bitbucket.org/heroes/test.git
CMD ["bash","/opt/app/bin/connect.sh"]
Docker image is generating with an image file size of 1.7G. I need to have OpenJDK hence cannot use a standard python package as a base package. When I perform docker history , I can see 2 or 3 layers (installing packages above like Python3.8, OpenJDK and libsecp256k1-dev) taking up to 400MB to 500MB in size. Ubuntu as a base image takes only 64 MB however rest of size is taking by my dockerfile layers.
I believe I need to re-write the dockerfile in order to reduce the file size which I did but nothing happened concrete.
Please assist me on reducing the image less than 1 GB at least.
[Update]
Below is my updated Dockerfile:-
FROM ubuntu:18.04 AS builder
WORKDIR /opt/app
COPY requirements.txt /opt/app/aws/requirements.txt
RUN mkdir -p /opt/app/aws \
&& apt-get update -yq \
&& apt-get install -y python3.8 python3-pip openjdk-8-jre -yq && apt-get -y clean all \
&& chown -R root:root /opt/app && cd /opt/app/aws && pip3 install -r requirements.txt
FROM alpine
COPY --from=builder /opt/app /opt/app
SHELL ["/bin/bash", "-c"]
CMD ["bash","/opt/app/aws/bin/connector/connect.sh"]
Screenshot of image size:-
After removing unwanted libraries like git, etc and using the multi-stage build, the image is now approx 1.7 GB which I believe is a lot. Any suggestion to improve this?
You have multiple issues going on.
First, each of your RUN apt install is increasing your image size, you should have them all in the same RUN stage, and at the end of the stage, delete all cached apt files.
Second, you're installing unnecessary stuff. Why would you need vim and git for instance? Why are you installing build-essential and other build-related stuff if you're not building anything?
Third, it seems you tried to do a multi-stage build but ended up adding everything to the same image. Read up on python multi-stage builds.
If we consider best practices instead of multiple RUN use single RUN.
For example
RUN apt-get update -yq \
&& apt-get install -y python3-dev build-essential -yq \
&& apt-get install curl -yq \
&& pip install -r requirements.txt \
&& apt-get purge -y --auto-remove gcc python3-dev build-essential
you can use multistage builds if you don't require git in your final image you can remove in final stage
Also if possible you can use alpine version also.
Try disabling recommended packages of APT with --no-install-recommends, you can read more about it from here.
Now the image is smaller:
FROM ubuntu:18.04 AS builder
RUN apt update -y
RUN apt install python3-pip -y
RUN apt install build-essential automake pkg-config libtool libffi-dev libgmp-dev -y
RUN apt install libsecp256k1-dev -y
RUN apt install openjdk-8-jre-headless -y
RUN apt install git -y
RUN apt install libkrb5-dev -y
RUN apt install vim -y
RUN mkdir /opt/app
RUN chown -R root:root /opt/app
COPY ["requirements.txt","/opt/app/requirements.txt"]
SHELL ["/bin/bash", "-c"]
WORKDIR /opt/app
RUN pip3 install -r requirements.txt && apt-get -y clean all
RUN mkdir /opt/app/
RUN chown -R root:root /opt/app/
RUN cd /opt/app/
RUN git clone -b master https://bitbucket.org/heroes/test.git
CMD ["bash","/opt/app/bin/connect.sh"]
This is my Dockerfile:
FROM debian:latest
LABEL MAINTAINER DINESH
LABEL version="1.0"
LABEL description="First image with Dockerfile & DINESH."
RUN apt-get clean
RUN apt-get update
RUN apt-get install -qy git
RUN apt-get install -qy locales
RUN apt-get install -qy nano
RUN apt-get install -qy tmux
RUN apt-get install -qy wget
RUN apt-get install -qy python3
RUN apt-get install -qy python3-psycopg2
RUN apt-get install -qy python3-pystache
RUN apt-get install -qy python3-yaml
RUN apt-get -qy autoremove
# ** ERROR IS BELOW **
ADD .bashrc /root/.bashrc
ADD .profile /root/.profile
ADD app /app
RUN locale-gen C.UTF-8 && /usr/sbin/update-locale LANG=C.UTF-8
ENV PYTHONIOENCODING UTF-8
ENV PYTHONPATH /app/
When i run this command docker build -t myimage ., it is giving error below.
"Step 17/20 : ADD app /app
ADD failed: stat /var/lib/docker/tmp/docker-builder687980062/.bashrc: no such file or directory"
I gave permission the above give path but it is not resolved. Please let me know how I can solve it.
First please make sure file is existing in proper directory. as error suggesting no such file or directory
Please instead of ADD try using COPY working for me
COPY .bashrc /root/
COPY .profile /root/
also make file exist at source place and destination is proper.
Also as per best practices you can merge line and make a single command
RUN apt-get update -yq \
&& apt-get install -y python3-dev build-essential -yq \
&& apt-get install curl -yq \
&& pip install -r requirements.txt \
&& apt-get purge -y --auto-remove gcc python3-dev build-essential
change to:
ADD .bashrc /root/
ADD .profile /root/
ADD app /
From documentation:
ADD src ... dest.
The dest is an absolute path, or a path relative
to WORKDIR, into which the source will be copied inside the
destination container.
When I build just the main image, all the packages instead. But as soon as I turn it into a multi-stage build and it gets to RUN apt-get install -y python3-pip, I get "E: Unable to locate package in multistage Docker build"
FROM gcc:8.2.0 as builder
# FROM ownyourbits/debiandev:latest
RUN apt-get update
# RUN apt-get install -y libxerces-c-dev automake cmake libboost-all-dev build-essential
RUN apt-get install -y libxerces-c-dev automake cmake libboost-all-dev build-essential
RUN git clone https://github.com/mypackage/mypackage-d.git
WORKDIR /mypackage-d/
RUN autoreconf -if
RUN ./configure --enable-silent-rules 'CFLAGS=-g -O0 -w' 'CXXFLAGS=-g -O0 -w' 'LDFLAGS=-g -O0 -w'
RUN make
RUN make install
RUN ls .
# Main Image
FROM library/python:3.7-stretch
COPY --from=builder /mypackage-d/mypackaged.bin /mypackage-d
RUN apt-get update
RUN apt-get install -y python3
RUN apt-get install -y python3-pip
RUN apt-get install -y postgresql-client
RUN apt-get install -y libxerces-c-dev
# For VIM
RUN apt-get install -y apt-file
RUN apt-file update
RUN apt-get install -y vim
RUN pip install --upgrade pip
COPY requirements.txt /
RUN pip3 install --trusted-host pypi.org -r /requirements.txt
WORKDIR /code
ENTRYPOINT ["/bin/bash", "start.sh"]
Moving the COPY --from=builder command below the apt-get install and pip install statements worked for me.
I have the below Dockerfile.
FROM ubuntu:14.04
MAINTAINER Samuel Alexander <samuel#alexander.com>
RUN apt-get -y install software-properties-common
RUN apt-get -y update
# Install Java.
RUN echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | debconf-set-selections
RUN add-apt-repository -y ppa:webupd8team/java
RUN apt-get -y update
RUN apt-get install -y oracle-java8-installer
RUN rm -rf /var/lib/apt/lists/*
RUN rm -rf /var/cache/oracle-jdk8-installer
# Define working directory.
WORKDIR /work
# Define commonly used JAVA_HOME variable
ENV JAVA_HOME /usr/lib/jvm/java-8-oracle
# JAVA PATH
ENV PATH /usr/lib/jvm/java-8-oracle/bin:$PATH
# Install maven
RUN apt-get -y update
RUN apt-get -y install maven
# Install Open SSH and git
RUN apt-get -y install openssh-server
RUN apt-get -y install git
# clone Spark
RUN git clone https://github.com/apache/spark.git
WORKDIR /work/spark
RUN mvn -DskipTests clean package
# clone and build zeppelin fork
RUN git clone https://github.com/apache/incubator-zeppelin.git
WORKDIR /work/incubator-zeppelin
RUN mvn clean package -Pspark-1.6 -Phadoop-2.6 -DskipTests
# Install Supervisord
RUN apt-get -y install supervisor
RUN mkdir -p var/log/supervisor
# Configure Supervisord
COPY conf/supervisord.conf /etc/supervisor/conf.d/supervisord.conf
# bash
RUN sed -i s#/home/git:/bin/false#/home/git:/bin/bash# /etc/passwd
EXPOSE 8080 8082
CMD ["/usr/bin/supervisord"]
While building image it failed in step 23 i.e.
RUN mvn clean package -Pspark-1.6 -Phadoop-2.6 -DskipTests
Now when I rebuild it starts to build from step 23 as docker is using cache.
But if I want to rebuild the image from step 21 i.e.
RUN git clone https://github.com/apache/incubator-zeppelin.git
How can I do that?
Is deleting the cached image is the only option?
Is there any additional parameter to do that?
You can rebuild the entire thing without using the cache by doing a
docker build --no-cache -t user/image-name
To force a rerun starting at a specific line, you can pass an arg that is otherwise unused. Docker passes ARG values as environment variables to your RUN command, so changing an ARG is a change to the command which breaks the cache. It's not even necessary to define it yourself on the RUN line.
FROM ubuntu:14.04
MAINTAINER Samuel Alexander <samuel#alexander.com>
RUN apt-get -y install software-properties-common
RUN apt-get -y update
# Install Java.
RUN echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | debconf-set-selections
RUN add-apt-repository -y ppa:webupd8team/java
RUN apt-get -y update
RUN apt-get install -y oracle-java8-installer
RUN rm -rf /var/lib/apt/lists/*
RUN rm -rf /var/cache/oracle-jdk8-installer
# Define working directory.
WORKDIR /work
# Define commonly used JAVA_HOME variable
ENV JAVA_HOME /usr/lib/jvm/java-8-oracle
# JAVA PATH
ENV PATH /usr/lib/jvm/java-8-oracle/bin:$PATH
# Install maven
RUN apt-get -y update
RUN apt-get -y install maven
# Install Open SSH and git
RUN apt-get -y install openssh-server
RUN apt-get -y install git
# clone Spark
RUN git clone https://github.com/apache/spark.git
WORKDIR /work/spark
RUN mvn -DskipTests clean package
# clone and build zeppelin fork, changing INCUBATOR_VER will break the cache here
ARG INCUBATOR_VER=unknown
RUN git clone https://github.com/apache/incubator-zeppelin.git
WORKDIR /work/incubator-zeppelin
RUN mvn clean package -Pspark-1.6 -Phadoop-2.6 -DskipTests
# Install Supervisord
RUN apt-get -y install supervisor
RUN mkdir -p var/log/supervisor
# Configure Supervisord
COPY conf/supervisord.conf /etc/supervisor/conf.d/supervisord.conf
# bash
RUN sed -i s#/home/git:/bin/false#/home/git:/bin/bash# /etc/passwd
EXPOSE 8080 8082
CMD ["/usr/bin/supervisord"]
And then just run it with a unique arg:
docker build --build-arg INCUBATOR_VER=20160613.2 -t user/image-name .
To change the argument with every build, you can pass a timestamp as the arg:
docker build --build-arg INCUBATOR_VER=$(date +%Y%m%d-%H%M%S) -t user/image-name .
or:
docker build --build-arg INCUBATOR_VER=$(date +%s) -t user/image-name .
As an aside, I'd recommend the following changes to keep your layers smaller, the more you can merge the cleanup and delete steps on a single RUN command after the download and install, the smaller your final image will be. Otherwise your layers will include all the intermediate steps between the download and cleanup:
FROM ubuntu:14.04
MAINTAINER Samuel Alexander <samuel#alexander.com>
RUN DEBIAN_FRONTEND=noninteractive \
apt-get -y install software-properties-common && \
apt-get -y update
# Install Java.
RUN echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | debconf-set-selections && \
add-apt-repository -y ppa:webupd8team/java && \
apt-get -y update && \
DEBIAN_FRONTEND=noninteractive \
apt-get install -y oracle-java8-installer && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* && \
rm -rf /var/cache/oracle-jdk8-installer && \
# Define working directory.
WORKDIR /work
# Define commonly used JAVA_HOME variable
ENV JAVA_HOME /usr/lib/jvm/java-8-oracle
# JAVA PATH
ENV PATH /usr/lib/jvm/java-8-oracle/bin:$PATH
# Install maven
RUN apt-get -y update && \
DEBIAN_FRONTEND=noninteractive \
apt-get -y install
maven \
openssh-server \
git \
supervisor && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
# clone Spark
RUN git clone https://github.com/apache/spark.git
WORKDIR /work/spark
RUN mvn -DskipTests clean package
# clone and build zeppelin fork
ARG INCUBATOR_VER=unknown
RUN git clone https://github.com/apache/incubator-zeppelin.git
WORKDIR /work/incubator-zeppelin
RUN mvn clean package -Pspark-1.6 -Phadoop-2.6 -DskipTests
# Configure Supervisord
RUN mkdir -p var/log/supervisor
COPY conf/supervisord.conf /etc/supervisor/conf.d/supervisord.conf
# bash
RUN sed -i s#/home/git:/bin/false#/home/git:/bin/bash# /etc/passwd
EXPOSE 8080 8082
CMD ["/usr/bin/supervisord"]
One workaround:
Locate the step you want to execute from.
Before that step put a simple dummy operation like "RUN pwd"
Then just build your Dockerfile. It will take everything up to that step from the cache and then execute the lines after the dummy command.
To complete Dmitry's answer, you can use uniq arg like date +%s to keep always same commanline
docker build --build-arg DUMMY=`date +%s` -t me/myapp:1.0.0
Dockerfile:
...
ARG DUMMY=unknown
RUN DUMMY=${DUMMY} git clone xxx
...
A simpler technique.
Dockerfile:Add this line where you want the caching to start being skipped.
COPY marker /dev/null
Then build using
date > marker && docker build .
Another option is to delete the cached intermediate image you want to rebuild.
Find the hash of the intermediate image you wish to rebuild in your build output:
Step 27/42 : RUN lolarun.sh
---> Using cache
---> 328dfe03e436
Then delete that image:
$ docker image rmi 328dfe03e436
Or if it gives you an error and you're okay with forcing it:
$ docker image rmi -f 328dfe03e436
Finally, rerun your build command, and it will need to restart from that point because it's not in the cache.
If place ARG INCUBATOR_VER=unknown at top, then cache will not be used in case of change of INCUBATOR_VER from command line (just tested the build).
For me worked:
# The rebuild starts from here
ARG INCUBATOR_VER=unknown
RUN INCUBATOR_VER=${INCUBATOR_VER} git clone https://github.com/apache/incubator-zeppelin.git
As there is no official way to do this, the most simple way is to temporarily change the specific RUN command to include something harmless like echo:
RUN echo && apt-get -qq update && \
apt-get install nano
After building remove it.