I'm using the following Dockerfile to install grpc on a build image, build a cpp microservice and put that into a runtime container.
https://github.com/npclaudiu/grpc-cpp-docker/blob/master/Dockerfile
But the part that builds grpc/protobuf takes 2hours+ and that is for one service.
RUN echo "-- installing protobuf" && \
cd /var/local/git/grpc/third_party/protobuf && \
./autogen.sh && ./configure --enable-shared && \
make -j$(nproc) && make -j$(nproc) check && make install && ldconfig
I see that all language implementations are compiled while I only need C++.
Is there any way to speed up this process? Are there debian or preferably alpine linux images with grpc and protocol buffers for c++ or all languages preinstalled?
You can create an intermediate image. Split your Dockefile into 2 parts after these lines:
RUN echo "-- installing grpc" && \
cd /var/local/git/grpc && \
make -j$(nproc) && make install && make clean && ldconfig
build the first one once and for all (you'll wait 2hours only once), tag the image locally and use this image as your base image for the 2nd Dockerfile.
Related
When I try to scan docker image with nexus IQ, it flagged Component-Unknown for libraries in openjdk alpine.
usr/lib/jvm/java-1.8-openjdk/jre/lib/charsets.jar
usr/lib/jvm/java-1.8-openjdk/jre/lib/cldrdata.jar
usr/lib/jvm/java-1.8-openjdk/jre/lib/dnsns.jar
My docker file is as follows
FROM alpine:3.14
RUN apk update \
&& apk upgrade \
&& apk add --no-cache openjdk8 dumb-init \
&& rm -rf \
/usr/share/man/* \
/usr/includes/* \
/var/cache/apk/*
Is there other repo I should be getting from to get the proper libraries?
Finding the right package based on a file you know is missing on Alpine is pretty straightforward.
You just need to go to the page https://pkgs.alpinelinux.org/contents and fill in the file name.
Here, filling in the file name charsets.jar is pointing at the fact that you should install the package openjdk8-jre-lib.
Furthermore, mind that the --no-cache flag you are using is already doing update, upgrade and the rm on /var/cache/apk/*.
So, a trimmed working Dockerfile would be
FROM alpine:3.14
RUN apk add --no-cache \
dumb-init \
openjdk8 \
openjdk8-jre-lib \
&& rm -rf \
/usr/includes/* \
/usr/share/man/*
Also mind about the good practice "sort multi-line arguments", applied here.
I am trying to rewrite a Dockerfile (https://github.com/orangefoil/rcssserver-docker/blob/master/Dockerfile) so that it uses alpine instead of ubuntu. Goal is to reduce the file size.
In the original image the robocup soccer server is built from scratch using g++, flex, bison, etc.
FROM ubuntu:18.04 AS build
ARG VERSION=16.0.0
WORKDIR /root
RUN apt update && \
apt -y install autoconf bison clang flex libboost-dev libboost-all-dev libc6-dev make wget
RUN wget https://github.com/rcsoccersim/rcssserver/archive/rcssserver-$VERSION.tar.gz && \
tar xfz rcssserver-$VERSION.tar.gz && \
cd rcssserver-rcssserver-$VERSION && \
./bootstrap && \
./configure && \
make && \
make install && \
ldconfig
I tried to do the same on alpine and had to exchange some packages:
FROM alpine:latest
ARG VERSION=16.0.0
WORKDIR /root
# Add basics first
RUN apk — no-cache update \
&& apk upgrade \
&& apk add autoconf bison clang-dev flex-dev boost-dev make wget automake libtool-dev g++ build-base
RUN wget https://github.com/rcsoccersim/rcssserver/archive/rcssserver-$VERSION.tar.gz
RUN tar xfz rcssserver-$VERSION.tar.gz
RUN cd rcssserver-rcssserver-$VERSION && \
./bootstrap && \
./configure && \
make && \
make install && \
ldconfig
Unfortunately, my version doesn't work yet. It fails with
/usr/lib/gcc/x86_64-alpine-linux-musl/9.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find -lrcssclangparser
From what I found so far, this can happen, if dev packages are not installed (see ld cannot find an existing library), but I changed to dev packages where I could find them and still no luck.
So, my current assumption is that ubuntu has some package installed, that I need to add in my alpine image. I would exclude a code problem, since the ubuntu version works.
Any ideas, what could be missing? I would also be happy to understand how to compare the packages myself, but the package namings are not the same in ubuntu and alpine, so I find it pretty hard to figure this out.
You should break this up using a multi-stage build. In the image you're building now, the final image contains the C toolchain and all of the development libraries and headers that those -dev packages install; you don't need any of those to actually run the built application. The basic idea is to build the application exactly as you have it now, but then COPY only the built application into a new image with fewer dependencies.
This would look something like this (untested):
FROM ubuntu:18.04 AS build
# ... exactly what's in the original question ...
FROM ubuntu:18.04
# Install the shared libraries you need to run the application,
# but not -dev headers or the full C toolchain. You may need to
# run `ldd` on the built binary to see what exactly it needs.
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive \
apt-get install --assume-yes --no-install-recommends \
libboost-atomic1.65.1 \
libboost-chrono1.65.1 \
# ... more libboost-* libraries as required ...
# Get the built application out of the original image.
# Autoconf's default is to install into /usr/local, and in a
# typical Docker base image nothing else will be installed there.
COPY --from=build /usr/local /usr/local
RUN ldconfig
# Describe how to run a container.
EXPOSE 12345
CMD ["/usr/local/bin/rcssserver"]
Compared to the size of the C toolchain, header files, and build-time libraries, the difference between an Alpine and Ubuntu image is pretty small, and Alpine has well-documented library compatibility issues with its minimal libc implementation.
I have tried to build a docker image and found that the PATH variable I set has some issues. A Minimal non-working example is:
FROM ubuntu:latest
SHELL ["/bin/bash", "-cu"]
ARG CTAGS_DIR=/root/tools/ctags
# Install common dev tools
RUN apt-get update --allow-unauthenticated \
&& apt-get install --allow-unauthenticated -y git curl autoconf pkg-config zsh
# Compile ctags
RUN cd /tmp \
&& git clone https://github.com/universal-ctags/ctags.git \
&& cd ctags \
&& ./autogen.sh \
&& ./configure --prefix=${CTAGS_DIR} \
&& make -j$(nproc) \
&& make install \
&& rm -rf /tmp/ctags
ENV PATH=$HOME/tools/ctags/bin:$PATH
RUN echo "PATH is $PATH"
RUN which ctags
In the above Dockerfile, the line ENV PATH=$HOME/tools/ctags/bin:$PATH does not work as expected. It seems that $HOME is not correctly expanded. The following two instructions also do not work:
ENV PATH=~/tools/ctags/bin:$PATH
ENV PATH="~/tools/ctags/bin:$PATH"
Only settings the absolute path works:
# the following setting works.
ENV PATH="/root/tools/ctags/bin:$PATH"
I have looked up the docker references but can not find document about this.
In general, when you're building a Docker image, it's okay to install things into the normal "system" directories. Whatever you're building will be isolated inside the image, and it can't conflict with other tools.
The easiest answer to your immediate question is to arrange things so you don't need to set $PATH.
In the example you give, you can safely use Autoconf's default installation directory of /usr/local. That will almost certainly be empty when you start your image build and only things you install will be there.
RUN ... \
&& ./configure \
&& make \
&& make install
(The Python corollary is to not create a virtual environment for your application; just use the system pip to install things into the default Python library directories.)
Don't expect there to be a home directory. If you have to install in some non-default place, /app is common, and /opt/whatever is consistent with non-Docker Linux practice. Avoid $HOME or ~, they aren't generally well-defined in Docker (unless you go out of your way to make them be).
I have the following Dockerfile for a php runtime based on the official [php][1] image.
FROM php:fpm
WORKDIR /var/www/root/
RUN apt-get update && apt-get install -y \
libfreetype6-dev \
libjpeg62-turbo-dev \
libmcrypt-dev \
libpng12-dev \
zip \
unzip \
&& docker-php-ext-install -j$(nproc) iconv mcrypt \
&& docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ \
&& docker-php-ext-install -j$(nproc) gd \
&& docker-php-ext-install mysqli \
&& docker-php-ext-enable opcache \
&& php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" \
&& php -r "if (hash_file('SHA384', 'composer-setup.php') === '669656bab3166a7aff8a7506b8cb2d1c292f042046c5a994c43155c0be6190fa0355160742ab2e1c88d40d5be660b410') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;" \
&& php composer-setup.php \
&& php -r "unlink('composer-setup.php');" \
&& mv composer.phar /usr/local/bin/composer
I am having trouble running composer install.
I am guessing that the Dockerfile runs before a volume is mounted because I receive a composer.json file not found error if adding:
...
&& mv composer.phar /usr/local/bin/composer \
&& composer install
to the above.
But, adding the following property to docker-compose.yml:
command: sh -c "composer install && composer require drush/drush"
seems to terminate the container after the command finishes executing.
Is there a way to:
wait for a volume to become mounted
run composer install using the mounted composer.json file
have the container keep running afters
?
I generally agree with Chris's answer for local development. I am going to offer something that combines with a recent Docker feature that may set a path for doing both local development and eventual production deployment with the same image.
Let's first start with the image that we can build in a manner that can be used for either local development or deployment somewhere that contains the code and dependencies. In the latest Docker version (17.05) there is a new multi-stage build feature that we can take advantage of. In this case we can first install all your Composer dependencies to a folder in the build context and then later copy them to the final image without needing to add Composer to the final image. This might look like:
FROM composer as composer
COPY . /app
RUN composer install --ignore-platform-reqs --no-scripts
FROM php:fpm
WORKDIR /var/www/root/
RUN apt-get update && apt-get install -y \
libfreetype6-dev \
libjpeg62-turbo-dev \
libmcrypt-dev \
libpng12-dev \
zip \
unzip \
&& docker-php-ext-install -j$(nproc) iconv mcrypt \
&& docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ \
&& docker-php-ext-install -j$(nproc) gd \
&& docker-php-ext-install mysqli \
&& docker-php-ext-enable opcache
COPY . /var/www/root
COPY --from=composer /app/vendor /var/www/root/vendor
This removes all of Composer from the application image itself and instead uses the first stage to install the dependencies in another context and copy them over to the final image.
Now, during development you have some options. Based on your docker-compose.yml command it sounds like you are mounting the application into the container as .:/var/www/root. You could add a composer service to your docker-compose.yml similar to my example at https://gist.github.com/andyshinn/e2c428f2cd234b718239. Here, you just do docker-compose run --rm composer install when you need to update dependencies locally (this keeps the dependencies build inside the container which could matter for native compiled extensions, especially if you are deploying as containers and developing on Windows or Mac).
The other option is to just do something similar to what Chris has already suggested, and use the official Composer image to update and manage dependencies when needed. I've done something like this locally before where I had private dependencies on GitHub which required SSH authentication:
docker run --rm --interactive --tty \
--volume $PWD:/app:rw,cached \
--volume $SSH_AUTH_SOCK:/ssh-auth.sock \
--env SSH_AUTH_SOCK=/ssh-auth.sock \
--volume $COMPOSER_HOME:/composer \
composer:1.4 install --ignore-platform-reqs --no-scripts
To recap, the reasoning for this method of building the image and installing Composer dependencies using an external container / service:
Platform specific dependencies will be built correctly for the container (Linux architecture vs Windows or Mac).
No Composer or PHP is required on your local computer (it is all contained inside Docker and Docker Compose).
The initial image you built is runnable and deployable without needing to mount code into it. In development, you are just overriding the /var/www/root folder with a local volume.
I've been down this rabbit hole for 5 hours, all of the solutions out there are way too complicated. The easiest solution is to exclude vendor or node_modules and similar directories from volume.
#docker-compose.yml
volumes:
- .:/srv/app/
- /srv/app/vendor/
So this will map current project directory but exclude its vendor subdirectory. Dont forget the trailing slash!
So now you can easily run composer install in dockerfile and when docker mounts your volume it will ignore vendor directory.
If this is is for a general development environment, then the intention is not really ideal because it's coupling the application to the Docker configuration.
Just run composer install seperately by some other means (there is an image available for this on dockerhub, which allows you to just do (docker run -it --rm -v $(pwd):/app composer/composer install).
But yes it is possible you would need the last line in the Dockerfile to be bash -c "composer install && php-fpm".
wait for a volume to become mounted
No, volumes are not able to be mounted during a docker build process. Though you can copy the source code in.
run composer install using the mounted composer.json file
No, see above response.
have the container keep running after
Yes, you would need to execute php-fpm --nodaemonize ( which is a long running process, hence it won't terminate.
To execute a command after you have mounted a volume on a docker container
Assuming that you are fetching dependencies from a public repo
docker run --interactive -t --privileged --volume ${pwd}:/xyz composer /bin/sh -c 'composer install'
For fetching dependencies from a private git repo, you would need to copy/create ssh keys, I guess that should be out of scope of this question.
It's generally understood that if the RUN instruction cleans up after large file operations, the resulting layer size would be smaller. However when I execute the following script the resulting layer is large and roughly corresponds to the size of the files initially copied and cleaned.
How could this size of a layer be committed if the files copied are cleaned in the same layer?
RUN mkdir -p /etc/puppet && \
cd /etc/puppet && \
apt-get update && apt-get install -y wget puppet && \
wget -rnH --level=10 -e robots=off --reject "index.html*" http://somefileserver.org/puppet/allpuppetstuff && \
puppet module install puppetlabs/stdlib && \
puppet module install 7terminals-java && \
puppet apply -e "include someserver" && \
apt-get purge -y --auto-remove puppet wget && \
rm -rfv /etc/puppet/* && \
rm -rfv /var/lib/apt/lists/*
Found some details after few tweaks.
It seems docker duplicates on old layer, if a newer layer is modifying files persisted on the earlier one. This is the cause of the fat layer created by the above RUN command, even when the total file size diff at the end of operation is minimal. I couldn't find any definite resources to cite this, but this is the experience so far.
Basically, file operations related to the image should always be done in one layer.