I want to create a docker container to load the MEAN stack (Mongo - Node specifically). From my understanding I can't use multiple FROM statements in my Dockerfile, what is the easiest way to setup both Node and Mongo on a docker image?
Do i do this,
FROM node:0.10.40
RUN <whatever the mongo install command is>
or this,
FROM mongo:2.6.11
RUN <whatever the npm install command is>
or something else?
Look at the Dockerfiles backing these sources!
If they're both FROM comparable sources (ie. ubuntu), then you should be able to take the mongo dockerfile and modify it to go FROM the node image, thus generating an image with both services available.
Thus, amending the mongo:2.6.11 dockerfile:
FROM node:0.10.40
RUN groupadd -r mongodb && useradd -r -g mongodb mongodb
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
ca-certificates curl \
numactl \
&& rm -rf /var/lib/apt/lists/*
# grab gosu for easy step-down from root
RUN gpg --keyserver ha.pool.sks-keyservers.net --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4
RUN curl -o /usr/local/bin/gosu -SL "https://github.com/tianon/gosu/releases/download/1.6/gosu-$(dpkg --print-architecture)" \
&& curl -o /usr/local/bin/gosu.asc -SL "https://github.com/tianon/gosu/releases/download/1.6/gosu-$(dpkg --print-architecture).asc" \
&& gpg --verify /usr/local/bin/gosu.asc \
&& rm /usr/local/bin/gosu.asc \
&& chmod +x /usr/local/bin/gosu
RUN gpg --keyserver ha.pool.sks-keyservers.net --recv-keys DFFA3DCF326E302C4787673A01C4E7FAAAB2461C
ENV MONGO_VERSION 2.6.11
RUN curl -SL "https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-$MONGO_VERSION.tgz" -o mongo.tgz \
&& curl -SL "https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-$MONGO_VERSION.tgz.sig" -o mongo.tgz.sig \
&& gpg --verify mongo.tgz.sig \
&& tar -xvf mongo.tgz -C /usr/local --strip-components=1 \
&& rm mongo.tgz*
VOLUME /data/db
COPY docker-entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
EXPOSE 27017
...of course, you'll need to amend the entry point to run both services, if you were to do this.
However: Don't do this at all! The best-practices approach is to have multiple containers, one for each service, rather than building only one container that runs all services involved in your stack. Keeping your components each in their own, sandboxed namespace in this way reduces complexity in several respects: There's less room for security breaches to cross containers; there's less interdependence between containers (a software update needed for a new release of node won't break mongodb or the inverse); your containers don't need an init system or other components related to supervising multiple services; etc.
See the Container Linking documentation on the Docker website to understand how to configure your containers to be able to communicate.
Related
I am a beginner to Docker and networking concepts. I have a dockerized application in my virtualbox. I am connecting to MySQL in another computer.
I was able to connect to it, then I was able to view the app, from my virtualbox using the --net=host option. From what I understand, this option maps both the docker and virtualbox machine's networks, and that is why I was able to see it in the browser in my virtualbox.
Should I be changing anything in my Dockerfile to make sure the connections work? How would I test it out after deployment whether or not this works? My current Dockerfile contains just RUN commands to install the necessary software.
My confusion arises like this. If say I deploy this (Azure) (beginner to deployment also), what port/address should I map/expose to make sure the app works after deployment also? Currently it is a flask app which is running on 127.0.0.1 address. But should I be changing anything in the Dockerfile?
Dockerfile
FROM python:3.6.3
COPY . /app
WORKDIR /app
RUN apt update && \
apt install -y gcc g++ gfortran git patch wget && \
apt install -y vim-tiny && \
pip install --upgrade pip && \
pip install -r requirements.txt && \
cd / && \
wget http://download.redis.io/releases/redis-5.0.5.tar.gz && \
tar xzf redis-5.0.5.tar.gz && \
cd redis-5.0.5 && \
make && \
# make test && \
cd / && \
rm redis-5.0.5.tar.gz && \
cd / && \
wget https://www.coin-or.org/download/source/Ipopt/Ipopt-3.12.13.tgz && \
tar xvzf Ipopt-3.12.13.tgz && \
cd Ipopt-3.12.13/ThirdParty/Blas/ && \
./get.Blas && \
cd ../Lapack && \
./get.Lapack && \
cd ../Mumps && \
./get.Mumps && \
cd ../Metis && \
./get.Metis && \
cd ../../ && \
mkdir build && \
cd build && \
../configure && \
make -j 4 && \
make install && \
cd / && \
rm Ipopt-3.12.13.tgz
Currently I run
sudo docker run -it --net=host 5h71 /bin/bash
python app.py # inside the container
### which gives
Running on http://127.0.0.1:8050/
Debugger PIN: 052-293-642
* Serving Flask app "server" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: on
Running on http://127.0.0.1:8050/
Debugger PIN: 671-290-156
Then I access 127.0.0.1:8050 to access the app on my chrome.
With respect to your first question:
How can I include my --net=host in my Dockerfile before I deploy my container?
You can't. Runtime options that impact your host (such as network attachments, volume mounting, etc) cannot be included in your Dockerfile. This is a security restriction: if I could trick you into running an image that somehow ran with --net=host by default, I could do all sorts of nasty things.
If you have an application that requires options like that in order to be useful, and you get tired of typing long docker run command lines (or you want it to be easier for someone else to use) you can use a tool like docker-compose and create a docker-compose.yml that specifies all the appropriate runtime options.
I have been trying for a while to copy files via ssh from a remote server (not gihub) inside the docker image I want to build, but I can't connect to host. Here is the Dockerfile up until the critical point:
FROM r-base:latest
### Install libs
RUN apt-get update && apt-get install -y \
sudo \
gdebi-core \
pandoc \
pandoc-citeproc \
openssh-server \
openssh-client \
libcurl4-gnutls-dev \
libcairo2-dev \
libxt-dev \
xtail \
wget \
libssl-dev \
libxml2 \
libxml2-dev \
libv8-dev \
curl \
gnupg \
git
COPY ./setup setup
RUN mv setup/.ssh ~/.ssh
RUN touch ~/.ssh/known_hosts
RUN chmod -R 400 ~/.ssh
RUN ssh-agent sh -c 'ssh-add /root/.ssh/id_rsa'
#RUN eval "$(ssh-agent -s)"
#RUN ssh-add -K ~/.ssh/id_rsa #This is commented out as it causes an error
RUN ssh-keyscan hostname > ~/.ssh/known_host
RUN ssh-keygen -R hostname
## THIS IS THE COMMAND WE NEED TO RUN...
RUN scp -r user#hostname:/path/to/folder ./
The owner of the folder is user. The id_rsa.pub was added to the authorized_keys file of the user user on the host, and ssh was restarted there. However I get a Failed authentication error. I tried to use my personal id_rsa which works from the command line, but it also fails inside docker. Is this a docker issuor is it solvable?
I finally managed to do it by generating a key with the command suggested in this post
So to reproduce my case, locally:
cd setup/.ssh/
ssh-keygen -q -t rsa -N '' -f id_rsa
Then on the server add the id_rsa.pub contents to the known hosts for the user user. Can copy the contents to clipboard using xclip: xclip -sel clip < setup/.ssh/id_rsa.pub)
Dockerfile:
I have been trying for a while to copy files via ssh from a remote server (not gihub) inside the docker image I want to build, but I can't connect to host. Here is the Dockerfile up until the critical point:
FROM r-base:latest
### Install libs
RUN apt-get update && apt-get install -y \
sudo \
gdebi-core \
pandoc \
pandoc-citeproc \
openssh-server \
openssh-client \
libcurl4-gnutls-dev \
libcairo2-dev \
libxt-dev \
xtail \
wget \
libssl-dev \
libxml2 \
libxml2-dev \
libv8-dev \
curl \
gnupg \
git
COPY ./setup setup
RUN chmod -R 600 ~/.ssh
RUN echo "IdentityFile /root/.ssh/id_rsa" >> /etc/ssh/ssh_config
RUN echo "StrictHostKeyChecking no" >> /etc/ssh/ssh_config
## THIS IS THE COMMAND WE NEED TO RUN...
RUN scp -r user#hostname:/path/to/folder ./
There’s no specific requirement that you must do everything inside your Dockerfile. Especially things that require remote ssh access are better done outside Docker: consider that anyone who gets your image later on can docker cp a valid ssh key out of it and potentially get access to your internal systems.
For Docker caching reasons, it’s also not a good idea to git clone or otherwise try to remotely retrieve your application from inside the Dockerfile. If you re-run docker build, and nothing else in your Dockerfile has changed, then Docker will skip over the scp step too, even if the remote content has changed.
My general recommendation would be to copy this content from outside the Dockerfile, then build it
# Using whatever credentials are in your local ssh-agent
scp -r user#hostname:/path/to/stuff dist/
# Then your Dockerfile doesn’t need scp or credentials
docker build .
Your Dockerfile then doesn’t need a bunch of extra packages that are only relevant to this path: you should be able to remove sudo openssh-server openssh-client xtail curl gnupg git without actually affecting the single main process you’re trying to run inside your container.
I am following these instructions to build a docker image from python3.7 that also contains ruby - i need 2.1.5 specifically (don't ask).
So my Dockerfile is this:
FROM python:3.7
SHELL ["/bin/bash", "-c"]
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
build-essential \
curl \
dirmngr \
gnupg2 \
&& curl -sSL https://rvm.io/mpapis.asc | gpg2 --import - \
&& gpg2 --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB \
&& curl -sSL https://get.rvm.io | bash -s stable \
&& usermod -a -G rvm root \
&& source /etc/profile.d/rvm.sh \
&& rvm install ruby-2.1.5
The build completes with success but the end image does not have ruby in its path which is the result of this command source /etc/profile.d/rvm.sh actually not having any effect during the build.
As soon as I get a cmd in the container and running it, ruby is available.
Why is that?
See this:
The RUN instruction will execute any commands in a new layer on top of the current image and commit the results.
So, all RUN in Dockerfile will just affect the image build, not container. You should put any source in entrypoint or cmd. Or make symbol link in Dockerfile to link it to /bin/.
If you source a script inside a shell that only exists for the command, it will not have any lasting effect on any future commands.
Try this:
RUN /bin/bash -c "source /etc/profile.d/rvm.sh; command1; command2; command3;"
docker image inspect <name>
gives me 16GB
and about 20 layers
When I am logged as root, this
du -hs /
show me just 2GB
FYI, there are already very multi-lines RUN commands in Dockerfile.
can I squash all layers into one layer without touching Dockerfile, rebuilding etc?
or possibly by adding extra action to Dockerfile which clear/improve caching
Dockerfile is
FROM heroku/heroku:18
ENV PYENV_ROOT="/pyenv"
ENV PATH="/pyenv/shims:/pyenv/bin:$PATH"
ENV PYTHON_VERSION 3.5.6
ENV GPG_KEY <value>
ENV PYTHONUNBUFFERED 1
ENV TERM xterm
ENV EDITOR vim
RUN apt-get update && apt-get install -y \
build-essential \
gdal-bin \
binutils \
iputils-ping \
libjpeg8 \
libproj-dev \
libjpeg8-dev \
libtiff-dev \
zlib1g-dev \
libfreetype6-dev \
liblcms2-dev \
libxml2-dev \
libxslt1-dev \
libssl-dev \
libncurses5-dev \
virtualenv \
python-pip \
python3-pip \
python-dev \
libmysqlclient-dev \
mysql-client-5.7 \
libpq-dev \
libcurl4-gnutls-dev \
libgnutls28-dev \
libbz2-dev \
tig \
git \
vim \
nano \
tmux \
tmuxinator \
fish \
sudo \
libnet-ifconfig-wrapper-perl \
ruby \
libssl-dev \
nodejs \
strace \
tcpdump \
# npm & grunt
&& curl -L https://npmjs.com/install.sh | sh \
&& npm install -g grunt-cli grunt \
# ruby & foreman
&& gem install foreman \
# installing pyenv
&& curl https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash
COPY . /app
COPY ./requirements /requirements
COPY ./requirements.txt /requirements.txt
COPY ./docker/docker_compose/django/foreman.sh /foreman.sh
COPY ./docker/docker_compose/django/Procfile /Procfile
COPY ./docker/docker_compose/django/entrypoint.sh /entrypoint.sh
# ADD sudoer user django with password django
RUN groupadd -r django -g 1000 && \
useradd -ms /usr/bin/fish -p $(openssl passwd -1 django) --uid 1000 --gid 1000 -r -g django django && \
usermod -a -G sudo django && \
chown -R django:django /app
COPY --chown=django:django ./docker/docker_compose/django/fish /home/django/.config/fish
COPY --chown=django:django ./docker/docker_compose/django/tmuxinator /home/django/.tmuxinator
COPY ./docker/docker_compose/django/fish /root/.config/fish
WORKDIR /app
RUN sed -i 's/\r//' /entrypoint.sh \
&& sed -i 's/\r//' /foreman.sh \
&& chmod +x /entrypoint.sh \
&& chown django /entrypoint.sh \
&& chmod +x /foreman.sh \
&& chown django /foreman.sh \
&& chown -R django:django /home/django/ \
&& pyenv install ${PYTHON_VERSION%%} \
&& mkdir -p /app/log \
&& pyenv global ${PYTHON_VERSION%%} \
&& pyenv rehash \
&& ${PYENV_ROOT%%}/versions/${PYTHON_VERSION%%}/bin/pip install -U pip \
&& ${PYENV_ROOT%%}/versions/${PYTHON_VERSION%%}/bin/pip install -r /requirements.txt \
&& chown -R django:django /pyenv/ \
&& ${PYENV_ROOT%%}/versions/${PYTHON_VERSION%%}/bin/pip install -r /requirements/dev_requirements.txt
# this user receives ENVs from the top
USER django
ENTRYPOINT ["/entrypoint.sh"]
What I've tried so far:
The --squash option from experimental mode of docker build is rather not for me. That Dockerfile is one of more Dockerfiles inside docker-compose.
I've also checked this:
https://github.com/jwilder/docker-squash
but seems docker load cannot load a squashed image.
also, that squash gives me 8GB (still far away from expected ~2GB)
docker save <image_id> | docker-squash -t latest_tiny | docker load
update after answers:
when I've added this:
&& apt-get autoremove \ # ? to consider
&& apt-get clean \ # ? to consider
&& rm -rf /var/lib/apt/lists/*
to apt-get and --no-cache-dir to each pip, the result was 72GB (yes, even much more - docker images shows 36GB before pip command, and 72GB as final size).
my working directory is clear (regarding COPY). du -hs / (as a root) still has 2GB. And all images were removed before rebuilding.
Following the #Mihai approach, I was able to slim down the image from 16GB to 9GB.
There is a simple trick to get rid of the intermediate layers. It will bring down the size as well but with how much depends on how it was built.
Create a Dockerfile like this:
FROM your_image as initial
FROM your_image_base
COPY --from=initial / /
your_image_base should be something like 'alpine' - so the smallest image from which your image and its parents descend from.
Now build the image and check the history and size:
docker build -t your-image:2.0 .
docker image history your-image:2.0
docker image ls
This way you do create a new Dockerfile (if that is acceptable for your process) without touching the initial Dockerfile.
Let me know if this solves your issue.
UPDATE AFTER SEEING THE Dockerfile:
maybe I miss it but I don't see you cleaning up the apt-get cache after you perform the installations. Your big RUN command should end with "&& rm -rf /var/lib/apt/lists/*" on the same line so that it doesn't store the whole cache on the layer.
Definitely add && rm -rf /var/lib/apt/lists/* on the end of your main run command, like Mihai said. Another thing that may help (depending on how big your dependencies are) is installing with pip using the --no-cache-dir option . Also, make sure you understand build context and consider using either a .dockerignore or sending the context to another directory (totally depends on how you're directory is setup)
I've also had luck exploring an image using dive. Honestly this looks like a pretty big image so not sure how much you're going to be able to get it down
To squash a (Docker) container image, without re-building the image or manipulating the original Dockerfile,
You can extend from your image and squash it:
docker build --squash -t your_image_squashed - <<< "FROM your_image"
It's very easy, just use
docker commit YOUR_CONTAINER_ID NEW_IMAGE_ID
The docker will throw away the intermediate layers, you lost history but the size is small
I am working on continuous integration of a .net project using jenkins. By far, i am able to set a jenkins job in window. But now i need to replicate this all in jenkins running as docker container. I am able to start jenkins in docker, using github as source repository but when i try to build this project, it fails. My project is using asp.net core so i am assuming it should run on linux as well (which is the OS of docker virtual machine).
What i am missing here? Any help is highly appreciated
I'm working on a project with .Net core and we started to use Jenkins in Docker container, so far the only way I found was to create a custom Jenkins image. This is my docker file:
FROM jenkins
USER root
# Work around https://github.com/dotnet/cli/issues/1582 until Docker releases a
# fix (https://github.com/docker/docker/issues/20818). This workaround allows
# the container to be run with the default seccomp Docker settings by avoiding
# the restart_syscall made by LTTng which causes a failed assertion.
ENV LTTNG_UST_REGISTER_TIMEOUT 0
# Install .NET CLI dependencies
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
libc6 \
libcurl3 \
libgcc1 \
libgssapi-krb5-2 \
libicu52 \
liblttng-ust0 \
libssl1.0.0 \
libstdc++6 \
libunwind8 \
libuuid1 \
zlib1g \
&& rm -rf /var/lib/apt/lists/*
# Install .NET Core SDK
ENV DOTNET_SDK_VERSION 1.0.0-preview2-003131
ENV DOTNET_SDK_DOWNLOAD_URL https://dotnetcli.blob.core.windows.net/dotnet/preview/Binaries/$DOTNET_SDK_VERSION/dotnet-dev-debian-x64.$DOTNET_SDK_VERSION.tar.gz
RUN curl -SL $DOTNET_SDK_DOWNLOAD_URL --output dotnet.tar.gz \
&& mkdir -p /usr/share/dotnet \
&& tar -zxf dotnet.tar.gz -C /usr/share/dotnet \
&& rm dotnet.tar.gz \
&& ln -s /usr/share/dotnet/dotnet /usr/bin/dotnet
# Trigger the population of the local package cache
ENV NUGET_XMLDOC_MODE skip
RUN mkdir warmup \
&& cd warmup \
&& dotnet new \
&& cd .. \
&& rm -rf warmup \
&& rm -rf /tmp/NuGetScratch
USER jenkins
I'm still looking for a better way to do it.
The Jenikins task should invoke dotnet commands to build. MSBuild is not yet supported for dotnet.
Basically, it has to do something similar to what we do in KoreBuild:
dotnet restore
dotnet build / dotnet publish
dotnet test
etc