I use inotify-tools and unison to synchronize folders between machines.
Because I have a large folder to synchronize, I just simply write an inotifywait script to do the job automatically.
Is it sensible to let inotifywait to monitor the subdirectories of the large folder to gain a better performance?
You should get better performance if you ditch inotify-tools and just use unison's native support for watching your folders for changes. By using inotify-tools and then calling unison when a change occurs, unison has to "re-find" the change before it syncs. You could instead add the line repeat = watch to your unison profile and unison will run continually and sync whenever there is a change. It detects the change with its own file-watcher utility unison-fsmonitor that communicates directly with unison.
For more information, check out the latest changelog for unison 2.48.3 with major changes to unison-fsmonitor.
The unison-fsmonitor is not provided by ubuntu package until now:
https://bugs.launchpad.net/ubuntu/+source/unison/+bug/1558737
https://github.com/bcpierce00/unison/issues/208
If you want it fast locally
UNISON_VERSION=2.51.2
echo "Install Unison." \
&& apt install wget ocaml
&& pushd /tmp \
&& wget https://github.com/bcpierce00/unison/archive/v$UNISON_VERSION.tar.gz \
&& tar -xzvf v$UNISON_VERSION.tar.gz \
&& rm v$UNISON_VERSION.tar.gz \
&& pushd unison-$UNISON_VERSION \
&& make \
&& cp -t /usr/local/bin ./src/unison ./src/unison-fsmonitor \
&& popd \
&& rm -rf unison-$UNISON_VERSION \
&& popd
Related
I'm using the following Dockerfile to install grpc on a build image, build a cpp microservice and put that into a runtime container.
https://github.com/npclaudiu/grpc-cpp-docker/blob/master/Dockerfile
But the part that builds grpc/protobuf takes 2hours+ and that is for one service.
RUN echo "-- installing protobuf" && \
cd /var/local/git/grpc/third_party/protobuf && \
./autogen.sh && ./configure --enable-shared && \
make -j$(nproc) && make -j$(nproc) check && make install && ldconfig
I see that all language implementations are compiled while I only need C++.
Is there any way to speed up this process? Are there debian or preferably alpine linux images with grpc and protocol buffers for c++ or all languages preinstalled?
You can create an intermediate image. Split your Dockefile into 2 parts after these lines:
RUN echo "-- installing grpc" && \
cd /var/local/git/grpc && \
make -j$(nproc) && make install && make clean && ldconfig
build the first one once and for all (you'll wait 2hours only once), tag the image locally and use this image as your base image for the 2nd Dockerfile.
I have tried to build a docker image and found that the PATH variable I set has some issues. A Minimal non-working example is:
FROM ubuntu:latest
SHELL ["/bin/bash", "-cu"]
ARG CTAGS_DIR=/root/tools/ctags
# Install common dev tools
RUN apt-get update --allow-unauthenticated \
&& apt-get install --allow-unauthenticated -y git curl autoconf pkg-config zsh
# Compile ctags
RUN cd /tmp \
&& git clone https://github.com/universal-ctags/ctags.git \
&& cd ctags \
&& ./autogen.sh \
&& ./configure --prefix=${CTAGS_DIR} \
&& make -j$(nproc) \
&& make install \
&& rm -rf /tmp/ctags
ENV PATH=$HOME/tools/ctags/bin:$PATH
RUN echo "PATH is $PATH"
RUN which ctags
In the above Dockerfile, the line ENV PATH=$HOME/tools/ctags/bin:$PATH does not work as expected. It seems that $HOME is not correctly expanded. The following two instructions also do not work:
ENV PATH=~/tools/ctags/bin:$PATH
ENV PATH="~/tools/ctags/bin:$PATH"
Only settings the absolute path works:
# the following setting works.
ENV PATH="/root/tools/ctags/bin:$PATH"
I have looked up the docker references but can not find document about this.
In general, when you're building a Docker image, it's okay to install things into the normal "system" directories. Whatever you're building will be isolated inside the image, and it can't conflict with other tools.
The easiest answer to your immediate question is to arrange things so you don't need to set $PATH.
In the example you give, you can safely use Autoconf's default installation directory of /usr/local. That will almost certainly be empty when you start your image build and only things you install will be there.
RUN ... \
&& ./configure \
&& make \
&& make install
(The Python corollary is to not create a virtual environment for your application; just use the system pip to install things into the default Python library directories.)
Don't expect there to be a home directory. If you have to install in some non-default place, /app is common, and /opt/whatever is consistent with non-Docker Linux practice. Avoid $HOME or ~, they aren't generally well-defined in Docker (unless you go out of your way to make them be).
I am a beginner to Docker and networking concepts. I have a dockerized application in my virtualbox. I am connecting to MySQL in another computer.
I was able to connect to it, then I was able to view the app, from my virtualbox using the --net=host option. From what I understand, this option maps both the docker and virtualbox machine's networks, and that is why I was able to see it in the browser in my virtualbox.
Should I be changing anything in my Dockerfile to make sure the connections work? How would I test it out after deployment whether or not this works? My current Dockerfile contains just RUN commands to install the necessary software.
My confusion arises like this. If say I deploy this (Azure) (beginner to deployment also), what port/address should I map/expose to make sure the app works after deployment also? Currently it is a flask app which is running on 127.0.0.1 address. But should I be changing anything in the Dockerfile?
Dockerfile
FROM python:3.6.3
COPY . /app
WORKDIR /app
RUN apt update && \
apt install -y gcc g++ gfortran git patch wget && \
apt install -y vim-tiny && \
pip install --upgrade pip && \
pip install -r requirements.txt && \
cd / && \
wget http://download.redis.io/releases/redis-5.0.5.tar.gz && \
tar xzf redis-5.0.5.tar.gz && \
cd redis-5.0.5 && \
make && \
# make test && \
cd / && \
rm redis-5.0.5.tar.gz && \
cd / && \
wget https://www.coin-or.org/download/source/Ipopt/Ipopt-3.12.13.tgz && \
tar xvzf Ipopt-3.12.13.tgz && \
cd Ipopt-3.12.13/ThirdParty/Blas/ && \
./get.Blas && \
cd ../Lapack && \
./get.Lapack && \
cd ../Mumps && \
./get.Mumps && \
cd ../Metis && \
./get.Metis && \
cd ../../ && \
mkdir build && \
cd build && \
../configure && \
make -j 4 && \
make install && \
cd / && \
rm Ipopt-3.12.13.tgz
Currently I run
sudo docker run -it --net=host 5h71 /bin/bash
python app.py # inside the container
### which gives
Running on http://127.0.0.1:8050/
Debugger PIN: 052-293-642
* Serving Flask app "server" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: on
Running on http://127.0.0.1:8050/
Debugger PIN: 671-290-156
Then I access 127.0.0.1:8050 to access the app on my chrome.
With respect to your first question:
How can I include my --net=host in my Dockerfile before I deploy my container?
You can't. Runtime options that impact your host (such as network attachments, volume mounting, etc) cannot be included in your Dockerfile. This is a security restriction: if I could trick you into running an image that somehow ran with --net=host by default, I could do all sorts of nasty things.
If you have an application that requires options like that in order to be useful, and you get tired of typing long docker run command lines (or you want it to be easier for someone else to use) you can use a tool like docker-compose and create a docker-compose.yml that specifies all the appropriate runtime options.
I am working on continuous integration of a .net project using jenkins. By far, i am able to set a jenkins job in window. But now i need to replicate this all in jenkins running as docker container. I am able to start jenkins in docker, using github as source repository but when i try to build this project, it fails. My project is using asp.net core so i am assuming it should run on linux as well (which is the OS of docker virtual machine).
What i am missing here? Any help is highly appreciated
I'm working on a project with .Net core and we started to use Jenkins in Docker container, so far the only way I found was to create a custom Jenkins image. This is my docker file:
FROM jenkins
USER root
# Work around https://github.com/dotnet/cli/issues/1582 until Docker releases a
# fix (https://github.com/docker/docker/issues/20818). This workaround allows
# the container to be run with the default seccomp Docker settings by avoiding
# the restart_syscall made by LTTng which causes a failed assertion.
ENV LTTNG_UST_REGISTER_TIMEOUT 0
# Install .NET CLI dependencies
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
libc6 \
libcurl3 \
libgcc1 \
libgssapi-krb5-2 \
libicu52 \
liblttng-ust0 \
libssl1.0.0 \
libstdc++6 \
libunwind8 \
libuuid1 \
zlib1g \
&& rm -rf /var/lib/apt/lists/*
# Install .NET Core SDK
ENV DOTNET_SDK_VERSION 1.0.0-preview2-003131
ENV DOTNET_SDK_DOWNLOAD_URL https://dotnetcli.blob.core.windows.net/dotnet/preview/Binaries/$DOTNET_SDK_VERSION/dotnet-dev-debian-x64.$DOTNET_SDK_VERSION.tar.gz
RUN curl -SL $DOTNET_SDK_DOWNLOAD_URL --output dotnet.tar.gz \
&& mkdir -p /usr/share/dotnet \
&& tar -zxf dotnet.tar.gz -C /usr/share/dotnet \
&& rm dotnet.tar.gz \
&& ln -s /usr/share/dotnet/dotnet /usr/bin/dotnet
# Trigger the population of the local package cache
ENV NUGET_XMLDOC_MODE skip
RUN mkdir warmup \
&& cd warmup \
&& dotnet new \
&& cd .. \
&& rm -rf warmup \
&& rm -rf /tmp/NuGetScratch
USER jenkins
I'm still looking for a better way to do it.
The Jenikins task should invoke dotnet commands to build. MSBuild is not yet supported for dotnet.
Basically, it has to do something similar to what we do in KoreBuild:
dotnet restore
dotnet build / dotnet publish
dotnet test
etc
It's generally understood that if the RUN instruction cleans up after large file operations, the resulting layer size would be smaller. However when I execute the following script the resulting layer is large and roughly corresponds to the size of the files initially copied and cleaned.
How could this size of a layer be committed if the files copied are cleaned in the same layer?
RUN mkdir -p /etc/puppet && \
cd /etc/puppet && \
apt-get update && apt-get install -y wget puppet && \
wget -rnH --level=10 -e robots=off --reject "index.html*" http://somefileserver.org/puppet/allpuppetstuff && \
puppet module install puppetlabs/stdlib && \
puppet module install 7terminals-java && \
puppet apply -e "include someserver" && \
apt-get purge -y --auto-remove puppet wget && \
rm -rfv /etc/puppet/* && \
rm -rfv /var/lib/apt/lists/*
Found some details after few tweaks.
It seems docker duplicates on old layer, if a newer layer is modifying files persisted on the earlier one. This is the cause of the fat layer created by the above RUN command, even when the total file size diff at the end of operation is minimal. I couldn't find any definite resources to cite this, but this is the experience so far.
Basically, file operations related to the image should always be done in one layer.