Problem Statement
I am building a docker of my computational bioinformatics pipeline which contains many tools that will be called at different steps of pipelines. In this process, I am trying to add one tool The ViennaRNA Package which will be downloaded and compliled using source code. I have tried many ways to compile it in docker build (as shown below) but none of them is working.
Failed attempts
Code-1 :
FROM jupyter/scipy-notebook
USER root
MAINTAINER Vivek Ruhela <vivekr#iiitd.ac.in>
# Copy the application folder inside the container
ADD . /test1
# Set the default directory where CMD will execute
WORKDIR /test1
# Set environment variable
ENV HOME /test1
# Install RNAFold
RUN wget https://www.tbi.univie.ac.at/RNA/download/sourcecode/2_4_x/ViennaRNA-2.4.14.tar.gz -P ~/Tools
RUN tar xvzf ~/Tools/ViennaRNA-2.4.14.tar.gz -C ~/Tools
WORKDIR "~/Tools/ViennaRNA-2.4.14/"
RUN ./configure
RUN make && make check && make install
Error : configure file not found
Code-2 :
FROM jupyter/scipy-notebook
USER root
MAINTAINER Vivek Ruhela <vivekr#iiitd.ac.in>
# Copy the application folder inside the container
ADD . /test1
# Set the default directory where CMD will execute
WORKDIR /test1
# Set environment variable
ENV HOME /test1
# Install RNAFold
RUN wget https://www.tbi.univie.ac.at/RNA/download/sourcecode/2_4_x/ViennaRNA-2.4.14.tar.gz -P ~/Tools
RUN tar xvzf ~/Tools/ViennaRNA-2.4.14.tar.gz -C ~/Tools
RUN bash ~/Tools/ViennaRNA-2.4.14/configure
WORKDIR "~/Tools/ViennaRNA-2.4.14/"
RUN make && make check && make install
Error : make: *** No targets specified and no makefile found. Stop.
I also tried another way to tell the file location explicitly e.g.
RUN make -C ~/Tools/ViennaRNA-2.4.14/
Sill this approach is not working.
Expected Procedure
I have installed this tool in my system many times using the standard procedure as mentioned in tool documentation as
./configure
make
make check
make install
Similarly for docker, the following code should work
WORKDIR ~/Tools/ViennaRNA-2.4.14/
RUN ./configure && make && make check && make install
But this code is not working because I don't see any effect of workdir. I have checked that configure is creating makefile properly in my system. So it should create the make file in docker also.
Any suggestions on why this code is not working.
you are extract all the files in Tools folder which is in home ,try this:
WORKDIR $HOME/Tools/ViennaRNA-2.4.14
RUN ./configure
RUN make && make check && make install
the problem is WORKDIR ~/Tools/ViennaRNA-2.4.14/ is translated to exactly ~/Tools/ViennaRNA-2.4.14/ which is created a folder named ~ , you may also use $HOME instead
Related
I would like to say that this is my first container and actually my first JAVA app so maybe I will have basic questions so be lenient, please.
I wrote spring boot app and my colleague has written the frontend part for it in angular. What I would like to achieve is to have "one button/one command" in IntelliJ to create a container containing whole app backend and front end.
What I need to do is:
Clone FE from company repository (I am using ssh key now)
Clone BE from GitHub
Build FE
Copy built FE to static folder in java app
Build BE
Create a container running this app
My current solution is to create "builder" container and there build FE and BE and then copy it to "production" container like this:
#BUILDER
FROM alpine AS builder
WORKDIR /src
# add credentials on build
ARG SSH_PRIVATE_KEY
RUN mkdir /root/.ssh/ \
&& echo "${SSH_PRIVATE_KEY}" > /root/.ssh/id_rsa \
&& echo "github.com,140.82.121.3 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==" >> /root/.ssh/known_hosts \
&& chmod 600 /root/.ssh/id_rsa
# installing dependencies
RUN apk update && apk upgrade && apk add --update nodejs nodejs-npm \
&& npm install -g #angular/cli \
&& apk add openjdk11 \
&& apk add maven \
&& apk add --no-cache openssh \
&& apk add --no-cache git
#cloning repositories
RUN git clone git#code.siemens.com:apcprague/edge/metal-forming-fe.git
RUN git clone git#github.com:bzumik1/metalForming.git
# builds front end
WORKDIR /src/metal-forming-fe
RUN npm install && ng build
# builds whole java app with front end
WORKDIR /src/metalForming
RUN cp -a /src/metal-forming-fe/dist/metal-forming/. /src/metalForming/src/main/resources/static \
&& mvn install -DskipTests=true
#PRODUCTION CONTAINER
FROM adoptopenjdk/openjdk11:debian-slim
LABEL maintainer jakub.znamenacek#siemens.com
RUN mkdir app
RUN ["chmod", "+rwx", "/app"]
WORKDIR /app
COPY --from=builder /src/metalForming/target/metal_forming-0.0.1-SNAPSHOT.jar .
EXPOSE 4200
RUN java -version
CMD java -jar metal_forming-0.0.1-SNAPSHOT.jar
This works but I takes very long time so I guess this is not correct way how to do it. Could anyone point me in correct direction? I was thinking if there is a way how to make maven to all these steps for me but maybe this is totally off.
Also if you will find any problem in my Dockerfile please let me know as I said this is my first Dockerfile so I could overlook something.
EDITED:
BTW does anyone know how can I get rid of this part:
echo "github.com,140.82.121.3 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==" >> /root/.ssh/known_hosts \
it adds GitHub to known_hosts (I also need to add a company repository there). It is because when I run git clone it will ask me if I trust this ... and I have to write yes but I don't know how to do it if it is automatically running in docker and I have no option to write there yes. I have tried yes | git clone ... but this is also not working
a few things:
1, if this runs "slow" on the machine than it will run slow inside a container too.
2, remove --no-cache,* you want to cache everything that is static, because next time when you build those commands will not run where there is no change. Once there is change in one command than that command will rerun instead using the builder cache and also all subsequent commands will have to rerun too.
*UPDATE: I have mistaken "apk update --no-cache" with "docker build --no-cache". I stated wrong that "apk add --no-cache" would mean that command is not cached, because this command is cached on docker builder level. However with this parameter you wouldn't need to delete in a later step the /var/cache/apk/ directory to make you image smaller, but that you wouldn't need to do here, because you are already using multi stage build, so this would not affect your final image size.
One more thing to clarify, all statements in Dockerfile are checked if they changed, if they did not than docker builder uses the cached layer for it and won't run that statement. Exception is ADD and COPY commands, here builder also checks the copied, added files if they changed with checksum. Also if a statement is changed or ADD-ed COPY-ed file(s) changed than that statement is re-run and all subsequent statements re-run too, so you want to put your source code copy statemant as much at the end as it is possible
If you want to disable this cache, do "docker build --no-cache ..." this way all the steps will be re-run that is in the Dockerfile.
3, specify WORKDIR at the top once, if you need to switch directory later use this:
RUN cd /someotherdir && mycommand
Also specifying a Subsequent WORKDIR will be relativ to the previous WORKDIR so it will mess up readibilty what is the (probably) sole purpose of WORKDIR statement.
4, Enable BuildKit:
Either declare environment variable
DOCKER_BUILDKIT=1
or add this to /etc/docker/daemon.json
{ "features": { "buildkit": true } }
BuildKit might not help in this case, but if you do more complex Dockerfiles with more stages Buildkit can run those parallel so overall build will be faster.
5, Do not skip tests with DskipTests=true :)
6, as stated in a comment, do not clone the repo inside the image build, you do not need to do that at all. Just put the Dockerfile in the / of the repo, and COPY the repo files with a Dockerfile command:
COPY . .
First dot is the source that is your current directory on your machine, second dot is the target, the working dir inside the image, /src with your Dockerfile. You build the image and publish it, push it to a docker registry so others can pull it and start using it. If you want more complex stuff building and publishing with a help of a server, look up CI/CD techniques.
I have a Go app. Some of its dependencies are in a private Github repo and another part of dependencies are local packages in my app folder. The app compiles and works on my computer without a problem (when I simply compile it without docker). I am using the below Dockerfile.
FROM ubuntu as intermediate
# install git
RUN apt-get update
RUN apt-get install -y git
RUN mkdir /root/.ssh/
COPY github_rsa.ppk /root/.ssh/github_rsa.ppk
RUN chmod 700 /root/.ssh/github_rsa.ppk
RUN eval $(ssh-agent) && \
ssh-add /root/.ssh/github_rsa.ppk && \
ssh-keyscan -H github.com >> /etc/ssh/ssh_known_hosts && \
git clone git#github.myusername/shared.git
FROM golang:latest
ENV GOPATH=/go
RUN echo $GOPATH
ADD . /go/src/SCMicroServer
WORKDIR /go/src/SCMicroServer
COPY --from=intermediate /shared /go/src/github.com/myusername/shared
RUN go get /go/src/SCMicroServer
RUN go install SCMicroServer
ENTRYPOINT /go/src/SCMicroServer
EXPOSE 8080
First build section related to Git is working fine, It works until this line: RUN go get /go/src/SCMicroServer in second section. I receive this error in mentioned step:
package SCMicroServer/controllers/package1: unrecognized import path "SCMicroServer/controllers/package1" (import path does not begin with hostname)
The command '/bin/sh -c go get /go/src/SCMicroServer' returned a non-zero code: 1
"SCMicroServer/controllers/package1" is one of the local packages in my app folder (or its subfolders) and I have many more in my local folder. I am setting GOPATH env variable in my Dockerfile, so I am not sure what I am missing.
I found the answer, it was not really Dockerfile problem, I referenced my package 2 times in 2 different way in my main file:
package1 "SCMicroServer/controllers/package1"
"SCMicroServer/controllers/package1"
After I removed the second one, I stopped receiving the error.
I'm trying to build a Docker container for this module. Main idea is that I'm trying to use, as much as possible, the packages provided by Ubuntu to avoid problem; I'm also using the default perl that comes with the container, which apparently is 5.22.
This is the Dockerfile:
FROM ubuntu:16.04
LABEL version="1.0" maintainer="JJ Merelo <jjmerelo#GMail.com>" perl5version="5.22"
ADD data/* ./
ADD . .
RUN mkdir /test \
&& apt-get update \
&& apt-get install -y build-essential curl hunspell-en-us libtext-hunspell-perl myspell-es libencode-perl cpanminus libfile-slurp-tiny-perl libversion-perl\
&& curl https://raw.githubusercontent.com/SublimeText/Dictionaries/master/Spanish.dic -o Spanish.dic
RUN cpanm .
RUN perl --version
VOLUME /test
WORKDIR /test
# Will run this
ENTRYPOINT prove
It builds locally without a problem (using Docker version 17.05.0-ce, build 89658be). However, it does not work in Docker hub due to this problem
/etc/ssl/certs/AddTrust_Low-Value_Services_Root.pem is encountered a second time at /usr/share/perl/5.22/File/Find.pm line 79.
Which seems to happen in a line where I do perl Makefile.PL && make install
I really have no idea what could be the cause for that, of why it works locally and fails there. Any idea?
I think the problem is that you do WORKDIR /test too late. In the beginning of Dockerfile you do ADD . . this will copy all files from the current directory of your local filesystem to the root directory of the image. The problem is that there can be conflicts between the directories already present in the root of the image, like /lib. Try instead something like this:
FROM ubuntu:16.04
LABEL version="1.0" maintainer="JJ Merelo <jjmerelo#GMail.com>" perl5version="5.22"
WORKDIR /test
ADD data/* ./
ADD . .
RUN apt-get update \
&& apt-get install -y build-essential curl hunspell-en-us libtext-hunspell-perl myspell-es libencode-perl cpanminus libfile-slurp-tiny-perl libversion-perl\
&& curl https://raw.githubusercontent.com/SublimeText/Dictionaries/master/Spanish.dic -o Spanish.dic
RUN perl --version
RUN cpanm Test::More
RUN cpanm .
VOLUME /test
# Will run this
ENTRYPOINT prove
The problem is that we don't use a working directory for copying the files we want to have included in the directory, and then we do ADD . .. That includes a lib directory, so we are then copying the contents of our module to the lib directory, that is, the image /lib directory.
That's not bad per se. The problem is that the Perl installation procedure is then copying the contents of its own lib directory (which is the system's lib directory) onto blib, and then again, copyint that to some directory of /usr/lib. Somehow there's installation happening in other directories, which means that that certificate mentioned in the error is eventually copied somewhere else, hence the duplicate.
That does not explain why it was working locally, and as a matter of fact working in subsequent builds in Docker Hub. Maybe some slightly different version of runc. But even if it works, you end up with stuff duplicated, and a bigger image than you should have.
Baseline is: don't use root as working directory. Use WORKDIR from the very beginning, even more so if that directory is going to have stuff that's going to be installed somewhere else.
I have created a docker file in which I have installed golang dep tool which will be used to install the dependencies required by golang project. I have been able to install the tool. But unable to install the dependencies using that tool. I am not sure how to configure dep tool to be able to run dep command in docker image which will install all the dependencies required by golang project
I am using below command to run dep tool and it works in local machine
# initialize the project and install dependencies
RUN dep init
I am always getting an error:
init failed: unable to determine the import path for the root project
/go: /go is not within any GOPATH/src
Now I do not know if I need to set path to o binary files or how I can achieve that. There are tutorials to build a docker file to build golang project but nothing is there on internet to install dependencies using golang dep tool.
Here is an example of Dockerfile with dep:
FROM golang:latest
LABEL version="1.0"
RUN mkdir /go/src/app
RUN go get -u github.com/golang/dep/cmd/dep
ADD ./main.go /go/src/app
COPY ./Gopkg.toml /go/src/app
WORKDIR /go/src/app
RUN dep ensure
RUN go test -v
RUN go build
CMD ["./app"]
You need to change the directory to that of your project. Also, in order to get dependencies, you will usually already have a Gopkg.toml and Gopkg.lock - dep init is ONLY used when you're moving from a project which was using another vendoring tool, no vendoring at all or you're starting a project from scratch.
To sum up, I'd do something like this:
FROM golang:latest
RUN go get -u github.com/golang/dep/cmd/dep \
&& mkdir /go/src/github.com/you \
&& git clone https://github.com/you/yourproject /go/src/github.com/you/yourproject
WORKDIR /go/src/github.com/you/yourproject
RUN dep ensure -v
&& go build
CMD ["./yourproject"]
I am trying to add Glide to my Golang project but I'm not getting my container working. I am currently using:
# create image from the official Go image
FROM golang:alpine
RUN apk add --update tzdata bash wget curl git;
# Create binary directory, install glide and fresh
RUN mkdir -p $$GOPATH/bin
RUN curl https://glide.sh/get | sh
RUN go get github.com/pilu/fresh
# define work directory
ADD . /go
WORKDIR /go/src
RUN glide update && fresh -c ../runner.conf main.go
as per #craigchilds94's post. When I run
docker build -t docker_test .
It all works. However, when I change the last line from RUN glide ... to CMD glide ... and then start the container with:
docker run -it --volume=$(PWD):/go docker_test
It gives me an error: /bin/sh: glide: not found. Ignoring the glide update and directly starting fresh results in the same: /bin/sh fresh: not found.
The end goal is to be able to mount a volume (for the live-reload) and be able to use it in docker-compose so I want to be able to build it, but I do not understand what is going wrong.
This should probably work for your purposes:
# create image from the official Go image
FROM golang:alpine
RUN apk add --update tzdata bash wget curl git;
# Create binary directory, install glide and fresh
RUN go get -u github.com/Masterminds/glide
RUN go get -u github.com/pilu/fresh
# define work directory
ADD . /go
WORKDIR /go/src
ENTRYPOINT $GOPATH/bin/fresh -c /go/src/runner.conf /go/src/main.go
As far as I know you don't need to run the glide update after you've just installed glide. You can check this Dockerfile I wrote that uses glide:
https://github.com/timogoosen/dockerfiles/blob/master/btcd/Dockerfile
and here is the REAMDE: https://github.com/timogoosen/dockerfiles/blob/master/btcd/README.md
This article gives a good overview of the difference between: CMD, RUN and entrypoint: http://goinbigdata.com/docker-run-vs-cmd-vs-entrypoint/
To quote from the article:
"RUN executes command(s) in a new layer and creates a new image. E.g., it is often used for installing software packages."
In my opinion installing packages and libraries can happen with RUN.
For starting your binary or commands I would suggest use ENTRYPOINT see:"ENTRYPOINT configures a container that will run as an executable." you could use CMD too for running:
$GOPATH/bin/fresh -c /go/src/runner.conf /go/src/main.go
something like this might work, I didn't test this part:
CMD ["$GOPATH/bin/fresh", "-c", "/go/src/runner.conf /go/src/main.go"]