Docker supports the following syntax for mounting a tmpfs at a particular path while performing a RUN step as part of a container image build:
RUN --mount=type=tmpfs,target=/build cd /build && cmake /src && cmake --build .
See: https://docs.docker.com/engine/reference/builder/#run---mounttypetmpfs
If I'm building on a system with too little memory, this might cause an issue. Can I conditionally disable the use of tmpfs mount?
It does not work to pass it as a build argument:
ARG TMPFS_MOUNTS=--mount=type=tmpfs,target=/build
RUN $TMPFS_MOUNTS ls
> [2/2] RUN --mount=type=tmpfs,target=/build ls:
#5 0.168 /bin/sh: 1: --mount=type=tmpfs,target=/build: not found
One way is to mount the tmpfs unconditionally, but use it conditionally by creating a symlink depending on the value of a build argument.
ARG USE_TMPFS=true
RUN --mount=type=tmpfs,target=/tmpfs \
([ "$USE_TMPFS" = "true" ] && \
ln -s /tmpfs /build || \
mkdir /build) && \
cd /build && cmake /src && cmake --build .
I could not get a bind mount to work.
Related
I have built a local uaa docker image and tried to run in local.
But I am getting this error when I am trying to start the docker image.
I built the docker image via this below command and the build is successful too.
docker build -t uaa-local --build-arg uaa_yml_name=local.yml .
when I am trying to run the local uaa docker image, I am getting this below error. What I am doing wrong
Content of DockerFile
FROM openjdk:11-jre
ARG uaa_yml_name=local.yml
ENV UAA_CONFIG_PATH /uaa
ENV CATALINA_HOME /tomcat
ADD run.sh /tmp/
ADD conf/$uaa_yml_name /uaa/uaa.yml
RUN chmod +x /tmp/run.sh
RUN wget -q https://archive.apache.org/dist/tomcat/tomcat-8/v8.5.57/bin/apache-tomcat-8.5.57.tar.gz
RUN tar zxf apache-tomcat-8.5.57.tar.gz
RUN rm apache-tomcat-8.5.57.tar.gz
RUN mkdir /tomcat
RUN mv apache-tomcat-8.5.57/* /tomcat
RUN rm -rf /tomcat/webapps/*
ADD dist/cloudfoundry-identity-uaa-74.22.0.war /tomcat/webapps/
RUN mv /tomcat/webapps/cloudfoundry-identity-uaa-74.22.0.war /tomcat/webapps/ROOT.war
RUN mkdir -p /tomcat/webapps/ROOT && cd /tomcat/webapps/ROOT && unzip ../ROOT.war
ADD conf/log4j2.properties /tomcat/webapps/ROOT/WEB-INF/classes/log4j2.properties
RUN rm -rf /tomcat/webapps/ROOT.war
EXPOSE 8080
CMD ["/tmp/run.sh"]
On further investigation I think it is looking for run.sh file in the /tmp/ folder which is added on line 5 in Dockerfile..but when I checked for the file in /tmp/ folder it is not there..Is it because of that?And how to resolve that? I already have the run.sh in my current folder.
I had built a Docker container from this Dockerfile previously and it worked fine:
FROM perl:5.32
MAINTAINER Matthew Jordan Oldach, moldach686#gmail.com
WORKDIR /usr/local/bin
# Install cpan modules
RUN cpanm install --force Cwd Getopt::Long POSIX File::Basename List::Util Bio::DB::Fasta Bio::Seq Bio::SeqUtils Bio::SeqIO Set::IntervalTree Set::IntSpan
RUN apt-get install tar
# Download CooVar-v0.07
RUN wget http://genome.sfu.ca/projects/coovar/CooVar-0.07.tar.gz
RUN tar xvf CooVar-0.07.tar.gz
RUN cd coovar-0.07; chmod +x scripts/* coovar.pl
# Set WORKDIR to /data -- predefined mount location.
RUN mkdir /data
WORKDIR /data
# Set Entrypoint
ENTRYPOINT ["perl", "/usr/local/bin/coovar-0.07/coovar.pl"]
The only issue was that I found there was a slight difference between what is on the repo and the coovar-0.07 which is on our server (there was slight difference in the extract-cdna.pl script).
In order to reproduce our pipeline I'll need to COPY CooVar locally into the container (rather than wget).
I've therefore tried the following Dockerfile:
FROM perl:5.32
MAINTAINER Matthew Jordan Oldach, moldach686#gmail.com
WORKDIR /usr/local/bin
# Install cpan modules
RUN cpanm install --force Cwd Getopt::Long POSIX File::Basename List::Util Bio::DB::Fasta Bio::Seq Bio::SeqUtils Bio::SeqIO Set::IntervalTree Set::IntSpan
# Download CooVar-v0.07
COPY coovar-0.07 /usr/local/bin/coovar-0.07
RUN cd coovar-0.07; chmod +x scripts/* coovar.pl
# Set WORKDIR to /data -- predefined mount location.
RUN mkdir /data
WORKDIR /data
# Set Entrypoint
ENTRYPOINT ["perl", "/usr/local/bin/coovar-0.07/coovar.pl"]
It appears I could run the main script (coovar.pl) from Docker (no Permission Denied error):
# pull the container
$ sudo docker pull moldach686/coovar-v0.07:latest
# force entry point of `moldach686/coovar-v0.07` to /bin/bash
## in order to investigate file system
$ sudo docker run -it --entrypoint /bin/bash moldach686/coovar-v0.07
root#c7459dbe216a:/data# perl /usr/local/bin/coovar-0.07/coovar.pl
USAGE: ./coovar.pl -e EXONS_GFF -r REFERENCE_FASTA (-t GVS_TAB_FORMAT | -v GVS_VCF_FORMAT) [-o OUTPUT_DIRECTORY] [--circos] [--feature_source] [--feature_type]
Program parameter details provided in file README.
However, when I tried to incorporate this into my Snakemake workflow I get the following Permission Denied error:
Workflow defines that rule get_vep_cache is eligible for caching between workflows (use the --cache argument to enable this).
Building DAG of jobs...
Using shell: /cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job counts:
count jobs
1 coovar
1
[Tue Nov 3 21:56:51 2020]
rule coovar:
input: variant_calling/varscan/MTG470.vcf, refs/c_elegans.PRJNA13758.WS265.genomic.fa
output: annotation/coovar/varscan/MTG470/categorized-gvs.gvf, annotation/coovar/varscan/MTG470.annotated.vcf, annotation/coovar/varscan/filtration/MTG470_keep.tsv, annotation/coovar/varscan/filtration/MTG470_exclude.tsv
jobid: 0
wildcards: sample=MTG470
resources: mem=4000, time=10
Activating singularity image /scratch/moldach/COOVAR/cbc22e3a26af1c31fb0e4fcae240baf8.simg
Can't open perl script "/usr/local/bin/coovar-0.07/coovar.pl": Permission denied
The solution I found to work was adding the following line to the Dockerfile:
RUN echo "user ALL=NOPASSWD: ALL" >> /etc/sudoers
This adds the user to the sudoers file giving permissions:
FROM perl:5.32
MAINTAINER Matthew Jordan Oldach, moldach686#gmail.com
USER root
WORKDIR /usr/local/bin
# Install cpan modules
RUN cpanm install --force Cwd Getopt::Long POSIX File::Basename List::Util Bio::DB::Fasta Bio::Seq Bio::SeqUtils Bio::SeqIO Set::IntervalTree Set::IntSpan
RUN echo "user ALL=NOPASSWD: ALL" >> /etc/sudoers
# Download CooVar-v0.07
COPY coovar-0.07 /usr/local/bin/coovar-0.07
RUN cd coovar-0.07; chmod a+rwx scripts/* coovar.pl
# Download Bedtools 2.27.1
ENV VERSION 2.27.1
ENV NAME bedtools2
ENV URL "https://github.com/arq5x/bedtools2/releases/download/v${VERSION}/bedtools-${VERSION}.tar.gz"
WORKDIR /tmp
RUN wget -q -O - $URL | tar -zxv && \
cd ${NAME} && \
make -j 4 && \
cd .. && \
cp ./${NAME}/bin/bedtools /usr/local/bin/ && \
strip /usr/local/bin/*; true && \
rm -rf ./${NAME}/
# Set WORKDIR to /data -- predefined mount location.
RUN mkdir /data
WORKDIR /data
# Set Entrypoint
ENTRYPOINT ["perl", "/usr/local/bin/coovar-0.07/coovar.pl"]
Using the docker build command line I can pass in a build secret as follows
docker build \
--secret=id=gradle.properties,src=$HOME/.gradle/gradle.properties \
--build-arg project=template-ms \
.
Then use it in a Dockerfile
# syntax = docker/dockerfile:1.0-experimental
FROM gradle:jdk12 AS build
COPY *.gradle .
RUN --mount=type=secret,target=/home/gradle/gradle.properties,id=gradle.properties gradle dependencies
COPY src/ src/
RUN --mount=type=secret,target=/home/gradle/gradle.properties,id=gradle.properties gradle build
RUN ls -lR build
FROM alpine AS unpacker
ARG project
COPY --from=build /home/gradle/build/libs/${project}.jar /tmp
RUN mkdir -p /opt/ms && unzip -q /tmp/${project}.jar -d /opt/ms && \
mv /opt/ms/BOOT-INF/lib /opt/lib
FROM openjdk:12
EXPOSE 8080
WORKDIR /opt/ms
USER nobody
CMD ["java", "-Xdebug", "-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=0.0.0.0:8000", "-Dnetworkaddress.cache.ttl=5", "org.springframework.boot.loader.JarLauncher"]
HEALTHCHECK --start-period=600s CMD curl --silent --output /dev/null http://localhost:8080/actuator/health
COPY --from=unpacker /opt/lib /opt/ms/BOOT-INF/lib
COPY --from=unpacker /opt/ms/ /opt/ms/
I want to do a build using docker-compose, but I can't find in the docker-compose.yml reference how to pass the secret.
That way the developer just needs to type in docker-compose up
You can use environment or args to pass variables to container in docker-compose.
args:
- secret=id=gradle.properties,src=$HOME/.gradle/gradle.properties
environment:
- secret=id=gradle.properties,src=$HOME/.gradle/gradle.properties
Docker COPY is not copying over the bash script
FROM alpine:latest
#Install Go and Tini - These remain.
RUN apk add --no-cache go build-base gcc go
RUN apk add --no-cache --update ca-certificates redis git && update-ca-certificates
# Set Env Variables for Go and add Go to Path.
ENV GOPATH /go
ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH
RUN go get github.com/rakyll/hey
RUN echo GOLANG VERSION `go version`
COPY ./bench.sh /root/bench.sh
RUN chmod +x /root/bench.sh
ENTRYPOINT /root/bench.sh
Here is the script -
#!/bin/bash
set -e;
echo "entered";
hey;
I try running the above Dockerfile with
$ docker build -t test-bench .
$ docker run -it test-bench
But I get the error
/bin/sh: /root/bench.sh: not found
The file does exist -
$ docker run --rm -it test-bench sh
/ # ls
bin dev etc go home lib media mnt opt proc root run sbin srv sys tmp usr var
/ # cd root
~ # ls
bench.sh
~ #
Is your docker build successful. When I tried to simulate this, found the following error
---> Running in 96468658cebd
go: missing Git command. See https://golang.org/s/gogetcmd
package github.com/rakyll/hey: exec: "git": executable file not found in $PATH
The command '/bin/sh -c go get github.com/rakyll/hey' returned a non-zero code: 1
Try installing git using Dockerfile RUN apk add --no-cache go build-base gcc go git and run again.
The COPY operation here seems to be correct. Make sure it is present in the directory from where docker build is executed.
Okay, the script is using /bin/bash the bash binary is not available in the alpine image. Either it has to be installed or a /bin/sh shell should be used
I want to create a docker image using either git sources, or the already build app. I created two Dockerfiles like these (note: this is pseudo code):
Runtime-Image:
FROM <baseimage>
EXPOSE 1234/tcp
EXPOSE 4321/tcp
VOLUME /foobar
COPY myapp.tgz .
RUN tar -xzf myapp.tgz && rm -f myapp.tgz
ENTRYPOINT ["myapp"]
myapp.tgz is created on a buildserver or maybe by compiling manually. It is available on the docker host server locally.
To build directly from source I use:
FROM <devimage> AS buildenv
ARG GIT_USER
ARG GIT_PASSWORD
RUN git clone http://${GIT_USER}:${GIT_PASSWORD}#<my.git.host>
RUN ./makefile && cp /source/build/myapp.tgz /drop/myapp.tgz
FROM <baseimage> AS runenv
EXPOSE 1234/tcp
EXPOSE 4321/tcp
VOLUME /foobar
COPY --from=buildenv /drop/myapp.tgz .
RUN tar -xzf myapp.tgz && rm -f myapp.tgz
ENTRYPOINT ["myapp"]
The instructions in the second build stage of this are obviously a duplicate of the Runtime-Image Dockerfile.
I'd like to have just ONE Dockerfile, which can build from source, or from context on the docker host, as required. I could put the duplicated commands in a custom baseimage and reuse that to build onto (FROM), but this would obfuscate the Dockerfile.
What is the recommended, most elegant way to do this?
I can't use a bind mount to get myapp.tgz in the current directory on the docker host, can I? For this I would have to start a Container to build my app?
There is no IF directive in the Dockfile for conditions?
If there is no myapp.tgz on the docker host, COPY myapp.tgz . will fail
If there is no buildenv, COPY --from=buildenv /drop/myapp.tgz . will fail.
I could use COPY ./* . and then check with
[ -f /myapp.tgz ] && <prepare-container> || <build-from-git-source>
I guess? Our would you rather just create a seperate Dockerfile just for building from source and then use something like
docker run --rm -v /SomewhereOnHost/drop:/drop my-compile-image
For the past 2 days I have been trying to figure this out, now I have a good solution to achieve a conditional build (a if in Dockerfile)
ARG mode=local
FROM alpine as build_local
ONBUILD COPY myapp.tgz .
FROM alpine as build_remote
ONBUILD RUN git clone GIT_URL
ONBUILD RUN cd repo && ./makefile && cp /source/build/myapp.tgz .
FROM build_${mode} AS runenv
EXPOSE 1234/tcp
EXPOSE 4321/tcp
VOLUME /foobar
RUN tar -xzf myapp.tgz && rm -f myapp.tgz
ENTRYPOINT ["myapp"]
The toplevel mode allows you to pass the condition with docker build --build-arg mode=remote .. ONBUILD is used so the command is only executed if the corresponding branch is selected.