I have a Dockerfile that looks like the following.
FROM ubuntu:20.04
RUN apt-get update && \
rm -rf /var/lib/apt/lists/*
ADD part_aa /data/
ADD part_ab /data/
ADD part_ac /data/
ADD part_ad /data/
ADD part_ae /data/
ADD part_af /data/
ADD part_ag /data/
CMD entrypoint.sh
The Dockerfile adds some files into a directory called data, and at the end entrypoint.sh merges together the files that are in the data directory into a single file.
How can I mount a volume so I can move the final output file into my local machine?
I know that I can use the -v flag (volume), but I cannot figure out how to incorporate it into running the the image.
The entrypoint.sh looks like this:
cd /data
MODEL_FILE="merged_file"
if [ ! -f "$MODEL_FILE" ]; then
echo "combining model file parts."
cat part_* > $MODEL_FILE
echo "combining model file parts done"
fi
A Dockerfile is used to build a docker image. The -v switch is applicable when you run an image... you are mixing two things: building and running a docker image.
Related
I want to decrease the number of layers used in my Dockerfile.
So I want to combine the COPY commands in a RUN cp.
dependencies
folder1
file1
file2
Dockerfile
The following below commands work which I want to combine using a single RUN cp command
COPY ./dependencies/file1 /root/.m2
COPY ./dependencies/file2 /root/.sbt/
COPY ./dependencies/folder1 /root/.ivy2/cache
This following below command says No such file or directory present error. Where could I be going wrong ?
RUN cp ./dependencies/file1 /root/.m2 && \
cp ./dependencies/file2 /root/.sbt/ && \
cp ./dependencies/folder1 /root/.ivy2/cache
You can't do that.
COPY copies from the host to the image.
RUN cp copies from a location in the image to another location in the image.
To get it all into a single COPY statement, you can create the file structure you want on the host and then use tar to make it a single file. Then when you COPY or ADD that tar file, Docker will unpack it and put the files in the correct place. But with the current structure your files have on the host, it's not possible to do in a single COPY command.
Problem
The COPY is used to copy files from your host to your container. So, when you run
COPY ./dependencies/file1 /root/.m2
COPY ./dependencies/file2 /root/.sbt/
COPY ./dependencies/folder1 /root/.ivy2/cache
Docker will look for file1, file2, and folder1 on your host.
However, when you do it with RUN, the commands are executed inside the container, and ./dependencies/file1 (and so on) does not exist in your container yet, which leads to file not found error.
In short, COPY and RUN are not interchangeable.
How to fix
If you don't want to use multiple COPY commands, you can use one COPY to copy all files from your host to your container, then use the RUN command to move them to the proper location.
To avoid copying unnecessary files, use .dockerignore. For example:
.dockerignore
./dependencies/no-need-file
./dependencies/no-need-directory/
Dockerfile
COPY ./dependencies/ /root/
RUN mv ./dependencies/file1 /root/.m2 && \
mv ./dependencies/file2 /root/.sbt/ && \
mv ./dependencies/folder1 /root/.ivy2/cache
You a re missing final slash in /root/.ivy2/cache/
I have a script that creates several files starting with part_, and I would like to create a for loop inside the Dockerfile to search the directory for files starting with part_ and ADD them to the docker image. It is important to ADD each file as a separate layer (and not using wildcards like ADD part_* /directory/). How can I dynamically add files to a docker image?
I would like to use something like the RUN for f in part_*; do ADD $f /data/; done, but I do not know how I can incorporate ADD inside the bash script.
Here is an example of the Dockerfile.
FROM ubuntu:20.04
RUN apt-get update && \
rm -rf /var/lib/apt/lists/*
ADD part_aa /directory/
ADD part_ab /directory/
ADD part_ac /directory/
ADD part_ad /directory/
ADD part_ae /directory/
ADD part_af /directory/
COPY entrypoint.sh /usr/local/bin/
CMD entrypoint.sh
You can use scripting to create your Dockerfile;
echo "FROM ubuntu:20.04" > Dockerfile
echo "RUN apt-get update && \
rm -rf /var/lib/apt/lists/*" >> Dockerfile
for f in part_*; do echo "ADD $f /data/" >> Dockerfile; done
echo "COPY entrypoint.sh /usr/local/bin/" >> Dockerfile
echo "CMD entrypoint.sh" >> Dockerfile
The Dockerfile COPY directive (and ADD, but you should usually prefer COPY) understands ordinary shell globs, so you can just
COPY part_* .
(If these files came from the split(1) command, and you're just going to cat(1) them together, it will probably be easier and substantially smaller to combine them on the host before you docker build the image; RUN rm never actually makes an image smaller so your image size will include both the size of the parts and the size of the combined file.)
I want to create a docker image using either git sources, or the already build app. I created two Dockerfiles like these (note: this is pseudo code):
Runtime-Image:
FROM <baseimage>
EXPOSE 1234/tcp
EXPOSE 4321/tcp
VOLUME /foobar
COPY myapp.tgz .
RUN tar -xzf myapp.tgz && rm -f myapp.tgz
ENTRYPOINT ["myapp"]
myapp.tgz is created on a buildserver or maybe by compiling manually. It is available on the docker host server locally.
To build directly from source I use:
FROM <devimage> AS buildenv
ARG GIT_USER
ARG GIT_PASSWORD
RUN git clone http://${GIT_USER}:${GIT_PASSWORD}#<my.git.host>
RUN ./makefile && cp /source/build/myapp.tgz /drop/myapp.tgz
FROM <baseimage> AS runenv
EXPOSE 1234/tcp
EXPOSE 4321/tcp
VOLUME /foobar
COPY --from=buildenv /drop/myapp.tgz .
RUN tar -xzf myapp.tgz && rm -f myapp.tgz
ENTRYPOINT ["myapp"]
The instructions in the second build stage of this are obviously a duplicate of the Runtime-Image Dockerfile.
I'd like to have just ONE Dockerfile, which can build from source, or from context on the docker host, as required. I could put the duplicated commands in a custom baseimage and reuse that to build onto (FROM), but this would obfuscate the Dockerfile.
What is the recommended, most elegant way to do this?
I can't use a bind mount to get myapp.tgz in the current directory on the docker host, can I? For this I would have to start a Container to build my app?
There is no IF directive in the Dockfile for conditions?
If there is no myapp.tgz on the docker host, COPY myapp.tgz . will fail
If there is no buildenv, COPY --from=buildenv /drop/myapp.tgz . will fail.
I could use COPY ./* . and then check with
[ -f /myapp.tgz ] && <prepare-container> || <build-from-git-source>
I guess? Our would you rather just create a seperate Dockerfile just for building from source and then use something like
docker run --rm -v /SomewhereOnHost/drop:/drop my-compile-image
For the past 2 days I have been trying to figure this out, now I have a good solution to achieve a conditional build (a if in Dockerfile)
ARG mode=local
FROM alpine as build_local
ONBUILD COPY myapp.tgz .
FROM alpine as build_remote
ONBUILD RUN git clone GIT_URL
ONBUILD RUN cd repo && ./makefile && cp /source/build/myapp.tgz .
FROM build_${mode} AS runenv
EXPOSE 1234/tcp
EXPOSE 4321/tcp
VOLUME /foobar
RUN tar -xzf myapp.tgz && rm -f myapp.tgz
ENTRYPOINT ["myapp"]
The toplevel mode allows you to pass the condition with docker build --build-arg mode=remote .. ONBUILD is used so the command is only executed if the corresponding branch is selected.
I am trying to add a directory to my docker image. I tried the below methods. During the build I dont see any errors, but once I run the container ( I am using docker-compose) and get into it docker exec -it 410e434a7304 /bin/sh I dont see the directory in the path I am copying it into nor do I see it as a volume when I do docker inspect.
Approach 1 : Classic mkdir
# Define working directory
WORKDIR /root
RUN cd /var \
mdkir www \\ no www directory created
COPY <fileDirectory> /var/www/<fileDirectory>
Approach 2 : Volume
FROM openjdk:8u171 as build
# Define working directory
WORKDIR /root
VOLUME["/var/www"]
COPY <fileDirectory> /var/www/<fileDirectory>
Your first approach is correct in principle, only that your RUN statement is faulty. Try:
RUN cd /var && mkdir www
Also, please note the fundamental difference between RUN mkdir and VOLUME: the former simply creates a directory on your container, while the latter is chiefly intended for mounting directories from your container to the host your container is running on.
This is how I made it work:
# Define working directory
WORKDIR /root
COPY <fileDirectory> /root/<fileDirectory>
RUN cd /var && mkdir www && cp -R /root/<fileDirectory> /var/www
RUN rm -rf /root/email-media
I had to copy the from my host machine to docker image's working directory /root and from /root to the desired destination. Later removed the directory from/root`
Not sure if thats the cleanest way, if I followed the approach 1 with the right syntax suggested by #Fritz it could never find the the path created and throw an error.
After running the RUN layer it would remove the container (as below) and in the COPY line it would not have the reference to the path created in the run line.
Step 16/22 : RUN cd /var && mkdir www && cp -R /root/<fileDirectory> /var/www
---> Running in a9c7df27116e
Removing intermediate container a9c7df27116e
I've been expiriencing a bit of weird behavior regarding volumes. We have a container which contains a database, and is expected to bind mount folders from the host which contain the data. I'm trying to create a child container which ships with test data, as it is just used for testing.
This requires that during the build step, some data is copied off the host machine, and then some scripts run which create additional files. I've noticed however though when I have a look at the running container, only the copied files exist, and the ones created by scripts do not. I've boiled down the steps to the following docker file:
FROM ubuntu:xenial-20180112.1
VOLUME /test
COPY /test/copydir/copyfile.txt /test/copydir/copyfile.txt
RUN mkdir -p /test/mkdir && \
touch /test/mkdir/touch.txt
Note that when I bash into the running container and do an
ls -l /test
I only get the 'copydir' folder. If I run an ls in my dockerfile however, I see that both folders exist.
What's going on here?
edit:
For additional context, the following prints out that both directories exist:
FROM ubuntu:xenial-20180112.1
VOLUME /test
COPY /test/copydir/copyfile.txt /test/copydir/copyfile.txt
RUN mkdir -p /test/mkdir && \
touch /test/mkdir/touch.txt && \
ls -l /test
But the following only shows that copydir exists:
FROM ubuntu:xenial-20180112.1
VOLUME /test
COPY /test/copydir/copyfile.txt /test/copydir/copyfile.txt
RUN mkdir -p /test/mkdir && \
touch /test/mkdir/touch.txt
RUN ls -l /test
I don't have the exact explanation of this but when you build an image with a Dockerfile it will make the lightest image possible. When you use RUN you don't make data persistant but you only do an action that will give a result that will not stay in the image.
Note that apt-get and yum commands make installations persist. It's kinda weird.
Try to change your Dockerfile to:
FROM ubuntu:xenial-20180112.1
RUN mkdir -p /test
COPY /test/copydir/copyfile.txt /test/copydir/copyfile.txt
RUN mkdir -p /test/mkdir && \
touch /test/mkdir/touch.txt
VOLUME /test
In a remark you said "The example I provided was cut down for clarity, in actuality the volume is defined by a parent image." That would relate the problem to the case that is not possible to undeclare a volume entry for a derived image. If that is possible (e.g. using docker-copyedit) then your problem may go away. ;)