I have a script that creates several files starting with part_, and I would like to create a for loop inside the Dockerfile to search the directory for files starting with part_ and ADD them to the docker image. It is important to ADD each file as a separate layer (and not using wildcards like ADD part_* /directory/). How can I dynamically add files to a docker image?
I would like to use something like the RUN for f in part_*; do ADD $f /data/; done, but I do not know how I can incorporate ADD inside the bash script.
Here is an example of the Dockerfile.
FROM ubuntu:20.04
RUN apt-get update && \
rm -rf /var/lib/apt/lists/*
ADD part_aa /directory/
ADD part_ab /directory/
ADD part_ac /directory/
ADD part_ad /directory/
ADD part_ae /directory/
ADD part_af /directory/
COPY entrypoint.sh /usr/local/bin/
CMD entrypoint.sh
You can use scripting to create your Dockerfile;
echo "FROM ubuntu:20.04" > Dockerfile
echo "RUN apt-get update && \
rm -rf /var/lib/apt/lists/*" >> Dockerfile
for f in part_*; do echo "ADD $f /data/" >> Dockerfile; done
echo "COPY entrypoint.sh /usr/local/bin/" >> Dockerfile
echo "CMD entrypoint.sh" >> Dockerfile
The Dockerfile COPY directive (and ADD, but you should usually prefer COPY) understands ordinary shell globs, so you can just
COPY part_* .
(If these files came from the split(1) command, and you're just going to cat(1) them together, it will probably be easier and substantially smaller to combine them on the host before you docker build the image; RUN rm never actually makes an image smaller so your image size will include both the size of the parts and the size of the combined file.)
Related
I want to decrease the number of layers used in my Dockerfile.
So I want to combine the COPY commands in a RUN cp.
dependencies
folder1
file1
file2
Dockerfile
The following below commands work which I want to combine using a single RUN cp command
COPY ./dependencies/file1 /root/.m2
COPY ./dependencies/file2 /root/.sbt/
COPY ./dependencies/folder1 /root/.ivy2/cache
This following below command says No such file or directory present error. Where could I be going wrong ?
RUN cp ./dependencies/file1 /root/.m2 && \
cp ./dependencies/file2 /root/.sbt/ && \
cp ./dependencies/folder1 /root/.ivy2/cache
You can't do that.
COPY copies from the host to the image.
RUN cp copies from a location in the image to another location in the image.
To get it all into a single COPY statement, you can create the file structure you want on the host and then use tar to make it a single file. Then when you COPY or ADD that tar file, Docker will unpack it and put the files in the correct place. But with the current structure your files have on the host, it's not possible to do in a single COPY command.
Problem
The COPY is used to copy files from your host to your container. So, when you run
COPY ./dependencies/file1 /root/.m2
COPY ./dependencies/file2 /root/.sbt/
COPY ./dependencies/folder1 /root/.ivy2/cache
Docker will look for file1, file2, and folder1 on your host.
However, when you do it with RUN, the commands are executed inside the container, and ./dependencies/file1 (and so on) does not exist in your container yet, which leads to file not found error.
In short, COPY and RUN are not interchangeable.
How to fix
If you don't want to use multiple COPY commands, you can use one COPY to copy all files from your host to your container, then use the RUN command to move them to the proper location.
To avoid copying unnecessary files, use .dockerignore. For example:
.dockerignore
./dependencies/no-need-file
./dependencies/no-need-directory/
Dockerfile
COPY ./dependencies/ /root/
RUN mv ./dependencies/file1 /root/.m2 && \
mv ./dependencies/file2 /root/.sbt/ && \
mv ./dependencies/folder1 /root/.ivy2/cache
You a re missing final slash in /root/.ivy2/cache/
I have GitHub Actions which uses rust-cross to perform cross-compilation for arm64 and other hardware platforms.
I perform cross-compilation on the host machine already and wish to just use the binaries and static libraries to be copied into the Dockerfile and create a light Alpine Container.
Caveat
In rust-cross the released binaries are under specific directories, for example:
arm64 -> target/aarch64-unknown-linux-gnu/release/
amd64 -> target/x86_64-unknown-linux-musl/release/
armv7 -> target/armv7-unknown-linux-gnueabihf/release/
Trials
I am trying to use case within my Dockerfile which relies on docker buildx kit provides and provide the TARGETPLATFORM based on some well documented repository from BretFisher/multi-platform-docker
FROM alpine as base
FROM --platform=${BUILDPLATFORM} alpine as tiny-project
# Use BuildKit to help translate architecture names
ARG TARGETPLATFORM
RUN case ${TARGETPLATFORM} in \
"linux/amd64") TARGET_DIR=x86_64-unknown-linux-musl ;; \
"linux/arm64") TARGET_DIR=aarch64-unknown-linux-gnu ;; \
*) exit 1 \ # ignore other architectures for now!
esac \
WORKDIR /app
RUN cp target/<HOW TO PASS VALUE TARGET_DIR>/release/myBinary .
RUN cp target/<HOW TO PASS VALUE TARGET_DIR>/release/*.so .
FROM base as release
COPY --from=tiny-project /app/* ./
RUN echo '#!/bin/ash' > /entrypoint.sh
RUN echo 'echo " * Starting: /myBinary $*"' >> /entrypoint.sh
RUN echo 'exec /myBinary $*' >> /entrypoint.sh
RUN chmod +x /entrypoint.sh
EXPOSE 7447/udp
EXPOSE 7447/tcp
EXPOSE 8000/tcp
ENV RUST_LOG info
ENTRYPOINT ["/entrypoint.sh"]
I have tried doing a lot of variations but it seems like TARGET_DIR is not being recognized on the host machine
RUN cp ./target/$(echo $TARGET_DIR)/release/myBinary /
RUN cp ./target/$(echo $TARGET_DIR)/release/*.so /
# as well as storing the value in a file and calling it
# echo aarch64-unknown-linux-gnu > /tmp/rust_target.txt
RUN cp ./target/$(cat /tmp/rust_target.txt)/release/zenohd /
RUN cp ./target/$(cat /tmp/rust_target.txt)/release/*.so /
But it seems like the neither the file nor the variable are available to the host and I keep getting an error during my GitHub Actions Workflow Log
Requirements
I wish to keep a single Dockerfile and based on the platform from docker buildx build command I want to copy the binaries from appropriate source directories to the destination directory in the Dockerfile.
How does one achieve this?
Each RUN command runs in its own shell (and its own container), so you can't set variables in one RUN command that last beyond that Dockerfile line.
However, each RUN line also is implicitly wrapped in sh -c, and so you can use ordinary shell constructs to run multiple commands in a single RUN instruction. Since you haven't left a single Dockerfile line, the shell variable you set will still be valid:
WORKDIR /app
# All in a single RUN line:
RUN case "${TARGETPLATFORM}" in \
"linux/amd64") TARGET_DIR=x86_64-unknown-linux-musl ;; \
"linux/arm64") TARGET_DIR=aarch64-unknown-linux-gnu ;; \
*) exit 1 ;; \
esac; \
cp target/$TARGET_DIR/release/myBinary .; \
cp target/$TARGET_DIR/release/*.so .
It would also be reasonable to put this logic into a shell script that you COPY and RUN, or to create a staging directory on the host that contains the files you want to include in the layout you want and use that directory as the docker build context directory.
rm -rf docker-build
mkdir docker-build
TARGET_DIR=...
cp "target/$TARGET_DIR/release/myBinary" docker-build/myBinary
cp Dockerfile docker-build
docker build ./docker-build
I have a Dockerfile that looks like the following.
FROM ubuntu:20.04
RUN apt-get update && \
rm -rf /var/lib/apt/lists/*
ADD part_aa /data/
ADD part_ab /data/
ADD part_ac /data/
ADD part_ad /data/
ADD part_ae /data/
ADD part_af /data/
ADD part_ag /data/
CMD entrypoint.sh
The Dockerfile adds some files into a directory called data, and at the end entrypoint.sh merges together the files that are in the data directory into a single file.
How can I mount a volume so I can move the final output file into my local machine?
I know that I can use the -v flag (volume), but I cannot figure out how to incorporate it into running the the image.
The entrypoint.sh looks like this:
cd /data
MODEL_FILE="merged_file"
if [ ! -f "$MODEL_FILE" ]; then
echo "combining model file parts."
cat part_* > $MODEL_FILE
echo "combining model file parts done"
fi
A Dockerfile is used to build a docker image. The -v switch is applicable when you run an image... you are mixing two things: building and running a docker image.
with a dockerfile like so
FROM python:3.5.5-alpine
ARG CONTAINER_HOME_DIR
ENV CONTAINER_HOME_DIR=$CONTAINER_HOME_DIR
WORKDIR $CONTAINER_HOME_DIR
COPY wall_e/src/requirements.txt .
RUN apk add --update alpine-sdk libffi-dev && \
apk add freetype-dev && \
apk add postgresql-dev && \
pip install --no-cache-dir Cython && \
pip install --no-cache-dir -r requirements.txt && \
apk --update add postgresql-client
COPY wall_e/src ./
CMD ["./wait-for-postgres.sh", "db", "python", "./main.py" ]
If I then use this dockerfile with docker-compose, at what point does docker-compose determine that it needs to re-create the docker image that it will use to make the docker container from that image?
Will it remake the docker image if I make changes to the wall_e/src/requirements.txt file or will it remake the docker image if I make a change to the RUN line or the make a change to any files located in wall_e/src or even change the COPY LINE entirely or the CMD line?
Lets assume that I am using docker-compose up and I am not using the option --force-recreate
Once the image is built, docker-compose will not look into your dockerfile.
It will use that image, and only apply its configuration (docker-compose.yml).
So let's say you docker-compose up, then edit your docker-compose.yml file, your requirements.txt and your dockerfile, when you use docker-compose restart only the docker-compose.yml changes will be taken into account.
If you need to rebuild your image, you'll have to use specifically :
docker-compose build [my_service]
Docker will not rebuild images unless 1) instructed to do so, or 2) if the named image doesn’t exist.
That being said, when you rebuild, it will attempt to rebuild based on any cached information it has, and does not re-process steps unless the dockerfile has been modified, or if a file referenced via COPY has been modified. Any subsequent steps after a re-processed step will also be re-processed, because the build process basically creates new sub-images based on sub-images built from previous steps in the dockerfile.
However, if you specify —no-cache, it will re-process all steps in the dockerfile.
In docker I want to do this:
git clone XYZ
cd XYZ
make XYZ
However because there is no cd command, I have to pass in the full path everytime (make XYZ /fullpath). Any good solutions for this?
To change into another directory use WORKDIR. All the RUN, CMD and ENTRYPOINT commands after WORKDIR will be executed from that directory.
RUN git clone XYZ
WORKDIR "/XYZ"
RUN make
You can run a script, or a more complex parameter to the RUN. Here is an example from a Dockerfile I've downloaded to look at previously:
RUN cd /opt && unzip treeio.zip && mv treeio-master treeio && \
rm -f treeio.zip && cd treeio && pip install -r requirements.pip
Because of the use of '&&', it will only get to the final 'pip install' command if all the previous commands have succeeded.
In fact, since every RUN creates a new commit & (currently) an AUFS layer, if you have too many commands in the Dockerfile, you will use up the limits, so merging the RUNs (when the file is stable) can be a very useful thing to do.
I was wondering if two times WORKDIR will work or not, but it worked :)
FROM ubuntu:18.04
RUN apt-get update && \
apt-get install -y python3.6
WORKDIR /usr/src
COPY ./ ./
WORKDIR /usr/src/src
CMD ["python3", "app.py"]
You can use single RUN command for all of them
RUN git clone XYZ && \
cd XYZ && \
make XYZ
In case you want to change the working directory for the container when you run a docker image, you can use the -w (short for --workdir) option:
docker run -it -w /some/valid/directory/inside/docker {image-name}
Ref:
docker run options: https://docs.docker.com/engine/reference/commandline/run/#options
Mind that if you must run in bash shell, you need not just the RUN make, but you need to call the bash shell before, since in Docker, you are in the sh shell by default.
Taken from /bin/sh: 1: gvm: not found, which would say more or less:
Your shell is /bin/sh, but source expects /bin/bash, perhaps because it
puts its initialization in ~/.bashrc.
In other words, this problem can occur in any setting where the "sh" shell is used instead of the "bash", causing "/bin/sh: 1: MY_COMMAND: not found".
In the Dockerfile case, use the recommended
RUN /bin/bash -c 'source /opt/ros/melodic/setup.bash'
or with the "[]" (which I would rather not use):
RUN ["/bin/bash", "-c", "source /opt/ros/melodic/setup.bash"]
Every new RUN of a bash is isolated, "starting at 0". For example, mind that setting WORKDIR /MY_PROJECT before the bash commands in the Dockerfile does not affect the bash commands since the starting folder would have to be set in the ".bashrc" again. It needs cd /MY_PROJECT even if you have set WORKDIR.
Side-note: do not forget the first "/" before "opt/../...". Else, it will throw the error:
/bin/bash: opt/ros/melodic/setup.bash: No such file or directory
Works:
=> [stage-2 18/21] RUN ["/bin/bash", "-c", "source /opt/ros/melodic/setup.bash"] 0.5s
=> [stage-2 19/21] [...]
See “/bin/sh: 1: MY_COMMAND: not found” at SuperUser for some more details on how this looks with many lines, or how you would fill the ".bashrc" instead. But that goes a bit beyond the actual question here.
PS: You might also put the commands you want to execute in a single bash script and run that bash script in the Dockerfile (though I would rather put the bash commands in the Dockerfile as well, just my opinion):
#!/bin/bash
set -e
source /opt/ros/melodic/setup.bash