This question already has answers here:
How to include files outside of Docker's build context?
(19 answers)
Closed 1 year ago.
Env: Windows 10, Docker Desktop (Docker Engine v20.10.6)
Build of a Docker image fails on command COPY (and ADD) when its first argument is a relative path starting with ../../. Error message shows that any path like ../../a/b/c is replaced with /a/b/c and hence cannot be found on the host.
Please, help me to find a way to workaround the problem. In multiple articles and forums I see that Docker can handle relative path properly, but cannot figure out what is wrong in this case.
Project folder structure
project_root
first
container
Dockerfile
second
target
artifact.file
Dockerfile
FROM whatever
RUN addgroup -S apprunner && adduser -S apprunner -G apprunner
COPY ../../second/target/artifact.file /home/apprunner/app.file
USER apprunner:apprunner
WORKDIR /home/apprunner
EXPOSE 8080
ENTRYPOINT blabla
Executing command from project_root
cd first/container
docker build -q -t my_image_name .
get this error
...
#7 [4/5] COPY ../../second/target/artifact.file /home/apprunner/app.file
#7 sha256:fefde24bc79e3e0b7a3ba0bf6754187537780b9c30fa81537cb5aea93ef9331c
#7 ERROR: "/second/target/artifact.file" not found: not found
------
> [4/5] COPY ../../second/target/artifact.file /home/apprunner/app.file:
------
failed to compute cache key: "/second/target/artifact.file" not found: not found
Cannot find a reason why the relative path is replaced with an absolute path.
The COPY and ADD instructions are relative to the context path passed as last parameter to the command docker build.
I solved that issue several times by:
copying the out of context file into the context path;
copying to the container image by referencing it from context path;
removing the (temporary) copied file.
So you will have:
cd first/container
cp ../../second/target/artifact.file artifact.file
docker build -q -t my_image_name .
rm artifact.file
and
FROM whatever
RUN addgroup -S apprunner && adduser -S apprunner -G apprunner
COPY artifact.file /home/apprunner/app.file
USER apprunner:apprunner
WORKDIR /home/apprunner
EXPOSE 8080
ENTRYPOINT blabla
Seems like a basic issue but couldnt find any answers so far ..
When using ADD / COPY in Dockerfile and running the image on linux, the default file permission of the file copied in the image is 644. The onwner of this file seems to be as 'root'
However, when running the image, a non-root user starts the container and any file thus copied with 644 permission cannot execute this copied/added file and if the file is executed at ENTRYPOINT it fails to start with permission denied error.
I read in one of the posts that COPY/ADD after Docker 1.17.0+ allows chown but in my case i dont know who will be the non-root user starting so i cannot set the permission as that user.
I also saw another work around to ADD/COPY files to a different location and use RUN to copy them from the temp location to actual folder like what am doing below. But this approach doesnt work as the final image doesnt have the files in /otp/scm
#Installing Bitbucket and setting variables
WORKDIR /tmp
ADD atlassian-bitbucket-${BITBUCKET_VERSION}.tar.gz .
COPY bbconfigupdater.sh .
#Copying Entrypoint script which will get executed when container starts
WORKDIR /tmp
COPY entrypoint.sh .
RUN ls -lrth /tmp
WORKDIR /opt/scm
RUN pwd && cp /tmp/bbconfigupdater.sh /opt/scm \
&& cp /tmp/entrypoint.sh /opt/scm \
&& cp -r /tmp/atlassian-bitbucket-${BITBUCKET_VERSION} /opt/scm \
&& chgrp -R 0 /opt/ \
&& chmod -R 755 /opt/ \
&& chgrp -R 0 /scm/bitbucket \
&& chmod -R 755 /scm/bitbucket \
&& ls -lrth /opt/scm && ls -lrth /scmdata
Any help is appreciated to figure out how i can get my entrypoint script copied to the desired path with execute permissions set.
The default file permission is whatever the file permission is in your build context from where you copy the file. If you control the source, then it's best to fix the permissions there to avoid a copy-on-write operation. Otherwise, if you cannot guarantee the system building the image will have the execute bit set on the files, a chmod after the copy operation will fix the permission. E.g.
COPY entrypoint.sh .
RUN chmod +x entrypoint.sh
A better option with newer versions of docker (and which didn't exist when this answer was first posted) is to use the --chmod flag (the permissions must be specified in octal at last check):
COPY --chmod=0755 entrypoint.sh .
You do not need to know who will run the container. The user inside the container is typically configured by the image creator (using USER) and doesn't depend on the user running the container from the docker host. When the user runs the container, they send a request to the docker API which does not track the calling user id.
The only time I've seen the host user matter is if you have a host volume and want to avoid permission issues. If that's your scenario, I often start the entrypoint as root, run a script called fix-perms to align the container uid with the host volume uid, and then run gosu to switch from root back to the container user.
A --chmod flag was added to ADD and COPY instructions in Docker CE 20.10. So you can now do.
COPY --chmod=0755 entrypoint.sh .
To be able to use it you need to enable BuildKit.
# enable buildkit for docker
DOCKER_BUILDKIT=1
# enable buildkit for docker-compose
COMPOSE_DOCKER_CLI_BUILD=1
Note: It seems to not be documented at this time, see this issue.
I'm building a Rust program in Docker (rust:1.33.0).
Every time code changes, it re-compiles (good), which also re-downloads all dependencies (bad).
I thought I could cache dependencies by adding VOLUME ["/usr/local/cargo"]. edit I've also tried moving this dir with CARGO_HOME without luck.
I thought that making this a volume would persist the downloaded dependencies, which appear to be in this directory.
But it didn't work, they are still downloaded every time. Why?
Dockerfile
FROM rust:1.33.0
VOLUME ["/output", "/usr/local/cargo"]
RUN rustup default nightly-2019-01-29
COPY Cargo.toml .
COPY src/ ./src/
RUN ["cargo", "build", "-Z", "unstable-options", "--out-dir", "/output"]
Built with just docker build ..
Cargo.toml
[package]
name = "mwe"
version = "0.1.0"
[dependencies]
log = { version = "0.4.6" }
Code: just hello world
Output of second run after changing main.rs:
...
Step 4/6 : COPY Cargo.toml .
---> Using cache
---> 97f180cb6ce2
Step 5/6 : COPY src/ ./src/
---> 835be1ea0541
Step 6/6 : RUN ["cargo", "build", "-Z", "unstable-options", "--out-dir", "/output"]
---> Running in 551299a42907
Updating crates.io index
Downloading crates ...
Downloaded log v0.4.6
Downloaded cfg-if v0.1.6
Compiling cfg-if v0.1.6
Compiling log v0.4.6
Compiling mwe v0.1.0 (/)
Finished dev [unoptimized + debuginfo] target(s) in 17.43s
Removing intermediate container 551299a42907
---> e4626da13204
Successfully built e4626da13204
A volume inside the Dockerfile is counter-productive here. That would mount an anonymous volume at each build step, and again when you run the container. The volume during each build step is discarded after that step completes, which means you would need to download the entire contents again for any other step needing those dependencies.
The standard model for this is to copy your dependency specification, run the dependency download, copy your code, and then compile or run your code, in 4 separate steps. That lets docker cache the layers in an efficient manner. I'm not familiar with rust or cargo specifically, but I believe that would look like:
FROM rust:1.33.0
RUN rustup default nightly-2019-01-29
COPY Cargo.toml .
RUN cargo fetch # this should download dependencies
COPY src/ ./src/
RUN ["cargo", "build", "-Z", "unstable-options", "--out-dir", "/output"]
Another option is to turn on some experimental features with BuildKit (available in 18.09, released 2018-11-08) so that docker saves these dependencies in what is similar to a named volume for your build. The directory can be reused across builds, but never gets added to the image itself, making it useful for things like a download cache.
# syntax=docker/dockerfile:experimental
FROM rust:1.33.0
VOLUME ["/output", "/usr/local/cargo"]
RUN rustup default nightly-2019-01-29
COPY Cargo.toml .
COPY src/ ./src/
RUN --mount=type=cache,target=/root/.cargo \
["cargo", "build", "-Z", "unstable-options", "--out-dir", "/output"]
Note that the above assumes cargo is caching files in /root/.cargo. You'd need to verify this and adjust as appropriate. I also haven't mixed the mount syntax with a json exec syntax to know if that part works. You can read more about the BuildKit experimental features here: https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md
Turning on BuildKit from 18.09 and newer versions is as easy as export DOCKER_BUILDKIT=1 and then running your build from that shell.
I would say, the nicer solution would be to resort to docker multi-stage build as pointed here and there
This way you can create yourself a first image, that would build both your application and your dependencies, then use, only, in the second image, the dependency folder from the first one
This is inspired by both your comment on #Jack Gore's answer and the two issue comments linked here above.
FROM rust:1.33.0 as dependencies
WORKDIR /usr/src/app
COPY Cargo.toml .
RUN rustup default nightly-2019-01-29 && \
mkdir -p src && \
echo "fn main() {}" > src/main.rs && \
cargo build -Z unstable-options --out-dir /output
FROM rust:1.33.0 as application
# Those are the lines instructing this image to reuse the files
# from the previous image that was aliased as "dependencies"
COPY --from=dependencies /usr/src/app/Cargo.toml .
COPY --from=dependencies /usr/local/cargo /usr/local/cargo
COPY src/ src/
VOLUME /output
RUN rustup default nightly-2019-01-29 && \
cargo build -Z unstable-options --out-dir /output
PS: having only one run will reduce the number of layers you generate; more info here
Here's an overview of the possibilities. (Scroll down for my original answer.)
Add Cargo files, create fake main.rs/lib.rs, then compile dependencies. Afterwards remove the fake source and add the real ones. [Caches dependencies, but several fake files with workspaces].
Add Cargo files, create fake main.rs/lib.rs, then compile dependencies. Afterwards create a new layer with the dependencies and continue from there. [Similar to above].
Externally mount a volume for the cache dir. [Caches everything, relies on caller to pass --mount].
Use RUN --mount=type=cache,target=/the/path cargo build in the Dockerfile in new Docker versions. [Caches everything, seems like a good way, but currently too new to work for me. Executable not part of image. Edit: See here for a solution.]
Run sccache in another container or on the host, then connect to that during the build process. See this comment in Cargo issue 2644.
Use cargo-build-deps. [Might work for some, but does not support Cargo workspaces (in 2019)].
Wait for Cargo issue 2644. [There's willingness to add this to Cargo, but no concrete solution yet].
Using VOLUME ["/the/path"] in the Dockerfile does NOT work, this is per-layer (per command) only.
Note: one can set CARGO_HOME and ENV CARGO_TARGET_DIR in the Dockerfile to control where download cache and compiled output goes.
Also note: cargo fetch can at least cache downloading of dependencies, although not compiling.
Cargo workspaces suffer from having to manually add each Cargo file, and for some solutions, having to generate a dozen fake main.rs/lib.rs. For projects with a single Cargo file, the solutions work better.
I've got caching to work for my particular case by adding
ENV CARGO_HOME /code/dockerout/cargo
ENV CARGO_TARGET_DIR /code/dockerout/target
Where /code is the directory where I mount my code.
This is externally mounted, not from the Dockerfile.
EDIT1: I was confused why this worked, but #b.enoit.be and #BMitch cleared up that it's because volumes declared inside the Dockerfile only live for one layer (one command).
You do not need to use an explicit Docker volume to cache your dependencies. Docker will automatically cache the different "layers" of your image. Basically, each command in the Dockerfile corresponds to a layer of the image. The problem you are facing is based on how Docker image layer caching works.
The rules that Docker follows for image layer caching are listed in the official documentation:
Starting with a parent image that is already in the cache, the next
instruction is compared against all child images derived from that
base image to see if one of them was built using the exact same
instruction. If not, the cache is invalidated.
In most cases, simply comparing the instruction in the Dockerfile with
one of the child images is sufficient. However, certain instructions
require more examination and explanation.
For the ADD and COPY instructions, the contents of the file(s) in the
image are examined and a checksum is calculated for each file. The
last-modified and last-accessed times of the file(s) are not
considered in these checksums. During the cache lookup, the checksum
is compared against the checksum in the existing images. If anything
has changed in the file(s), such as the contents and metadata, then
the cache is invalidated.
Aside from the ADD and COPY commands, cache checking does not look at
the files in the container to determine a cache match. For example,
when processing a RUN apt-get -y update command the files updated in
the container are not examined to determine if a cache hit exists. In
that case just the command string itself is used to find a match.
Once the cache is invalidated, all subsequent Dockerfile commands
generate new images and the cache is not used.
So the problem is with the positioning of the command COPY src/ ./src/ in the Dockerfile. Whenever there is a change in one of your source files, the cache will be invalidated and all subsequent commands will not use the cache. Therefore your cargo build command will not use the Docker cache.
To solve your problem it will be as simple as reordering the commands in your Docker file, to this:
FROM rust:1.33.0
RUN rustup default nightly-2019-01-29
COPY Cargo.toml .
RUN ["cargo", "build", "-Z", "unstable-options", "--out-dir", "/output"]
COPY src/ ./src/
Doing it this way, your dependencies will only be reinstalled when there is a change in your Cargo.toml.
Hope this helps.
With the integration of BuildKit into docker, if you are able to avail yourself of the superior BuildKit backend, it's now possible to mount a cache volume during a RUN command, and IMHO, this has become the best way to cache cargo builds. The cache volume retains the data that was written to it on previous runs.
To use BuildKit, you'll mount two cache volumes, one for the cargo dir, which caches external crate sources, and one for the target dir, which caches all of your built artifacts, including external crates and the project bins and libs.
If your base image is rust, $CARGO_HOME is set to /usr/local/cargo, so your command looks like this:
RUN --mount=type=cache,target=/usr/local/cargo,from=rust,source=/usr/local/cargo \
--mount=type=cache,target=target \
cargo build
If your base image is something else, you will need to change the /usr/local/cargo bit to whatever is the value of $CARGO_HOME, or else add a ENV CARGO_HOME=/usr/local/cargo line. As a side note, the clever thing would be to set literally target=$CARGO_HOME and let Docker do the expansion, but it
doesn't seem to work right - expansion happens, but buildkit still doesn't persist the same volume across runs when you do this.
Other options for achieving Cargo build caching (including sccache and the cargo wharf project) are described in this github issue.
I figured out how to get this also working with cargo workspaces, using romac's fork of cargo-build-deps.
This example has my_app, and two workspaces: utils and db.
FROM rust:nightly as rust
# Cache deps
WORKDIR /app
RUN sudo chown -R rust:rust .
RUN USER=root cargo new myapp
# Install cache-deps
RUN cargo install --git https://github.com/romac/cargo-build-deps.git
WORKDIR /app/myapp
RUN mkdir -p db/src/ utils/src/
# Copy the Cargo tomls
COPY myapp/Cargo.toml myapp/Cargo.lock ./
COPY myapp/db/Cargo.toml ./db/
COPY myapp/utils/Cargo.toml ./utils/
# Cache the deps
RUN cargo build-deps
# Copy the src folders
COPY myapp/src ./src/
COPY myapp/db/src ./db/src/
COPY myapp/utils/src/ ./utils/src/
# Build for debug
RUN cargo build
I'm sure you can adjust this code for use with a Dockerfile, but I wrote a dockerized drop-in replacement for cargo that you can save to a package and run as ./cargo build --release. This just works for (most) development (uses rust:latest), but isn't set up for CI or anything.
Usage: ./cargo build, ./cargo build --release, etc
It will use the current working directory and save the cache to ./.cargo. (You can ignore the entire directory in your version control and it doesn't need to exist beforehand.)
Create a file named cargo in your project's folder, run chmod +x ./cargo on it, and place the following code in it:
#!/bin/bash
# This is a drop-in replacement for `cargo`
# that runs in a Docker container as the current user
# on the latest Rust image
# and saves all generated files to `./cargo/` and `./target/`.
#
# Be sure to make this file executable: `chmod +x ./cargo`
#
# # Examples
#
# - Running app: `./cargo run`
# - Building app: `./cargo build`
# - Building release: `./cargo build --release`
#
# # Installing globally
#
# To run `cargo` from anywhere,
# save this file to `/usr/local/bin`.
# You'll then be able to use `cargo`
# as if you had installed Rust globally.
sudo docker run \
--rm \
--user "$(id -u)":"$(id -g)" \
--mount type=bind,src="$PWD",dst=/usr/src/app \
--workdir /usr/src/app \
--env CARGO_HOME=/usr/src/app/.cargo \
rust:latest \
cargo "$#"
In my dockerfile, I want to copy a file from ~/.ssh of my host machine into the container, so i worte it like this:
# create ssh folder and copy ssh keys from local into container
RUN mkdir -p /root/.ssh
COPY ~/.ssh/id_rsa /root/.ssh/
But when I run docker build -t foo to build it, it stopped with an error:
Step 2 : RUN mkdir -p /root/.ssh
---> Using cache
---> db111747d125
Step 3 : COPY ~/.ssh/id_rsa /root/.ssh/
~/.ssh/id_rsa: no such file or directory
It seems the ~ symbol is not recognized by dockerfile, how could I resolve this issue?
In Docker, it is not possible to copy files from anywhere on the system into the image, since this would be considered a security risk. COPY paths are always considered relative to the build context, which is current directory where you run the docker build command.
This is described in the documentation: https://docs.docker.com/reference/builder/#copy
As a result, the ~ has no useful meaning, since it would try and direct you to a location which is not part of the context.
If you want to put your local id_rsa file into the docker, you should put it into the context first, e.g. copy it along side the Dockerfile, and refer to it that way.
All, i'm trying to persistently copy files from my host to an image so those files are available with every container launched based on that image. Running on debian wheezy 64bit as virtualbox guest.
the Dockerfile is fairly simple (installing octave image):
FROM debian:jessie
MAINTAINER GG_Python <[redacted]#gmail.com>
RUN apt-get update
RUN apt-get update
RUN apt-get install -y octave octave-image octave-missing-functions octave-nan octave-statistics
RUN mkdir /octave
RUN mkdir /octave/libs
RUN mkdir /octave/libs/jsonlab
COPY ~/octave/jsonlab/loadjson.m /octave/libs/jsonlab/.
I'm getting the following trace after issuing a build command: docker build -t octave .
Sending build context to Docker daemon 423.9 kB
Sending build context to Docker daemon
Step 0 : FROM debian:jessie
---> 58052b122b60
Step 1 : MAINTAINER GG_Python <[..]#gmail.com>
---> Using cache
---> 90d2dd2f7ee8
Step 2 : RUN apt-get update
---> Using cache
---> 4c72c25cd829
Step 3 : RUN apt-get update
---> Using cache
---> b52f0bcb9f86
Step 4 : RUN apt-get install -y octave octave-image octave-missing-functions octave-nan octave-statistics
---> Using cache
---> f0637ab96d5e
Step 5 : RUN mkdir /octave
---> Using cache
---> a2d278b2819b
Step 6 : RUN mkdir /octave/libs
---> Using cache
---> 65efbbe01c99
Step 7 : RUN mkdir /octave/libs/jsonlab
---> Using cache
---> e41b80901266
Step 8 : COPY ~/octave/jsonlab/loadjson.m /octave/libs/jsonlab/.
INFO[0000] ~/octave/jsonlab/loadjson.m: no such file or directory
Docker absolutely refuses to copy this file from the host into the image. Needless to say a the file loadjson.m is there (cat displays), all my attempts to change the path (relative, absolute, etc.) failed. Any advice why this simple task is problematic?
At the time I originally wrote this, Docker didn’t expand ~ or $HOME. Now it does some expansions inside the build context, but even so they are probably not what you want—they aren’t your home directory outside the context. You need to reference the file explicitly, or package it relative to the Dockerfile itself.
Docker can only copy files from the context, the folder you are minus any file listed in the dockerignore file.
When you run 'docker build' docker tars the context and it sends it to the docker daemon you are connected to. It only lets you copy files inside of the context because the daemon might be a remote machine.
I couldn't get COPY to work until I understood the context (I was trying to copy a file from outside of the context)
The docker build command builds an image from a Dockerfile and a context. The build’s context is the files at a specified location PATH. The PATH is a directory on your local filesystem.
A context is processed recursively. So, a PATH includes any subdirectories.
The build is run by the Docker daemon, not by the CLI. The first thing a build process does is send the entire context (recursively) to the daemon. In most cases, it’s best to start with an empty directory as context and keep your Dockerfile in that directory. Add only the files needed for building the Dockerfile.
Warning: Do not use your root directory, /, as the PATH as it causes the build to transfer the entire contents of your hard drive to the Docker daemon.
Reference:
https://docs.docker.com/engine/reference/builder/#usage
I had similar issue. I solved it by checking two things:
Inside your, docker-compose.yaml check context of the service, docker will not copy any file outside of this directory. For example if the context is app/ then you cannot copy anything from ../app
Check .dockerignore to be sure that you are not ignoring the file you want to copy.
I got it working by first checking what the context was,
setting an absolute path before the source file in
your Dockerfile to get that information:
# grep COPY Dockerfile
COPY /path/to/foo /whatever/path/in/destination/foo
Building with that:
docker build -t bar/foo .
you'll get an error, which states the context-path that Docker
is apparently looking into for its files, e.g.
it turns out to be:
/var/lib/docker/tmp # I don't remember
Copying(!) your set of build-files in that directory (here: /var/lib/docker/tmp),
cd into it, build from there.
See if that works, and don't forget to do some housekeeping cleaning up
the tmp, deleting your files before the next visit(or).
HTH
Michael
Got this error using a Dockerfile for a linux container on a Windows machine:
#24 1.160 Skipping project "/src/Common/MetaData/Metadata.csproj" because it was not found.
Restore worked perfectly on the host machine.
Turned out to be the error mentioned here:
https://stackoverflow.com/a/68592423/3850405
A .csproj file did not match casing in Visual Studio vs the file system.