This is basically a follow-up question to How to include files outside of Docker's build context?: I'm using large files in all of my projects (several GBs) which I keep on an external drive, only used for development.
I want to COPY or ADD these files to my docker container when building it. The answer linked above allows one to specify a different path to a Dockerfile, potentially extending the build context. I find this unpractical, since this would require setting the build context to system root (?), to be able to include a single file.
Long story short: Is there any way or workaround to include a file that is far removed from the docker build context?
Three suggestions on things you could try:
include a file that is far removed from the docker build context?
You could construct your own build context by cp (or tar) files on the host into a dedicated directory tree. You don't have to use the actual source tree or your build tree.
rm -rf docker-build
mkdir docker-build
cp -a Dockerfile build/the-binary docker-build
cp -a /mnt/external/support docker-build
docker build ./docker-build
# reads docker-build/Dockerfile, and the files in the
# docker-build directory, but nothing else; only sends
# the docker-build directory to Docker as the build context
large files [...] (several GBs)
Docker doesn't deal well with build contexts this large. In the past I've at least seen docker build take a long time just on the step of sending the build context to itself, and docker push and docker pull have network issues when trying to send the gigabyte+ layer around.
It's a little hacky and breaks the "self-contained image" model a little bit, but you can provide these files as a Docker bind-mount instead of including them in the image. Your application needs to know what to do if the data isn't there. When you go to deploy the application, you also need to separately distribute the files alongside the Docker image and other deployment artifacts.
docker run \
-v /mnt/external/support:/app/support
...
the-image-without-the-support-files
only used for development
Potentially you can get away with not using Docker at all during this phase of development. Use a local source tree and local development tools; run your unit tests against these large test fixtures as needed. Build a Docker image only when you're about to run pre-commit integration tests; that may be late enough in the development cycle that you don't need these files.
I think the main thing you are worried about is that you do not want to send all files of a directory to docker daemon while it builds the image.
When directory was so big (in GBss) it takes lot of time to build an image.
If the requirement is to just use those files while you build anything inside docker, you can mount those to the container.
A tricky way
Run a container with base image and mount the direcotries inside it. docker run -d -v local-path:container-path
Get inside the container docker exec -it CONTAINER_ID bash
Run build step ./build-something.sh
Create image from the running container docker commit CONTAINER_ID
Tag the image docker tag IMAGE_ID tag:v1. You can get Image ID from previous command
From long term perspective this method may seem to be very tedious, but if you want to build image for 1 or 2 times , you can try this method.
I tried this for one of my docker image, as I want to avoid large amount of files sent to docker daemon during image build
The copy command gets source and destination values,
just specify full absolute path to your hard drive mount point as the src directory
COPY /absolute_path/to/harddrive /container/path
Related
Is it possible for Dockerfile to copy over some file from the host filesystem and not from the context it's being build from ?
# Inside Dockerfile
FROM gradle:latest
COPY ~/.super/secrets.yaml /opt
# I think you can work around it with but doesn't look nice
COPY ../../../../../../.super/secrets.yaml /opt
when I ran the command on the /home/user/some/path/to/project/ path ?
docker build .
The usual way to get "external" files into your docker container is by copying them into your build directory before starting the docker build. It is strongly recommended to create a script for this to ensure that your preparation step is reproducible.
No this is not possible to go up the directory. Here is why.
When runnig docker build . have you ever considered what this dot stand for at the end? Well, here is part of docker documentation
The docker build command builds Docker images from a Dockerfile and a
“context”. A build’s context is the set of files located in the
specified PATH or URL. The build process can refer to any of the files
in the context. For example, your build can use a COPY instruction to
reference a file in the context.
As you see, this dot is referencing context path (here it means "this directory"). All files under context path get sent to docker daemon and you can reference only these files in your Dockerfile. Of course, you can think that you are clever and reference / (root path) so you have access to all files on your machine. (I highly encourage you to try this and see what happens). What you should see that happens is that docker client is freezing. Or, does it really? Well, it's not really freezing, its sending all / directory to docker daemon and it can take ages or (what's more probable) you may run out of memory.
So now when you understand this limitation you see that the only way to make it work is to copy the file you are interested in to the location of context path and then run docker build command.
When you do docker build ., that last argument is the build context directory: you can only access files from it.
You could do docker build ../../../ but then every single file in that root directory will get packaged up and send to the Docker daemon, which will be slow.
So instead, do something like:
cp ../../../secret.yaml .
docker build .
rm secret.yaml
However, keep in mind that will result in the secret being embedded in the image forever, which might be a security risk. If it's a secret you need for runtime, better to pass it in via environment variable at runtime. If you only need the secret for building the image, there are other alternatives, e.g. https://pythonspeed.com/articles/docker-build-secrets/.
I have searched high and low for an answer to this question. Perhaps it's not possible!
I have some Dockerfiles in directories such as dev/Dockerfile and live/Dockerfile. I cannot find a way to provide these custom Dockerfiles to docker run. docker build has the -f option, but I cannot find a corresponding option for docker run in the root of the app. At the moment I think I will write my npm/gulp script to simply change into those directories, but this is clearly not ideal.
Any ideas?
You can't - that's not how Docker works:
The Dockerfile is used with docker build to build an image
The resulting image is used with docker run to run a container
If you need to make changes at run-time, then you need to modify your base image so that it can take, e.g. a command-line option to docker run, or a configuration file as a mount.
I ran this command in my home directory:
docker build .
and it sent 20 GB files to the Docker daemon before I knew what was happening. I have no space left on my laptop. How do I delete the files that were replicated? I can't locate them.
What happens when you run docker build . command:
Docker client looks for a file named Dockerfile at the same directory where your command runs. If that file doesn't exists, an error is thrown.
Docker client looks a file named .dockerignore. If that file exists, Docker client uses that in next step. If not exists nothing happens.
Docker client makes a tar package called build context. Default, it includes everything in the same directory with Dockerfile. If there are some ignore rules in .dockerignore file, Docker client excludes those files specified in the ignore rules.
Docker client sends the build context to Docker engine which named as Docker daemon or Docker server.
Docker engine gets the build context on the fly and starts building the image, step by step defined in the Dockerfile.
After the image building is done, the build context is released.
So, your build context is not replicated anywhere but in the image you just created if only it needs all the build context. You can check image sizes by running this: docker images. If you see some unused or unnecessary images, use docker rmi unusedImageName.
If your image does'nt need everything in the build context, I suggest you to use .dockerignore rules, to reduce build context size. Exclude everything which are not necessary for the image. This way, the building process will be shorter and you will see if there is any misconfigured COPY or ADD steps in the Dockerfile.
For example, I use something like this:
# .dockerignore
* # exclude everything
!build/libs/*.jar # include just what I need in the image
https://docs.docker.com/engine/reference/builder/#dockerignore-file
https://docs.docker.com/engine/docker-overview/
Likely the space is being used by the resulting image. Locate and delete it:
docker images
Search there by size column.
Then delete it:
docker rmi <image-id>
Also you can delete everything docker-related:
docker system prune -a
In case of stopping the building context for some reason, you can go as well to /var/lib/docker/tmp/, with root access, and then erase the tmp files of docker builder. In this situation, the building context doesn't build correctly, so the part that it did build, was saved in a tmp file in /var/lib/docker/tmp/
I am building an image that installs ant, maven, phantomjs, grunt-cli but when i run
docker build
It's takes a while to start sending build context to docker and then the size is
Sending build context to Docker daemon 7.451 GB
Is that normal or is there something I can do to speed this up?
docker build tars up and transfers the directory where you run it, including all subdirectories. If you want to include less data in the build context (and so speed things up), you should run docker build from a directory containing the minimal data for your build.
So docker build requires at least a Dockerfile and usually you want to include your source code in your local directory (though Docker can pull it from other sources).
How do I copy files from a docker container to the host machine during docker build command?
As a part of building my docker image for my app, I'm running some tests inside it, and I would like to copy the output of the test run into the host (which is a continuous integration server) to do some reporting.
I wouldn't run the tests during build, this will only increase the size of your image. I would recommend you to build the image and then run it mounting a host volume into the container and changing the working directory to the mount point.
docker run -v `pwd`/results:/results -w /results -t IMAGE test_script
There is no easy way to do this. Volumes are only created at run time. You can grab it out of the docker filesystem, (e.g. mine is /var/lib/docker/devicemapper/mnt/CONTAINER_ID/rootfs/PATH_TO_FILE) though there is no good way to figure out when your test process is complete. You could create a file when it's finished and do an inotify, but this is ugly.