So, I have a dummy project, which file structure looks like this:
docker-magic
├── Dockerfile
├── .dockerignore
├── just_a_file
├── src
│ ├── folder_one
│ │ ├── main.go
│ │ └── what_again.sh
│ ├── folder_two
│ │ └── what.kts
│ └── level_one.go
└── top_level.go
My Dockerfile looks like this:
FROM ubuntu:latest as builder
WORKDIR /workdir
COPY ./ ./
RUN find . -type f
ENTRYPOINT ["echo", "wow"]
I build this image with a docker build . -t docker-magic:test --no-cache command to avoid caching results.
The idea is simple - I copy all of the files from docker-magic folder into my image and then list all of them via find . -type f.
Also I want to ignore some of the files. I do that with a .dockerignore file. According to the official Docker docs:
The placement of ! exception rules influences the behavior: the last line of the .dockerignore that matches a particular file determines whether it is included or excluded.
Let's consider following contents of .dockerignore:
**/*.go
It should exclude all the .go files. And it does! I get following contents from find:
./.dockerignore
./src/folder_one/what_again.sh
./src/folder_two/what.kts
./just_a_file
./Dockerfile
Next, let's ignore everything. .dockerignore is now:
**/*
And, as expected, I get empty output from find.
Now it gets difficult. I want to ignore all the files except .go files. According to the docs, it would be the following:
**/*
!**/*.go
But I get the following output from find:
./top_level.go
Which is obviously not what is expected, because other .go files, as we have seen, also match this pattern. How do I get the result I wanted - copying only .go files to my image?
EDIT: my Docker version is 20.10.5, build 55c4c88.
Related
I try to build an image with Docker (10.20.13 on RH 7.9). But some of my resources are outside the Dockerfile directory. Below is my tree :
/dir1
├── dir2
│ ├── dir3
│ │ ├── dir4
│ │ │ ├── boost
│ │ │ │ └── lib
│ │ │ │ ├── linuxV2_6_18A32
│ │ │ │ │ ├── libboost_atomic-mt.a
│ │ │ │ │ ├── ....
/home/myproject/myDockerfile
I want to add in my image the resources that are in /dir1/dir2/dir3/dir4/boost which are not necessary my resources (but I do have at least read access).
My first try was to build an image from /home/myproject/myDockerfile with the following command :
/home/myproject/myDockerfile/docker build -t myimage:1.0 .
But it failed with, saying this:
ADD failed: file not found in build context or excluded by .dockerignore: stat dir1: file does not exist
Okay, the dir1 is not in the context. So I tried to make a link to dir1 in the Dockerfile directory, and again the same command, but different issue :
ADD failed: forbidden path outside the build context: netdata ()
Third try, I launch the command from the root directory (to get all the context as I understand), with the following command:
docker build -t myimage:1.0 -f /home/myproject/myDockerfile
This time I get this response:
error checking context: 'no permission to read from '/boot/System.map-3.10.0-1160.31.1.el7.x86_64''
So I image to add the last directory to my .dockerignore, but it should be in the context (root directory) which is impossible.
So is there a solution to my problem apart copying in project directory all the resources I need?
You have to copy all of the resources you need into the project directory. You can't really build a Docker image containing files from completely unrelated parts of the filesystem (you can't include a library from /usr/lib from a Dockerfile in your home directory).
Since what you're trying to include is a static library, you have a couple of options to get it. The easiest is to just install it via your base image's package manager:
FROM debian:stable
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive \
apt-get install --no-install-recommends --assume-yes \
libboost-atomic1.74-dev
A second harder option is to use a multi-stage build or a similar technique to build the library from source.
Since the specific file you're referencing is a static library, another option could be to build the binary on the host system and COPY it into the image unmodified. This requires you to be on a native-Linux host with a similar base Linux distribution.
FROM debian:stable
COPY ./myapp /usr/local/bin
CMD ["myapp"]
gcc -o myapp ... -lboost_atomic-mt
docker build -t myapp .
If all else fails then you can make a copy locally. You might write a script to do this; this is also a place where Make works well since it's largely dealing with concrete files.
#!/bin/sh
mkdir ./docker
cp -a Dockerfile .dockerignore src ./docker
cp /dir1/dir2/.../libboost_atomic-mt.a ./docker
docker build -t myapp ./docker
I'm trying to create a Dockerfile that copies all package.json files into the image but keeps the folder structure.
This what I have now:
FROM node:15.9.0-alpine as base
WORKDIR /app/
COPY ./**/package.json ./
CMD ls -laR /app
Running with: sudo docker run --rm -it $(sudo docker build -q .)
But it only copies 1 package.json and puts it in the base dir (/app)
Here is the directory I'm testings on:
├── Dockerfile
├── t1
│ └── package.json
└── t2
└── ttt
├── b.txt
└── package.json
And i would like it to look like this inside the container:
├── Dockerfile
├── t1
│ └── package.json
└── t2
└── ttt
└── package.json
The Dockerfile COPY directive is documented as using the Go filepath.Match function for glob expansion. That only supports the basic glob characters *, ?, [a-z], but not extensions like ** that some shells support.
Since COPY only takes a filename glob as input and it likes to flatten the file structure, I don't think there's a way to do the sort of selective copy you're describing in a single command.
Instead you need to list out the individual files you want to copy. COPY will create directories as needed, but that means you need to repeat paths on both sides of COPY.
COPY t1/package*.json t1/
COPY t2/ttt/package*.json t2/ttt/
I can imagine some hacky approaches using multi-stage builds; have an initial stage that copies in the entire source tree but then deletes all of the files except package*.json, then copies that into the actual build stage. I'd contemplate splitting my repository into smaller modules with separate Dockerfiles per module first.
Have a Dockerfile to build releases for an Elixir/Phoenix application...The tree directory structure is as follows, where the Dockerfile (which has a dependency on this other Dockerfile) is in the "infra" subfolder and needs access to all the files one level above "infra".
.
├── README.md
├── assets
│ ├── css
│ ├── js
│ ├── node_modules
│ ├── package-lock.json
│ ├── package.json
├── lib
├── infra
│ ├── Dockerfile
│ ├── config.yaml
│ ├── deployment.yaml
The Dockerfile looks like:
# https://github.com/bitwalker/alpine-elixir
FROM bitwalker/alpine-elixir:latest
# Set exposed ports
EXPOSE 4000
ENV PORT=4000
ENV MIX_ENV=prod
ENV APP_HOME /app
ENV APP_VERSION=0.0.1
COPY ./ ${HOME}
WORKDIR ${HOME}
RUN mix deps.get
RUN mix compile
RUN MIX_ENV=${MIX_ENV} mix distillery.release
RUN echo $HOME
COPY ${HOME}/_build/${MIX_ENV}/rel/my_app/releases/${APP_VERSION}/my_app.tar.gz .
RUN tar -xzvf my_app.tar.gz
USER default
CMD ./bin/my_app foreground
The command "mix distillery.release" is what builds the my_app.tar.gz file in the path indicated by the COPY command.
I invoke the docker build as follows in the top-level directory (the parent directory of "infra"):
docker build -t my_app:local -f infra/Dockerfile .
I basically then get an error with COPY:
Step 13/16 : COPY ${HOME}/_build/${MIX_ENV}/rel/my_app/releases/${APP_VERSION}/my_app.tar.gz .
COPY failed: stat /var/lib/docker/tmp/docker-builder246562111/opt/app/_build/prod/rel/my_app/releases/0.0.1/my_app.tar.gz: no such file or directory
I understand that the COPY command depends on the "build context" but I thought that by issuing the "docker build" in the parent directory of infra meant I had the appropriate context set for the COPY, but clearly that doesn't seem to be the case. Is there a way to have a Dockerfile one level below the parent directory that contains all the files needed to build an Elixir/Phoenix "release" (the my_app.tar.gz and associated files created via the command mix distillery.release)? What bits am I missing?
I have a project including multiple Dockerfiles.
The tree is like,
.
├── app1
│ ├── Dockerfile
│ ├── app.py
│ └── huge_modules/
├── app2
│ ├── Dockerfile
│ ├── app.py
│ └── huge_modules/
├── common
│ └── my_lib.py
└── deploy.sh
To build my application, common/ is necessary and we have to COPY it inside Dockerfile.
However, Dockerfile cannot afford to COPY files from its parent directory.
To be precise, it is possible if we run docker build with -f option in the project root.
But I would not like to do this because the build context will be unnecessarily large.
When building app1, I don't like to include app2/huge_modules/ in the build context (the same as when building app2).
So, I prepare a build script in each app directory.
Like this.
cd $(dirname $0)
cp ../common/* ./
docker build -t app1 .
But this solution seems ugly to me.
Is there a good solution for this case?
Build a base image containing your common library, and then build your two app images on top of that. You'll probably end up restructuring things slightly to provide a Dockerfile for your common files:
.
├── app1
│ ├── Dockerfile
│ ├── app.py
│ └── huge_modules/
├── app2
│ ├── Dockerfile
│ ├── app.py
│ └── huge_modules/
├── base
| ├── Dockerfile
| └── common
│ └── my_lib.py
└── deploy.sh
You start by building a base image:
docker build -t mybaseimage base/
And then your Dockerfile for app1 and app2 would start with:
FROM mybaseimage
One possible solution is to start the build process from the top directory, with the -f flag you mentioned, dynamically generating the .dockerignore file.
That is, lets say that you currently build app1. Then you would first create in the top directory a .dockerignore file with the content: app2, then run the build process. After finishing the build, remove the .dockerignore file.
Now you want to build app2? No problem! Similarly generate first dynamically a .dockerignore file with the content app1, build and remove the file. Voila!
Let's suppose I have an Elixir project tree organized like this:
$ tree .
.
├── apps
│ ├── a
│ │ ├── ...
│ │ └── mix.exs
│ ├── b
│ │ ├── ...
│ │ └── mix.exs
│ └── c
│ ├── ...
│ └── mix.exs
├── mix.exs
└── mix.lock
(unrelated files dropped)
The Dockerfile I wrote for this project looks roughly like this:
FROM bitwalker/alpine-elixir-phoenix:latest
ENV MIX_ENV=prod
# Elixir deps
# FIXME: do not hardcode apps/ subdirectories, use wildcards somehow
# FIXME: copy everything in one step somehow
ADD apps/a/mix.exs apps/a/
ADD apps/b/mix.exs apps/b/
ADD apps/c/mix.exs apps/c/
ADD mix.exs mix.lock ./
RUN mix local.hex --force && mix deps.get && mix deps.compile
<skipped>
ADD . .
RUN mix compile && mix phx.digest
CMD [ "mix", "phx.server" ]
Is there a way to ADD or COPY all the apps/*/mix.exs in a single step, preserving their directory structure (so that each apps/foo/mix.exs ends up in $WORKDIR/apps/foo/mix.exs)?
To the best of my knowledge, there is no obvious way to do what I want. E. g. saying something like ADD apps/*/mix.exs mix.exs mix.lock ./ just copies everything into the WORKDIR (and the files overwrite each other).
Maybe there is some non-obvious way?
A workaround I could think of is to add the following in the .dockerignore file
apps/**
!apps/*/mix.exs
and in the dockerfile,
COPY apps/ /apps
COPY mix.exs mix.lock /
or even
COPY . /
NOTE
If you don't have to stick to pure dockerfile solution, wrap your docker build in a shell script or makefile and use cp is also a good way to do it.