I have this structure:
project
└── pipable
└── common
└── common
└── setup.py # setup.py for package "common"
└── utils
└── utils
└── setup.py # setup.py for package "utils"
└── platform
└── service_one
└── Dockerfile
└── requirements.txt
└── service_two
└── Dockerfile
└── requirements.txt
docker-compose.yml
The package common and utils get pip installed in service_one and service_two, this is done from a remote pypi repository.
I want to make these packages editable, if you change in common then the change should be reflected in the services. I've looked at editable from git but it seems like it would create a copy within the services meaning that editing common outside the services won't update.
Solved it by adding the pipable folder as a volume for each service, after which I run pip install -e location_of_mount_setup_file.
Related
I try to build an image with Docker (10.20.13 on RH 7.9). But some of my resources are outside the Dockerfile directory. Below is my tree :
/dir1
├── dir2
│ ├── dir3
│ │ ├── dir4
│ │ │ ├── boost
│ │ │ │ └── lib
│ │ │ │ ├── linuxV2_6_18A32
│ │ │ │ │ ├── libboost_atomic-mt.a
│ │ │ │ │ ├── ....
/home/myproject/myDockerfile
I want to add in my image the resources that are in /dir1/dir2/dir3/dir4/boost which are not necessary my resources (but I do have at least read access).
My first try was to build an image from /home/myproject/myDockerfile with the following command :
/home/myproject/myDockerfile/docker build -t myimage:1.0 .
But it failed with, saying this:
ADD failed: file not found in build context or excluded by .dockerignore: stat dir1: file does not exist
Okay, the dir1 is not in the context. So I tried to make a link to dir1 in the Dockerfile directory, and again the same command, but different issue :
ADD failed: forbidden path outside the build context: netdata ()
Third try, I launch the command from the root directory (to get all the context as I understand), with the following command:
docker build -t myimage:1.0 -f /home/myproject/myDockerfile
This time I get this response:
error checking context: 'no permission to read from '/boot/System.map-3.10.0-1160.31.1.el7.x86_64''
So I image to add the last directory to my .dockerignore, but it should be in the context (root directory) which is impossible.
So is there a solution to my problem apart copying in project directory all the resources I need?
You have to copy all of the resources you need into the project directory. You can't really build a Docker image containing files from completely unrelated parts of the filesystem (you can't include a library from /usr/lib from a Dockerfile in your home directory).
Since what you're trying to include is a static library, you have a couple of options to get it. The easiest is to just install it via your base image's package manager:
FROM debian:stable
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive \
apt-get install --no-install-recommends --assume-yes \
libboost-atomic1.74-dev
A second harder option is to use a multi-stage build or a similar technique to build the library from source.
Since the specific file you're referencing is a static library, another option could be to build the binary on the host system and COPY it into the image unmodified. This requires you to be on a native-Linux host with a similar base Linux distribution.
FROM debian:stable
COPY ./myapp /usr/local/bin
CMD ["myapp"]
gcc -o myapp ... -lboost_atomic-mt
docker build -t myapp .
If all else fails then you can make a copy locally. You might write a script to do this; this is also a place where Make works well since it's largely dealing with concrete files.
#!/bin/sh
mkdir ./docker
cp -a Dockerfile .dockerignore src ./docker
cp /dir1/dir2/.../libboost_atomic-mt.a ./docker
docker build -t myapp ./docker
Have a Dockerfile to build releases for an Elixir/Phoenix application...The tree directory structure is as follows, where the Dockerfile (which has a dependency on this other Dockerfile) is in the "infra" subfolder and needs access to all the files one level above "infra".
.
├── README.md
├── assets
│ ├── css
│ ├── js
│ ├── node_modules
│ ├── package-lock.json
│ ├── package.json
├── lib
├── infra
│ ├── Dockerfile
│ ├── config.yaml
│ ├── deployment.yaml
The Dockerfile looks like:
# https://github.com/bitwalker/alpine-elixir
FROM bitwalker/alpine-elixir:latest
# Set exposed ports
EXPOSE 4000
ENV PORT=4000
ENV MIX_ENV=prod
ENV APP_HOME /app
ENV APP_VERSION=0.0.1
COPY ./ ${HOME}
WORKDIR ${HOME}
RUN mix deps.get
RUN mix compile
RUN MIX_ENV=${MIX_ENV} mix distillery.release
RUN echo $HOME
COPY ${HOME}/_build/${MIX_ENV}/rel/my_app/releases/${APP_VERSION}/my_app.tar.gz .
RUN tar -xzvf my_app.tar.gz
USER default
CMD ./bin/my_app foreground
The command "mix distillery.release" is what builds the my_app.tar.gz file in the path indicated by the COPY command.
I invoke the docker build as follows in the top-level directory (the parent directory of "infra"):
docker build -t my_app:local -f infra/Dockerfile .
I basically then get an error with COPY:
Step 13/16 : COPY ${HOME}/_build/${MIX_ENV}/rel/my_app/releases/${APP_VERSION}/my_app.tar.gz .
COPY failed: stat /var/lib/docker/tmp/docker-builder246562111/opt/app/_build/prod/rel/my_app/releases/0.0.1/my_app.tar.gz: no such file or directory
I understand that the COPY command depends on the "build context" but I thought that by issuing the "docker build" in the parent directory of infra meant I had the appropriate context set for the COPY, but clearly that doesn't seem to be the case. Is there a way to have a Dockerfile one level below the parent directory that contains all the files needed to build an Elixir/Phoenix "release" (the my_app.tar.gz and associated files created via the command mix distillery.release)? What bits am I missing?
I have a project including multiple Dockerfiles.
The tree is like,
.
├── app1
│ ├── Dockerfile
│ ├── app.py
│ └── huge_modules/
├── app2
│ ├── Dockerfile
│ ├── app.py
│ └── huge_modules/
├── common
│ └── my_lib.py
└── deploy.sh
To build my application, common/ is necessary and we have to COPY it inside Dockerfile.
However, Dockerfile cannot afford to COPY files from its parent directory.
To be precise, it is possible if we run docker build with -f option in the project root.
But I would not like to do this because the build context will be unnecessarily large.
When building app1, I don't like to include app2/huge_modules/ in the build context (the same as when building app2).
So, I prepare a build script in each app directory.
Like this.
cd $(dirname $0)
cp ../common/* ./
docker build -t app1 .
But this solution seems ugly to me.
Is there a good solution for this case?
Build a base image containing your common library, and then build your two app images on top of that. You'll probably end up restructuring things slightly to provide a Dockerfile for your common files:
.
├── app1
│ ├── Dockerfile
│ ├── app.py
│ └── huge_modules/
├── app2
│ ├── Dockerfile
│ ├── app.py
│ └── huge_modules/
├── base
| ├── Dockerfile
| └── common
│ └── my_lib.py
└── deploy.sh
You start by building a base image:
docker build -t mybaseimage base/
And then your Dockerfile for app1 and app2 would start with:
FROM mybaseimage
One possible solution is to start the build process from the top directory, with the -f flag you mentioned, dynamically generating the .dockerignore file.
That is, lets say that you currently build app1. Then you would first create in the top directory a .dockerignore file with the content: app2, then run the build process. After finishing the build, remove the .dockerignore file.
Now you want to build app2? No problem! Similarly generate first dynamically a .dockerignore file with the content app1, build and remove the file. Voila!
Let's suppose I have an Elixir project tree organized like this:
$ tree .
.
├── apps
│ ├── a
│ │ ├── ...
│ │ └── mix.exs
│ ├── b
│ │ ├── ...
│ │ └── mix.exs
│ └── c
│ ├── ...
│ └── mix.exs
├── mix.exs
└── mix.lock
(unrelated files dropped)
The Dockerfile I wrote for this project looks roughly like this:
FROM bitwalker/alpine-elixir-phoenix:latest
ENV MIX_ENV=prod
# Elixir deps
# FIXME: do not hardcode apps/ subdirectories, use wildcards somehow
# FIXME: copy everything in one step somehow
ADD apps/a/mix.exs apps/a/
ADD apps/b/mix.exs apps/b/
ADD apps/c/mix.exs apps/c/
ADD mix.exs mix.lock ./
RUN mix local.hex --force && mix deps.get && mix deps.compile
<skipped>
ADD . .
RUN mix compile && mix phx.digest
CMD [ "mix", "phx.server" ]
Is there a way to ADD or COPY all the apps/*/mix.exs in a single step, preserving their directory structure (so that each apps/foo/mix.exs ends up in $WORKDIR/apps/foo/mix.exs)?
To the best of my knowledge, there is no obvious way to do what I want. E. g. saying something like ADD apps/*/mix.exs mix.exs mix.lock ./ just copies everything into the WORKDIR (and the files overwrite each other).
Maybe there is some non-obvious way?
A workaround I could think of is to add the following in the .dockerignore file
apps/**
!apps/*/mix.exs
and in the dockerfile,
COPY apps/ /apps
COPY mix.exs mix.lock /
or even
COPY . /
NOTE
If you don't have to stick to pure dockerfile solution, wrap your docker build in a shell script or makefile and use cp is also a good way to do it.
First I would like to thank you for helping me.
Please be kind as it is my first post here.
I am trying to build a docker container but the COPY step seems not to work properly.
Here is my DockerFile
FROM maven:3.3-jdk-8
RUN apt-get update
RUN apt-get -y install vim tree unzip
WORKDIR /usr/src/app
COPY . /usr/src/app/
Here is my directory tree
├── Dockerfile
├── events
│ ├── backend drwxr-xr-x
│ │ ├── stuff drwxr-xr-x
│ ├── frontend drwxr-xr-x
│ │ ├── stuff drwxr-xr-x
I am part of the docker group.
The weird thing happening is that stuff from backend is copied but not stuff from frontend.
I remember using COPY this way and it worked.
I might have missed a behavior of COPY.
Thanks for helping me.
Cheers.
Did you check if you have .dockerignore in your project?
Docker COPY will silently ignore all patterns mentioned in .dockerignore.