I have the following Dockerfile which builds an image for my Go project.
FROM golang:1.11.2-alpine3.8 as go-compile
RUN apk update && apk add git
RUN mkdir /app
COPY src/ /app
WORKDIR /app
RUN go get github.com/joho/godotenv
RUN go build -o main .
FROM alpine:latest
RUN mkdir /app
COPY --from=go-compile /app/main /app/main
CMD ["/app/main"]
The image builds, but my ".env" file is not included in the Docker image.
I've tried to copy the ".env" from the src folder into the image using COPY src/.env /app/.env, but still the Go code can't find/read the file.
How can I include the ".env" file, or in fact any other non Go file?
You cannot include non-go files in the go build process. The Go tool doesn't support "embedding" arbitrary files into the final executable.
You should use go build to build your executable then, any non-go files, e.g. templates, images, config files, need to be made available to that executable. That is; the executable needs to know where the non-go files are on the filesystem of the host machine on which the go program is running, and then open and read them as needed. So forget about embeding .env into main, instead copy .env together with main to the same location from which you want to run main.
Then the issue with your dockerfile is the fact that the target host only copies the final executable file from go-compile (COPY --from=go-compile /app/main /app/main), it doesn't copy any other files that are present in the go-compile image and therefore your main app cannot access .env since they are not on the same host.
As pointed out in the comments by #mh-cbon, there do exist 3rd-party solutions for embedding non-go files into the go binary, one of which is gobuffalo/packr.
You can inject a dotenv file or single variables into your service using docker compose:
version: "3.9"
services:
backend:
...
environment:
- NODE_ENV=production
env_file:
- ./backend/.env
Related
I want to copy the contents of a parent directory (relative to the position of the Dockerfile) into my image.
This is the folder structure:
app-root/
docker/
php81aws/
some-folder
Dockerfile
start-container
supervisord.conf
app_folders
app_files
I'm calling docker build as follows:
app-root#> docker build -t laravel -f docker/php81aws/Dockerfile .
Or from docker compose with:
services:
laravel:
build:
context: ./
dockerfile: docker/php81aws/Dockerfile
Therefore, the context should be in the app-root directory.
In the dockerfile, I'm using COPY like so:
COPY docker/php81aws/start-container /usr/local/bin/start-container
COPY docker/php81aws/supervisord.conf /etc/supervisor/conf.d/supervisord.conf
RUN chmod +x /usr/local/bin/start-container
COPY --chown=www:www . /var/www/
It always gives me this error:
failed to compute cache key: "/start-container" not found: not found
I tried to use COPY ./docker/php81aws/start-container and COPY start-container but the error is always the same. Of course copying the parent directory also fails.
You mention in a comment that your top-level app-root directory has a .dockerignore file that excludes the entire docker directory. While the Dockerfile will still be available, nothing else in that tree can be COPYed into the image, and if you COPY ./ ./ to copy the entire build context into the image, that directory won't be present.
Deleting this line from the .dockerignore file should fix your issue.
In general you want the .dockerignore file to include anything that's part of your host build-and-test environment, but should not by default be included in an image, possibly because the Dockerfile is going to rebuild it. In a Node context, for example, you almost always want to exclude the host's node_modules directory, or in a Java context often the Gradle build or Maven target directories. Anything you do want to include in the image needs to not be listed in .dockerignore.
I'd like to instruct Docker to COPY my certificates from the local /etc/ folder on my Ubuntu machine.
I get the error:
COPY failed: file not found in build context or excluded by
.dockerignore: stat etc/.auth_keys/fullchain.pem: file does not exist
I have not excluded in .dockerignore
How can I do it?
Dockerfile:
FROM nginx:1.21.3-alpine
RUN rm /etc/nginx/conf.d/default.conf
RUN mkdir /etc/nginx/ssl
COPY nginx.conf /etc/nginx/conf.d
COPY ./etc/.auth_keys/fullchain.pem /etc/nginx/ssl/
COPY ./etc/.auth_keys/privkey.pem /etc/nginx/ssl/
WORKDIR /usr/src/app
I have also tried without the dot --> same error
COPY /etc/.auth_keys/fullchain.pem /etc/nginx/ssl/
COPY /etc/.auth_keys/privkey.pem /etc/nginx/ssl/
By placing the folder .auth_keys next to the Dockerfile --> works, but not desireable
COPY /.auth_keys/fullchain.pem /etc/nginx/ssl/
COPY /.auth_keys/privkey.pem /etc/nginx/ssl/
The docker context is the directory the Dockerfile is located in. If you want to build an image that is one of the restrictions you have to face.
In this documentation you can see how contexts can be switched, but to keep it simple just consider the same directory to be the context. Note; this also doesn't work with symbolic links.
So your observation was correct and you need to place the files you need to copy in the same directory.
Alternatively, if you don't need to copy them but still have them available at runtime you could opt for a mount. I can imagine this not working in your case because you likely need the files at startup of the container.
#JustLudo's answer is correct, in this case. However, for those who have the correct files in the build directory and still seeing this issue; remove any trailing comments.
Coming from a C and javascript background, one may be forgiven for assuming that trailing comments are ignored (e.g. COPY my_file /etc/important/ # very important!), but they are not! The error message won't point this out, as of my version of docker (20.10.11).
For example, the above erroneous line will give an error:
COPY failed: file not found in build context or excluded by .dockerignore: stat etc/important/: file does not exist
... i.e. no mention that it is the trailing # important! that is tripping things up.
It's also important to note that, as mentioned into the docs:
If you use STDIN or specify a URL pointing to a plain text file, the system places the contents into a file called Dockerfile, and any -f, --file option is ignored. In this scenario, there is no context.
That is, if you're running build like this:
docker build -t dh/myimage - < Dockerfile_test
Any COPY or ADD, having no context, will throw the error mentioned or another similar:
failed to compute cache key: "xyz" not found: not found
If you face this error and you're piping your Dockerfile, then I advise to use -f to target a custom Dockerfile.
docker build -t dh/myimage -f Dockerfile_test .
(. set the context to the current directory)
Here is a test you can do yourself :
In an empty directory, create a Dockerfile_test file, with this content
FROM nginx:1.21.3-alpine
COPY test_file /my_test_file
Then create a dummy file:
touch test_file
Run build piping the test Dockerfile, see how it fails because it has no context:
docker build -t dh/myimage - < Dockerfile_test
[..]
failed to compute cache key: "/test_file" not found: not found
[..]
Now run build with -f, see how the same Dockerfile works because it has context:
docker build -t dh/myimage -f Dockerfile_test .
[..]
=> [2/2] COPY test_file /my_test_file
=> exporting to image
[..]
Check your docker-compos.yml, it might be changing the context directory.
I had a similar problem, with the only clarification: I was running Dockerfile with docker-compos.yml
This is what my Dockerfile looked like when I got the error:
FROM alpine:3.17.0
ARG DB_NAME \
DB_USER \
DB_PASS
RUN apk update && apk upgrade && apk add --no-cache \
php \
...
EXPOSE 9000
COPY ./conf/www.conf /etc/php/7.3/fpm/pool.d #<--- an error was here
COPY ./tools /var/www/ #<--- and here
ENTRYPOINT ["sh", "/var/www/start.sh"]
This is part of my docker-compose.yml where I described my service.
wordpress:
container_name: wordpress
build:
context: . #<--- the problem was here
dockerfile: requirements/wordpress/Dockerfile
args:
DB_NAME: ${DB_NAME}
DB_USER: ${DB_USER}
DB_PASS: ${DB_PASS}
ports:
- "9000:9000"
depends_on:
- mariadb
restart: unless-stopped
networks:
- inception
volumes:
- wp:/var/www/
My docker-compos.yml was changing the context directory. Then I wrote a new path in the Dockerfile and it all worked.
COPY ./requirements/wordpress/conf/www.conf /etc/php/7.3/fpm/pool.d
COPY ./requirements/wordpress/tools /var/www/
project structure
FWIW this same error shows up when running gcloud builds submit if the files are included in .gitignore
Have you tried doing a simlink with ln -s to the /etc/certs/ folder in the docker build directory?
Alternatively you could have one image that has the certificates and in your image you just COPY FROM the docker image having the certs.
I had the same error. I resolved it by adding this to my Docker build command:
docker build --no-cache -f ./example-folder/example-folder/Dockerfile
This repoints Docker to the home directory. Even if your Dockerfile seems to run (i.e. the system seems to locate it and starts running it), I found I needed to have the home directory pre-defined above, for any copying to happen.
Inside my Dockerfile, I had the file copying like this:
COPY ./example-folder/example-folder /home/example-folder/example-folder
I merely had quoted the source file while building a windows container, e.g.,
COPY "file with space.txt" c:/some_dir/new_name.txt
Docker doesn't like the quotes.
I am using docker-compose to run my golang app.
Here is my Dockerfile
FROM golang:1.13
WORKDIR /app
COPY go.mod ./
RUN go mod download
COPY . .
RUN go build -o main .
CMD ["/app/main"]
and my docker-compose.yml
version: '3.7'
services:
app:
build: ./myapp
container_name: myapp
volumes:
- ./myapp:/app
When I run docker-compose build The main file not appear on myapp dir.
docker-compose up myapp not work, because main file not found.
But docker run mypp can work. How can i build main.go in dockerfile and stay the main in my host?
A docker-compose build step runs to completion, ignoring everything else in your docker-compose.yml file. The resulting image is then executed taking these options into account. So the sequence you show does two things:
Build the image, COPYing its source code in, and producing a /app/main binary inside the image; nothing on the host system is affected.
Run a container based on that image, but mounting the current ./myapp directory and its contents over /app in the container, hiding anything there.
In this sequence of steps nothing ever gets copied out of the container, you are only ever pushing things into Docker space. If you'd like to run the binary you built, delete the volumes: mount to let the image in the binary run.
In comments you suggest your goal is to just build the Go program and get a binary out. You don't want Docker for that; the Go Getting Started page has instructions for how to install a Go toolchain, and you can just go install your program.
I am facing an issue of large docker build context because of my project structure. In my root directory I have lib folder for common code and folders of micro-services. Now I want to build for miscroservice1 to include only lib folder and to ignore other microservices.
I am running docker build command in root folder because running command in microservice folder giving error Forbidden path outside the build context
rootFolder
-- lib
-- microservice1/Dockerfile
-- microservice2/Dockerfile
-- microservice3/Dockerfile
I have two solutions but didn't try for now
To add symlinks for lib in my microservice folder
To write script for each docker build to add lib folder in specific microservice folder and then run docker build.
I am trying the above two solutions. Can anyone suggest any best practice?
You can create .dockerignore in your root directory and add
microservice1/
microservice2/
microservice3/
to it, just like .gitignore does during tracking files, docker will ignore these folders/files during the build.
Update
You can include docker-compose.yml file in your root directory, look at docker-compose for all the options, such as setting environment, running a specific command, etc, that you can use during the build process.
version: "3"
services:
microservice1:
build:
context: .
dockerfile: ./microservice1/Dockerfile
volumes:
- "./path/to/share:/path/to/mount/on/container"
ports:
- "<host>:<container>"
links:
- rootservice # defines a dns record in /etc/hosts to point to rootservice
microservice2:
build:
context: .
dockerfile: ./microservice2/Dockerfile
volumes:
- "./path/to/share:/path/to/mount/on/container"
ports:
- "<host>:<container>"
links:
- rootservice # defines a dns record in /etc/hosts to point to rootservice
- microservice1
rootservice:
build:
context: .
dockerfile: ./Dockerfile
volumes:
- "./path/to/share:/path/to/mount/on/container"
ports:
- "<host>:<container>"
depends_on:
- microservice1
- microservice2
ports:
- "<host1>:<container1>"
- "<host2>:<container2>"
This will be your build recipe for your microservices, you can now run docker-compose build to build all your images.
If the only tool you have is Docker, there aren't very many choices. The key problem is that there is only one .dockerignore file. That means you always have to use your project root directory as the Docker context directory (including every services' sources), but you can tell Docker which specific Dockerfile within that to use. (Note that all COPY directives will be relative to the rootFolder in this case.)
docker build rootFolder -f microservice1/Dockerfile -t micro/service1:20190831.01
In many languages there is a way to package up the library (C .a, .h, and .so files; Java .jar files; Python wheels; ...). If your language supports that, another option is to build the library, then copy (not symlink) the library into each service's build tree. Using Python's wheel format as an example:
pip wheel ./lib
cp microlib.whl microservice1
docker build microservice1 -t micro/service1:20190831.01
# Dockerfile needs to
# RUN pip install ./microlib.whl
Another useful variant on this is a manual multi-stage build. You can have lib/Dockerfile pick some base image, and then install the library into that base image. Then each service's Dockerfile starts FROM the library image, and has it preinstalled. Using a C library as an example:
# I am lib/Dockerfile
# Build stage
FROM ubuntu:18.04 AS build
RUN apt-get update && apt-get install build-essential
WORKDIR /src
COPY ./ ./
RUN ./configure --prefix=/usr/local && make
# This is a typical pattern implemented by GNU Autoconf:
# it actually writes files into /src/out/usr/local/...
RUN make install DESTDIR=/src/out
# Install stage -- service images are based on this
FROM ubuntu:18.04
COPY --from=build /src/out /
RUN ldconfig
# I am microservice1/Dockerfile
ARG VERSION=latest
FROM micro/lib:${VERSION}
# From the base image, there are already
# /usr/local/include/microlib.h and /usr/local/lib/libmicro.so
COPY ...
RUN gcc ... -lmicro
CMD ...
There is also usually an option (again, depending on your language and its packaging system) to upload your built library to some server, possibly one you're running yourself. (A Python pip requirements.txt file can contain an arbitrary HTTP URL for a wheel, for example.) If you do this then you can just declare your library as an ordinary dependency, and this problem goes away.
Which of these works better for you depends on your language and runtime, and how much automation of multiple coordinated docker build commands you're willing to do.
I tried to copy some files from source to destination (flask app) in a dockerfile but it seems things are not working as expected when building the image. With last 2 line showing:
Step 3 : COPY pkl_objects/* /home/jovyan/work/movieclassifier/pkl_objects/
No source files were specified
This is the docker file.
FROM jupyter/datascience-notebook
RUN pip install flask flask-wtf
COPY pkl_objects/* /home/jovyan/work/movieclassifier/pkl_objects/
COPY static/* /home/jovyan/work/movieclassifier/static/
COPY templates/* /home/jovyan/work/movieclassifier/templates/
COPY app.py /home/jovyan/work/movieclassifier
COPY reviews.sqlite /home/jovyan/work/movieclassifier
COPY vectorizer.py /home/jovyan/work/movieclassifier
WORKDIR /home/jovyan/work/movieclassifier
ENV FLASK_APP=app.py
# ENV FLASK_DEBUG=0
CMD ["flask", "run", "--host=0.0.0.0"]
Looks like there are no files in the pkl_objects folder, and when the wildcard (*) is expanded, it results in no source files being specified.
Maybe you could add an empty file in there, so that when the wildcard picks up files, you at least get one source file.
Example file could be: .nonempty or something like that.