How can I build a Docker image of my Rust project containing local dependencies?
My Cargo.toml looks like this:
[package]
name = "backend"
version = "0.1.0"
edition = "2021"
[dependencies]
actix-web = "4"
juniper = "0.15.10"
my_libs = {path = "/Users/dev/projects/api/api_libs/"}
And my Dockerfile:
FROM rust:1.60.0-slim-buster as builder
WORKDIR /app
ENV CARGO_HOME=/workdir/.cargo
COPY ./Cargo.toml ./Cargo.lock ./
COPY . .
RUN cargo build --release
When I run docker build, I get the following error:
error: failed to get my_libs as a dependency of package backend v0.1.0 (/app)
How can I fix this?
The path you've given will be interpreted by docker as a path inside the container, so it's normal that it does not exist.
You're also not able to copy files that do not exist within the directory of the Dockerfile.
Either publish your lib on crates.io or make it available via a git url (i.e.: github) which can be installed with cargo.
Or copy the lib locally so that you can copy it into your container.
Well of course you have to copy your local dependencies to the container as well.
If I recall correctly docker doesn't work that great with absolute paths so I've moved the api_libs folder into the one for external and adjusted the Cargo.toml to:
api_libs = {path = "./api_libs/"}
now when we do the copy
COPY . .
we copy the dependencies as well and thus docker build . succeeds.
By the way your first COPY ./Cargo.toml ./Cargo.lock ./ is redundant since COPY . . would copy them anyways and you do nothing in between.
Related
I'm a newbie to docker, sorry if my question is too basic. I saw dockerfile like this:
FROM diamol/maven AS builder
WORKDIR /usr/src/iotd
COPY pom.xml .
RUN mvn -B dependency:go-offline
COPY . .
RUN mvn package
FROM diamol/openjdk
WORKDIR /app
COPY --from=builder /usr/src/iotd/target/iotd-service-0.1.0.jar .
EXPOSE 80
ENTRYPOINT ["java", "-jar", "/app/iotd-service-0.1.0.jar"]
I'm confused about COPY . . instruction, what does the first period and second period COPY . . mean?
Also, if I want to copy all files of the current working directory from my host machine into the image, then how can I modify COPY . . so that the first period means currenty directory of my machine?
In the Dockerfile COPY directive, the last argument is the path inside the container, relative to the current WORKDIR. All of the preceding arguments are paths inside the build context, the host directory passed as an argument to docker build.
I want to copy all files of the current working directory from my host machine into the image, then how can I modify COPY . . ...?
You probably don't need to. So long as you docker build . naming the current directory . as the last argument, that's exactly what COPY . . does. That instruction means to copy . – the entirety of the build context, from the original host system – to . – the current directory, inside the image.
WORKDIR /usr/src/iotd # `COPY anything .` will put it here inside the image
COPY pom.xml . # Copy a single file into that WORKDIR
COPY . . # Copy the entire build context into the WORKDIR
I've mentioned "build context" a couple of times. That is the directory argument to docker build
docker build \
-t myname/some-image: tag \
. # <--- the build context directory
or that you specify in a docker-compose.yml file
version: '3.8'
services:
one:
build: ./one # <-- this directory
two:
build:
context: ./two # <-- this directory
except that the files mentioned in a .dockerignore file are removed first.
In the question title you also ask
does dockerfile's instruction execute in order?
They do. The newer BuildKit backend has some capability to execute build stages not necessarily in the order they're written, but it ensures that you get the same results as if all of the COPY and RUN instructions from a previous stage had run before a COPY --from=... in a later stage happens.
From my perspective, one of the best ways to know all the details for COPY and WORKDIR docker commands is to go through following official documentation.
You can either search for COPY and WORKDIR keywords on the home page at below first link or please refer to last two links and find all the details including examples.
https://docs.docker.com/engine/reference/builder/
https://docs.docker.com/engine/reference/builder/#copy
https://docs.docker.com/engine/reference/builder/#workdir
This question is asked before yet After reviewing the answers I am still not able to copy the solution.
I am still new to docker and after watching tutorials and following articles I was able to create a Dockerfile for an existing GitHub repository.
I started by using the nearest available image as a base then adding what I need.
from what I read the problem is in WORKDIR and CMD commands
This is error message:
python: can't open file 'save_model.py': [Errno 2] No such file or directory*
This is my Dockerfile:
# syntax=docker/dockerfile:1
FROM tensorflow/serving:2.3.0-rc0-devel-gpu
WORKDIR app
COPY requirements-gpu.txt .
# install dependencies
RUN pip install -r requirements-gpu.txt
# copy the content of the local src directory to the working directory
COPY /home/pc/Desktop/yolo4_deep .
# command to run on container start
CMD ["python","./app/save_model.py","./app/object_tracker.py" ]
src
save_model.py
object_tracker.py
...
requirements.txt
Dockerfile
I tried WORKDIR command to set the absolute path: WORKDIR /home/pc/Desktop/yolo4_Deep_sort_nojupitor the result was Same Error.
I see multiple issues in your Dockerfile.
COPY /home/pc/Desktop/yolo4_deep .
The COPY command copies files from your local machine to the container. The path on your local machine must be path relative to your build context. The build context is the path you pass in when you run docker build . — in this case the . (the current directory) is the build context. Also the local machine path can only reference files located under the build context — i.e. paths containing .. (parent directory) or / (root directory) are not allowed.
WORKDIR app
WORKDIR sets the path inside the container not on your local machine. So WORKDIR /app means that all commands — RUN, CMD, ENTRYPOINT — will be executed from the /app directory.
CMD ["python","./app/save_model.py","./app/object_tracker.py" ]
As mentioned above WORKDIR /app causes all operations to be executed from the /app directory. So ./app/save_model.py is actually translated as /app/app/save_model.py.
Thanks for help Everyone.
As I mentioned earlier I'm beginner in the docker world. I solved the issue by editing the copy command.
# syntax=docker/dockerfile:1
FROM tensorflow/serving:2.3.0-rc0-devel-gpu
WORKDIR /home/pc/Desktop/yolo4_deep
COPY requirements-gpu.txt .
# install dependencies
RUN pip install -r requirements-gpu.txt
# copy the content of the local src directory to the working directory
COPY src/ .
# command to run on container start
ENTRYPOINT ["./start.sh"]
I have two go modules github.com/myuser/mymainrepo and github.com/myuser/commonrepo
Here is how i have the files in my local computer
- allmyrepos
- mymainrepo
- Dockerfile
- go.mod
- commonrepo
- go.mod
mymainrepo/go.mod
...
require (
github.com/myuser/commonrepo
)
replace (
github.com/myuser/commonrepo => ../commonrepo
)
It works well i can do local development with it. Problem happens when i'm building docker image of mymainrepo
mymainrepo/Dockerfile
...
WORKDIR /go/src/mymainrepo
COPY go.mod go.sum ./
RUN go mod download
COPY ./ ./
RUN go build -o appbinary
...
Here replace replaces github.com/myuser/commonrepo with ../commonrepo but in Docker /go/src/commonrepo does not exists.
I'm building the Docker image on CI/CD which needs to fetch directly from remote github url but i also need to do local development on commonrepo. How can i do both ?
I tried to put all my files in GOPATH so it's ~/go/src/github.com/myuser/commonrepo and go/src/github.com/myuser/mymainrepo. And i removed the replace directive. But it looks for commonrepo inside ~/go/pkg/mod/... that's downloaded from github.
Create two go.mod files: one for local development, and one for your build. You can name it go.build.mod for example.
Keep the replace directive in your go.mod file but remove it from go.build.mod.
Finally, in your Dockerfile:
COPY go.build.mod ./go.mod
COPY go.sum ./
I still can't find other better solution even the voted answer doesn't work for me. Here a trick I've done that workaround for me. This is an example structure for doing this:
|---sample
| |---...
| |---go.mod
| |---Dockerfile
|---core
| |---...
| |---go.mod
We know that docker build error when it can't find our local module. Let's make one in the builder process:
# Use the offical golang image to create a binary.
# This is based on Debian and sets the GOPATH to /go.
# https://hub.docker.com/_/golang
FROM golang:1.16.3-buster AS builder
# Copy core library
RUN mkdir /core
COPY core/ /core
# Create and change to the app directory.
WORKDIR /app
# Retrieve application dependencies.
# This allows the container build to reuse cached dependencies.
# Expecting to copy go.mod and if present go.sum.
COPY go.* ./
RUN go mod download
# Copy local code to the container image.
COPY . ./
# Build the binary
RUN go build -o /app/sample cmd/main.go
...
...
Ok, our working dir is /app and our core lib placed next to it /core.
Let's make a trick when build a docker image! Yeah, you know it.
cp -R ../core . && docker build --tag sample-service . && rm -R core/
Update
A way better, create a Makefile next to Dockerfile, with content below:
build:
cp -R ../core .
docker build -t sample-service .
rm -R core/
Then command, make build in the sample directory.
You can create make submit or make deploy commands as you like to.
=> Production ready!
Be aware that if there's an error occurs during docker build process, it won't delete back the core folder we have copied to sample.
Pls let me know if you find any better solution. ;)
I am really curious to how to interpret and debug with the following error:-
C:\users\project>docker-compose build
Step 6/15 : COPY *.csproj ./
ERROR: Service 'invoiceservice' failed to build: COPY failed: no source files were specified
This is particular micro-service as i have few more such services.
docker file :-
FROM mcr.microsoft.com/dotnet/core/runtime:2.2 AS base
WORKDIR /app
EXPOSE 80
FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build
WORKDIR /src
COPY *.csproj ./
RUN dotnet restore -s https://api.nuget.org/v3/index.json -s https://www.myget.org/F/autoweb/api/v3/index.json
COPY . .
WORKDIR /src/src/InvoiceManagement/InvoiceService
RUN dotnet publish -c Release -o out
FROM build AS publish
FROM base AS final
WORKDIR /app
COPY --from=publish /app/out .
ENTRYPOINT ["dotnet", "InvoiceService.dll"]
Interesting part is when I build this from Visual Studio IDE, its being built fine but it does not build on CLI.
docker compose file:-
invoiceservice:
image: ${DOCKER_REGISTRY-}invoiceservice
build:
context: .
dockerfile: src/InvoiceManagement/InvoiceService/Dockerfile
I don't understand why CLI could not find the source location and copy where as VS works fine. Any clue???
It's likely an issue with your Docker context. Commands in Dockerfiles are relative to where the docker/docker-compose CLI tool is run, not relative to the Dockerfile location.
In other words, if you run the command from the solution root, then your csproj file is actually under ./src/InvoiceManagement/InvoiceService. As a result, *.csproj finds no files, because there's no project files literally in your solution root.
I tried replicating your problem and I was able to copy all the files successfully (instead of .csproj, I used .txt). The problem occurred when there was no txt file to copy. My COPY command failed with exactly the same message as yours.
Now, why is VS able to build the image successfully? That is because VS build the project first! When the project's build procedure is completed, a .csproj file is generated and that gets copied to the image.
To confirm the results, ls your current directory (when the build fails from command line) and see if there is any .csproj file in that directory.
We are working with two project at the moment:
1 C++ based project
2 Nodejs based project
These two projectes are separated which means they have different codebase(git repoitory) and working directory.
C++ project will produce a node binding file .node which will be used by Nodejs project.
And we try to build an docker image for the Nodejs project with multi-stage like this:
from ubuntu:18.04 as u
WORKDIR /app
RUN apt-get........
copy (?) . #1 copy the c++ source codes
RUN make
from node:10
WORKDIR /app
copy (?) . #1 copy the nodejs cource codes
RUN npm install
copy --from=u /app/dist/xx.node ./lib/
node index.js
And I will build the image by docker build -t xx (?) #2.
However as commented in the dockerfile and the command, how to setup the context directory(see comment #2)? Since it will affect the path in the dockerfile (see comment #1).
Also which project should I put inside for the above dockerfile?
You will have two options on this, as the limiting factor is that Docker only allows copying from the same directory as the Dockerfile:
Create a new Repository
You can either create a new repo and use your repos as submodules or just for the Dockerfile (Than you would have to copy both repos into the root folder at build time). In the End what you want to achieve is the following structure:
/ (root)
|-- C-plus-plus-Repo
|-- |-- <Files>
|-- Node-Repo
|-- |-- <Files>
|-- Dockerfile
Than you can build your project with:
from ubuntu:18.04 as u
WORKDIR /app
RUN apt-get........
#1 copy the c++ source files
copy ./C-plus-plus-Repo .
RUN make
from node:10
WORKDIR /app
#1 copy the nodejs cource codes
copy ./Node-Repo .
RUN npm install
copy --from=u /app/dist/xx.node ./lib/
node index.js
In the root Directory execute:
docker build -t xx .
Build your staging containers extra
Docker allows to copy from an external container as stage.
So you can build the C++ container in your C++ Repo root
from ubuntu:18.04 as u
WORKDIR /app
RUN apt-get........
#1 copy the c++ source files
copy . .
RUN make
and Tag it:
# Build your C++ Container in root of the c++ repo
docker build . -t c-stage
then copy the files from it, using the tag (in your node Repo root):
from node:10
WORKDIR /app
#1 copy the nodejs source files
copy . .
RUN npm install
# Use the Tag-Name of the already build container "c-stage"
copy --from=c-stage /app/dist/xx.node ./lib/
node index.js
Both build steps can be executed from their respective repo roots.
Creating a deploy project with git submodules
How about creating a deploy project using git submodules?
This project would only exist for building the docker image and contains the Dockerfile and both of your projects as git submodules.
Since you not just copy the two projects, but manage them with git, you can always keep them up to date with git submodules update --remote, but note that this leaves your submodule in a detached head state. However this is not a problem as long as you do not try to update your C++ project or the node project from the deploy project.
You can create the project with the following commands:
mkdir deploy_project && cd deploy_project
git init
git submodule add git#your-gitserver.com:YourName/YourCppProject.git cpp_project
git submodule add git#your-gitserver.com:YourName/YourNodeProject.git nodejs_project
Then you can simply add the paths to the subprojects to your dockerfile and build the image in the root directory of the deploy project.
The dockerfile would look like this
FROM ubuntu:18.04 as u
WORKDIR /app
RUN apt-get........
COPY cpp_project/ . #1 copy the c++ source codes
RUN make
FROM node:10
WORKDIR /app
COPY nodejs_project/ . #1 copy the nodejs cource codes
RUN npm install
COPY --from=u /app/dist/xx.node ./lib/
You can use ADD command (her context watching to host directory where Dockerfile placed. It will copy everything placed in same directory as Dockerfile in host machine (in this case content of cpp_app directory) into the docker container.
...
ADD cpp_app /place/to/build
WORKDIR /place/to/build
RUN make
RUN mv result_file /place/where/result_file/have/to/be
WORKDIR /place/where/result_file/have/to/be
... execute your nodejs stuff