Cache Rust dependencies with Docker build and lib.rs - docker

Question

I've been trying to create a Dockerfile for my rust build that would allow me to build the application separately from the dependencies as demonstrated here:
Cache Rust dependencies with Docker build
However this doesn't not seem to be working for me, having a slightly different working tree with the lib.rs file. My Dockerfile is laid out like so:
FROM rust:1.60 as build
# create a new empty shell project
RUN USER=root cargo new --bin rocket-example-pro
WORKDIR /rocket-example-pro
# create dummy lib.rs file to build dependencies separately from changes to src
RUN touch src/lib.rs
# copy over your manifests
COPY ./Cargo.lock ./Cargo.lock
COPY ./Cargo.toml ./Cargo.toml
RUN cargo build --release --locked
RUN rm src/*.rs
# copy your source tree
COPY ./src ./src
# build for release
RUN rm ./target/release/deps/rocket_example_pro*
RUN cargo build --release --locked ## <-- fails
# our final base
FROM rust:1.60
# copy the build artifact from the build stage
COPY --from=build /rocket-example-pro/target/release/rocket_example_pro .
# set the startup command to run your binary
CMD ["./rocket_example_pro"]
As you can see initially I copy over the toml files and perform a build, similarly to the previously demonstrated. However with my project structure being slightly different I seem to be having an issues, as my main.rs pretty much only has one line that calls the main method in my lib.rs. lib.rs is also defined in my toml file that gets copied before building dependencies and requires me to touch the lib.rs file for it to not fail building here with it otherwise being missing.
Its at the second build step that I can't seem to resolve, after I've copied over the actual source files to build the application, I am getting the error message
Compiling rocket_example_pro v0.1.0 (/rocket-example-pro)
error[E0425]: cannot find function `run` in crate `rocket_example_pro`
--> src/main.rs:3:22
|
3 | rocket_example_pro::run().unwrap();
| ^^^ not found in `rocket_example_pro`
When performing these steps myself in a empty directory I don't seem to encounter the same errors myself, instead the last step succeeds, but the produced rocket-example-pro executable file still seems to be the shell example project only printing 'Hello world' and not the rocket application i copy over before the second build.
As far as i can figure it seems that the first build is affecting the second, perhaps when I touch the lib.rs file in the dummy shell project, it builds it without the run() method? so when the second one starts, it doesn't see the run method because its empty? but this doesn't make much sense to me as I have copied over the lib.rs file with the run() method inside it.
here's what the toml file looks like if it helps:
[package]
name = "rocket_example_pro"
version = "0.1.0"
edition = "2021"
[[bin]]
name = "rocket_example_pro"
path = "src/main.rs"
[lib]
name = "rocket_example_pro"
path = "src/lib.rs"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
...

Answer 1

(I couldn't reproduce this at first. Then I noticed that having at least one dependency seems to be a necessary condition.)
With the line
RUN rm ./target/release/deps/rocket_example_pro*
you're forcing rebuild of the rocket_example_pro binary. But the library will remain as built from the first empty file. Try changing to
RUN rm ./target/release/deps/librocket_example_pro*
Though personally, I think deleting random files from the target directory is a terribly hacky solution. I'd prefer to trigger the rebuild of the lib by adjusting the timestamp:
RUN touch src/lib.rs && cargo build --release --locked ## Doesn't fail anymore
For a clean solution, have a look at cargo-chef.
[Edit:] So what's happening here?
To decide whether to rebuild, cargo seems to compare the mtime of the target/…/*.d to the mtime of the files listed in the content of the *.d files.
Probably, src/lib.rs was created first, and then docker build was run. So src/lib.rs was older than target/release/librocket_example_pro.d, leading to target/release/librocket_example_pro.rlib not being rebuilt after copying in src/lib.rs.
You can partially verify that that's what's happening.
With the original Dockerfile, run cargo build, see it fail
Run echo >> src/lib.rs outside of docker to update its mtime and hash
Run cargo build, it succeeds
Note that for step 2, updating mtime with touch src/lib.rs is not sufficient because docker will set the mtime when COPYing a file, but it will ignore mtime when deciding whether to use a cached step.