I would to build a Docker image using multi-stage.
We are using yarn 2 and Zero installs feature which stores dependencies in .yarn/cache under zip format.
To minimize the size of my Docker image, I would like to only have the production dependencies.
Previsously, we would do
yarn install --non-interactive --production=true
But by doing that with a former version of yarn, we don't benefit from the .yarn/cache folder and it takes time to download dependencies whereas there are already here but not readable by the former version of yarn.
Is there a way to tell yarn 2 to get only production dependencies from the .yarn/cache folder and put it into another one ? Thus I could copy this folder inside my image and save time and space.
Related
I have a multistage Docker build file. The first stage creates an installable package (e.g. a .deb file). The second stage needs only to install the .deb package.
Here's an example:
FROM debian:buster AS build_layer
COPY src/ /src/
WORKDIR /src
RUN ./build_deb.sh
# A lot of stuff happens here and a big huge .deb package pops out.
# Suppose the package is 300MB.
FROM debian:buster AS app_layer
COPY --from=build_layer /src/myapp.deb /
RUN dpkg -i /myapp.deb && rm /myapp.deb
ENTRYPOINT ["/usr/bin/myapp"]
# This image will be well over 600MB, since it contains both the
# installed package as well as the deleted copy of the .deb file.
The problem with this is that the COPY stage runs in its own layer, and drops the large .deb package into the final build context. Then, the next step installs the package and removes the .deb file. However, since the COPY stage has to execute independently, the .deb package still takes up room in the final image. If it were a small package you might just deal with it, but in my case the package file is hundreds of MB, so its presence in the final build layers does increase the image size appreciably with no benefit.
There are related posts on SO, such as this one which discusses files containing secrets, and this one which is for copying a large installer from outside the container into it (and the solution for that one is still kinda janky, requiring you to run a temporary local http server). However neither of these address the situation of needing to copy from another build stage but not retain the copied file in the final package.
The only way I could think of to do this would be to extend the idea of a web server and make available an SFTP or similar server so that the build layer can upload the package somewhere. But this also requires extra infrastructure, and now you're also dealing with SSH secrets and such, and this starts to get real complex and is a lot less reproducible on another developer's system or in a CI/CD environment.
Alternatively I could use the --squash option in BuildKit, but this ends up killing the advantages of the layer system. I can't then reuse similar layers across multiple images (e.g. the image now can't take advantage of the fact that the Debian base image might exist on the end user's system already). This would minimize space usage, but wouldn't be ideal for a lot of other reasons.
What's the recommended way to approach this?
I am new to dockers, well we started working on docker file but I am stuck on how to maintain different versions of dependent software's of our web app.
suppose our web app uses crystal reports 1.X version, in runtime.
In future , if want to update version of crystal report to 1.2.X.
In these scenarios how a docker file and these dependent software's should be maintained(although version we can directly update in docker file)?
Should docker file be parametrised for the versions?
What would be the best approach?
Use your application language's native package dependency system (a Ruby Gemfile, Python Pipfile or requirements.txt, Node package.json, Scala build.sbt, ...). In a non-Docker development environment, maintain these dependencies the same way you would without Docker. When you go to translate this into a Dockerfile, copy these description files into the image and install them.
A near-universal Javascript Dockerfile, for example, would look like
FROM node:12
WORKDIR /app
# Copy in and install dependencies
COPY package.json yarn.lock .
RUN yarn install
# Copy in the rest of the application; build and set up to run it
COPY . .
RUN yarn build
EXPOSE 3000
CMD yarn start
If a dependency changed, you'd use a command like yarn up to update the package.json and yarn.lock files in your non-Docker development environment, and when you went to re-docker build, those updated files would update the dependency in the built image.
I have a Django app that uses Docker and has a bunch of library dependencies in the requirements.txt Any time I add a new dependency, I have to re-build the image and it downloads all of the dependencies from scratch. Is there a way to cache dependencies when building a docker image?
The most common solution is to create a new base image on top the one that already has all the dependencies. However, if you update all your dependencies very regularly, it might be easier to set a CI process where you build a new base image every so often (every week? every day?)
Multistage might not work in Docker because the dependencies are part of your base image, so then you do docker build . it will always want to pull all the dependencies when you do a pip3 install -r requirements.txt
If I want to install code from a release version in Github in Docker, how can I do that taking up the least space possible in the image? Currently, I've done something like:
RUN wget https://github.com/some/repo/archive/v1.5.1.tar.gz¬
RUN tar -xvzf v1.5.1.tar.gz¬
WORKDIR /unzipped-1.5.1/¬
RUN make; make install
Issue here is the final image will have the downloaded tar, the unzipped version, and everything that gets created during make. I don't need the vast majority of this. How do I install my library in my image without keeping all of this extra data?
This is the textbook definition of the problem that the docker multi-stage build aims to solve.
The idea is to use a separate build with the dependencies and use that docker image to build the final product.
Note that this is available only in the new versions of Docker (17.05 onwards).
I have a lot of devdepencencies in my npm script. npm install takes a few minutes the first time, that's ok.
But since I'm integrating with TFS build server, it only needs to npm install once. After that, npm install is just wasting time because it takes 2-3 minutes to just determin the packages are already installed. Also, it seems to always reinstall the packages with -g global flag, even when existing.
How can I make it check if packages exist, and if so, skip npm install?
You can use npm-cache as an alternative way if you use on-premise build agents for build.
It is useful for build processes that run [npm|bower|composer|jspm]
install every time as part of their build process. Since dependencies
don't change often, this often means slower build times. npm-cache
helps alleviate this problem by caching previously installed
dependencies on the build machine. npm-cache can be a drop-in
replacement for any build script that runs [npm|bower|composer|jspm]
install.
How it Works
When you run npm-cache install [npm|bower|jspm|composer], it first
looks for package.json, bower.json, or composer.json in the current
working directory depending on which dependency manager is requested.
It then calculates the MD5 hash of the configuration file and looks
for a filed named .tar.gz in the cache directory ($HOME/.package_cache
by default). If the file does not exist, npm-cache uses the system's
installed dependency manager to install the dependencies. Once the
dependencies are installed, npm-cache tars the newly downloaded
dependencies and stores them in the cache directory. The next time
npm-cache runs and sees the same config file, it will find the tarball
in the cache directory and untar the dependencies in the current
working directory.
And you can also try with npm-install-missing.
However, if you are using VSTS Hosted Build Agent, then you cannot do this since every time you queue a build with Hosted Build Agent, a clean build agent is assigned for the build. That means there is no dependency package installed on the agent. You need to perform a complete npm install.