I am using npm to manage dependencies for my Typescript project. This project references a handful of other, local npm modules (e.g. using npm install ../path_to_package).
All works fine locally but I would like to build a docker image for my main project. This involves copying each local package to the docker build context and building them all. Currently, I copy each required package:
COPY packages/package_1/ ./packages/package_1/
COPY packages/package_2/ ./packages/package_2/
COPY packages/package_3/ ./packages/package_3/
COPY main_project/ ./main_project/
and then build them in the order that they are needed (say packge_1 depends on package_2, package_2 must be built, then package_1):
RUN cd packages/package_2 && npm install && npm run build && cd ../ \
&& cd packages/package_1 && npm install && npm run build && cd ../ \
&& cd packages/package_3 && npm install && npm run build && cd ../ \
&& cd main_project && npm install && npm run build
This is obviously very inconvenient to have to manually check the dependency tree of main_project to then build the packages in order, especially when I create and install new local packages. Is there a way to, based on main_project's package.json for example (or package-lock.json) automatically copy and build the dependencies in order, then build the main project?
Related
I'm trying to install a specific version of npm package within the node:alpine image (latest tag), so this is what I specified in my Dockerfile:
Edit (this is the full Dockerfile):
FROM node:alpine as build-stage
ARG SONAR_VERSION=4.6.0.2311
RUN apk add --no-cache zip unzip
RUN wget -c https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-$SONAR_VERSION-linux.zip
RUN unzip -q sonar-scanner-cli-$SONAR_VERSION-linux.zip
RUN rm sonar-scanner-cli-$SONAR_VERSION-linux.zip
# Adding the $install_directory/bin directory to path
RUN cd sonar-scanner-$SONAR_VERSION-linux
RUN echo $PWD
RUN export PATH="$PATH:/$PWD/bin"
RUN npm i -g eslint#6.7.2
RUN npm i -g eslint-plugin-vue --save-dev
However when I run & execute the built image, the installed version of eslint is the latest instead of what I'd specified in the Dockerfile.
Result from npm list -g:
/ # npm list -g
/usr/local/lib
+-- eslint-plugin-vue#7.9.0
+-- eslint#7.25.0
`-- npm#7.11.2
If I run npm i -g eslint#6.7.2 within the container itself, the version from npm list -g will be exactly the one I specified.
What is the reason of this scenario?
I know that every line of RUN ... will add a layer to the docker image and that it is recommended to make RUN commands connected with &&. But my question is:
Is better this:
RUN apk update && apk upgrade \
&& apk add openssh \
&& apk add --update nodejs nodejs-npm \
&& npm install -g #angular/cli \
&& apk add openjdk11 \
&& apk add maven \
&& apk add git
Or this:
RUN apk update && apk upgrade
RUN apk add openssh
RUN apk add --update nodejs nodejs-npm
RUN npm install -g #angular/cli
RUN apk add openjdk11
RUN apk add maven
RUN apk add git
The first one creates just one layer but when a version of anything changes the image would have to start from the beginning, not from cash. The second approach will create more layers but when just the version of git changes only the git layer needs to be build again and all previous layers can be used from cash.
I'd recommend:
Install all the OS packages in a single apk invocation: there is some overhead in starting the package manager (more noticeable with dpkg/apt) and it is faster if you start it once and install several packages
If you need to run an update command, always run it in the same RUN command as your other package-manager steps. This avoids some trouble with Docker layer caching (again, very noticeable with apt) where docker build doesn't re-run update, but then it does try to run a changed install step; when it tries to install a package using yesterday's package index, the upload of that package that happened today deleted yesterday's file and the download will fail.
Don't npm install single packages. That means your package.json file is incomplete. Add it there.
I've seen recommendations both ways as to whether or not to run a full upgrade. Keeping up-to-date on security fixes is important; the underlying base images on Docker Hub also update pretty regularly. So if your image is FROM alpine:latest, doing a docker build --pull will get you much of the effect of an explicit apk upgrade.
Stylistically, if I need any substantial number of packages, I find the list a little more maintainable if I sort it alphabetically and put one package on a line, but this is purely personal preference.
Putting this all together would transform your example into:
RUN apk update \
&& apk upgrade \
&& apk add \
git \
maven \
nodejs \
nodejs-npm \
openjdk11 \
openssh
COPY package.json package-lock.json . # includes #angular/cli
RUN npm ci
Don't be afraid to use multiple containers, if that makes sense. (What's your application that uses both Java and Node together; can it be split into two single-language parts?) Don't install unnecessary developer-oriented tools in your image. (Does your application invoke git while it's running; do you install a dependency directly from GitHub; or can you remove git?) Don't try to run an ssh daemon in your container. (It breaks the "one process per container" rule which instantly makes things harder to manage; 90% of the SO examples have a hard-coded user password plus sudo rights, which is not really a security best practice; managing the credentials is essentially impossible.)
Both approaches are on the extreem, you need to try to minimize the layers for "reusability" at the same time to optimize for lower number of layers.
Based on your example, the build can be organized as follows:
RUN apk update && apk upgrade \
&& apk add openssh \
&& apk add --update nodejs nodejs-npm \
&& apk add openjdk11 \
&& apk add maven \
&& apk add git
RUN npm install -g #angular/cli \
Now I have only 2 layers, first one is bringing the OS packages and the second one dealing with the node.js packages. Now this can better be reused in other builds.
Once you have done this modification, you can move to multistage build where you will be able to better control and reuse the intermediate containers like in this example
I am running my monolith application in a docker container and k8s on GKE.
The application contains python & node dependencies also webpack for front end bundle.
We have implemented CI/CD which is taking around 5-6 min to build & deploy new version to k8s cluster.
Main goal is to reduce the build time as much possible. Written Dockerfile is multi stage.
Webpack is taking more time to generate the bundle.To buid docker image i am using already high config worker.
To reduce time i tried using the Kaniko builder.
Issue :
As docker cache layers for python code it's working perfectly. But when there is any changes in JS or CSS file we have to generate bundle.
When there is any changes in JS & CSS file instead if generate new bundle its use caching layer.
Is there any way to separate out build new bundle or use cache by passing some value to docker file.
Here is my docker file :
FROM python:3.5 AS python-build
WORKDIR /app
COPY requirements.txt ./
RUN pip install -r requirements.txt &&\
pip3 install Flask-JWT-Extended==3.20.0
ADD . /app
FROM node:10-alpine AS node-build
WORKDIR /app
COPY --from=python-build ./app/app/static/package.json app/static/
COPY --from=python-build ./app ./
WORKDIR /app/app/static
RUN npm cache verify && npm install && npm install -g --unsafe-perm node-sass && npm run sass && npm run build
FROM python:3.5-slim
COPY --from=python-build /root/.cache /root/.cache
WORKDIR /app
COPY --from=node-build ./app ./
RUN apt-get update -yq \
&& apt-get install curl -yq \
&& pip install -r requirements.txt
EXPOSE 9595
CMD python3 run.py
I would suggest to create separate build pipelines for your docker images, where you know that the requirements for npm and pip aren't so frequent.
This will incredibly improve the speed, reducing the time of access to npm and pip registries.
Use a private docker registry (the official one or something like VMWare harbor or SonaType Nexus OSS).
You store those build images on your registry and use them whenever something on the project changes.
Something like this:
First Docker Builder // python-builder:YOUR_TAG [gitrev, date, etc.)
docker build --no-cache -t python-builder:YOUR_TAG -f Dockerfile.python.build .
FROM python:3.5
WORKDIR /app
COPY requirements.txt ./
RUN pip install -r requirements.txt &&\
pip3 install Flask-JWT-Extended==3.20.0
Second Docker Builder // js-builder:YOUR_TAG [gitrev, date, etc.)
docker build --no-cache -t js-builder:YOUR_TAG -f Dockerfile.js.build .
FROM node:10-alpine
WORKDIR /app
COPY app/static/package.json /app/app/static
WORKDIR /app/app/static
RUN npm cache verify && npm install && npm install -g --unsafe-perm node-sass
Your Application Multi-stage build:
docker build --no-cache -t app_delivery:YOUR_TAG -f Dockerfile.app .
FROM python-builder:YOUR_TAG as python-build
# Nothing, already "stoned" in another build process
FROM js-builder:YOUR_TAG AS node-build
ADD ##### YOUR JS/CSS files only here, required from npm! ###
RUN npm run sass && npm run build
FROM python:3.5-slim
COPY . /app # your original clean app
COPY --from=python-build #### only the files installed with the pip command
WORKDIR /app
COPY --from=node-build ##### Only the generated files from npm here! ###
RUN apt-get update -yq \
&& apt-get install curl -yq \
&& pip install -r requirements.txt
EXPOSE 9595
CMD python3 run.py
A question is: why do you install curl and execute again the pip install -r requirements.txt command in the final docker image?
Triggering every time an apt-get update and install without cleaning the apt cache /var/cache/apt folder produces a bigger image.
As suggestion, use the docker build command with the option --no-cache to avoid caching result:
docker build --no-cache -t your_image:your_tag -f your_dockerfile .
Remarks:
You'll have 3 separate Dockerfiles, as I listed above.
Build the Docker images 1 and 2 only if you change your python-pip and node-npm requirements, otherwise keep them fixed for your project.
If any dependency requirement changes, then update the docker image involved and then the multistage one to point to the latest built image.
You should always build only the source code of your project (CSS, JS, python). In this way, you have also guaranteed reproducible builds.
To optimize your environment and copy files across the multi-stage builders, try to use virtualenv for python build.
I have a cli tool that I'm creating an E2E test suite using docker. The basic idea is in the docker container I'll build the local code and then run the tool over several public repositories. The main goal of this is to make it easier to see any missing functionality that my CLI tool may need to add.
I did manage to get it working but it's a bit of a pain to manage due to all the chaining and folder management.
## Build a node application
from node:8.11.1
WORKDIR /app
## Copy all the files
COPY . ./sortier
## Build run and test
RUN cd ./sortier \
&& npm install --unsafe-perm \
&& npm run test \
&& cd .. \
## Run react-redux-typescript-guide/playground test
&& pwd \
&& git clone https://github.com/piotrwitek/react-redux-typescript-guide \
&& cd react-redux-typescript-guide/playground \
&& npm install --unsafe-perm \
&& echo "{ isHelpMode: true }" > .sortierrc \
&& cd ../../sortier \
&& npm run start -- "../react-redux-typescript-guide/playground/src/**/*.ts" \
&& npm run start -- "../react-redux-typescript-guide/playground/src/**/*.tsx" \
&& cd ../react-redux-typescript-guide/playground \
&& npm run build \
&& cd ../.. \
## Run prettier test
&& pwd \
&& git clone https://github.com/prettier/prettier \
&& cd prettier \
&& npm install --unsafe-perm \
&& echo "{ isHelpMode: true }" > .sortierrc \
&& cd .. \
&& npm run start -- "prettier/src/**/*.js" \
&& cd prettier \
&& npm run build \
&& npm run test \
&& cd ..
I was trying to figure out how to use WORKDIR instead to change directories which would clean it up a lot but being able to reference work directories from one another didn't seem to work.
Any advice on how I can clean up this dockerfile?
And of course after I post the question I figure out the answer (do'h)
## Build a node application
from node:8.11.1
## Sortier creation, build and test
WORKDIR /sortier
COPY . .
RUN npm install --unsafe-perm
RUN npm run test
## react-redux-typescript-guide/playground
WORKDIR /react-redux-typescript-guide
RUN git clone https://github.com/piotrwitek/react-redux-typescript-guide .
WORKDIR /react-redux-typescript-guide/playground
RUN npm install --unsafe-perm
RUN echo "{ isHelpMode: true }" > .sortierrc
WORKDIR /sortier
RUN npm run start -- "/react-redux-typescript-guide/playground/src/**/*.ts"
RUN npm run start -- "/react-redux-typescript-guide/playground/src/**/*.tsx"
WORKDIR /react-redux-typescript-guide/playground
RUN npm run build
RUN set CI=true&&npm run test
## prettier
WORKDIR /prettier
RUN git clone https://github.com/prettier/prettier
WORKDIR /prettier
RUN npm install --unsafe-perm
RUN echo "{ isHelpMode: true }" > .sortierrc
WORKDIR /sortier
RUN npm run start -- "/prettier/src/**/*.js"
WORKDIR /prettier
RUN npm run build
RUN npm run test
For node applications, what is the better option for building lightweight images?
Single Docker Image. Might require build tools and would expose all build time environment variables to the container.
yarn install # install deps and devDeps
yarn build # build our application
yarn test # perform tests
yarn install --production --ignore-scripts --prefer-offline # Remove devDeps
rm -rf src # Remove source files
yarn start
Multiple Docker Images
In one docker container that has build tools, do the:
yarn install
yarn build
yarn test
Then take the build assets and package.json, and copy them into a new container which would have only runtime environment variables. The result is a much smaller image (perhaps node:alpine) that has only very limited source files.
yarn install --production --ignore-scripts --prefer-offline
yarn start
If you want to build lightweight image for your application, keep in mind the following:
Try to use alpine images, e.g. node:8.12.0-alpine as alpine images are lightest base os image. If you want to install packages, do RUN apk add --no-cache --virtual your_packages... && rm -rf /var/cache/apk/*
Try to reduce number of layers by running multiple commands in same RUN statement, e.g. RUN yarn install && yarn build && yarn test && yarn install --production --ignore-scripts --prefer-offline && rm
-rf src && yarn start
Try to club commands that cancel out each other, e.g. RUN apk update && apk add ... && rm -rf /var/cache/apk/*. Here apk update creates a cache and rm -rf /var/cache/apk/* clears it. No need to run these two commands separately as they are cancelling out each others work hence no point in having two layers which negate each other and inflate the size of the final image.
Note: Having multiple docker files instead of one is not going to reduce the number of layers or shrink the size. It only gives you logical separation of tasks that you want to handle individually.