I am using yarn workspaces and I have this packages in my package.json:
"workspaces": ["packages/*"]
I am trying to create a docker image to deploy and I have the following Dockerfile:
# production dockerfile
FROM node:9.2
# add code
COPY ./packages/website/dist /cutting
WORKDIR /cutting
COPY package.json /cutting/
RUN yarn install --pure-lockfile && yarn cache clean --production
CMD npm run serve
But I get the following error:
error An unexpected error occurred:
"https://registry.yarnpkg.com/#cutting%2futil: Not found"
#cutting/util is the name of one of my workspace packages.
So the problem is that there is no source code in the docker image so it is trying to install it from yarnpkg.
what is the best way to handle workspaces when deploying to a docker image.
This code won't work outside of the docker vm, so it will refuse in the docker, too.
The problem is you have built a code, and copy the bundled code. The yarn workspaces is looking for a package.json that you don't have in the dist folder. The workspaces is just creating a link in a common node_modules folder to the other workspace that you are using. The source code is needed there. (BTW why don't you build code inside the docker vm? That way source code and dist would also be available.)
Here is my dockerfile. I use yarn workspaces and lerna, but without lerna should be similar. You want to build your shared libraries and then test the build works locally by running your code in your dist folder.
###############################################################################
# Step 1 : Builder image
FROM node:11 AS builder
WORKDIR /usr/src/app
ENV NODE_ENV production
RUN npm i -g yarn
RUN npm i -g lerna
COPY ./lerna.json .
COPY ./package* ./
COPY ./yarn* ./
COPY ./.env .
COPY ./packages/shared/ ./packages/shared
COPY ./packages/api/ ./packages/api
# Install dependencies and build whatever you have to build
RUN yarn install --production
RUN lerna bootstrap
RUN cd /usr/src/app/packages/shared && yarn build
RUN cd /usr/src/app/packages/api && yarn build
###############################################################################
# Step 2 : Run image
FROM node:11
LABEL maintainer="Richard T"
LABEL version="1.0"
LABEL description="This is our dist docker image"
RUN npm i -g yarn
RUN npm i -g lerna
ENV NODE_ENV production
ENV NPM_CONFIG_LOGLEVEL error
ARG PORT=3001
ENV PORT $PORT
WORKDIR /usr/src/app
COPY ./package* ./
COPY ./lerna.json ./
COPY ./.env ./
COPY ./yarn* ./
COPY --from=builder /usr/src/app/packages/shared ./packages/shared
COPY ./packages/api/package* ./packages/api/
COPY ./packages/api/.env* ./packages/api/
COPY --from=builder /usr/src/app/packages/api ./packages/api
RUN yarn install
CMD cd ./packages/api && yarn start-production
EXPOSE $PORT
###############################################################################
Related
I'm running
docker build --build-arg npm_token=//NPM TOKEN HERE// -t test .
and the build is failing with an error message of
error Command "dotenv" not found even though the dotenv nom package is included in the package.json and the yarn.lock files. Here is my docker file:
# image has Cypress npm module installed globally in /root/.npm/node_modules
# and Cypress binary cached in /root/.cache/Cypress folder
FROM cypress/included:9.7.0
WORKDIR /usr/src/app
# Set up NPM token to access private GitHub packages
ARG npm_token
ENV NPM_TOKEN=$npm_token
COPY .npmrc ./
RUN npm config set //npm.pkg.github.com/:_authToken $NPM_TOKEN
COPY config cypress .env package.json cypress.json yarn.lock tsconfig.json ./
RUN yarn \
dotenv -- node e2e-tests.js
Does anyone know why this is happening?
you should install dotenv-cli to run .env commands :
yarn add dotenv-cli
I have the following project separated into packages as follows:
libs/one/package.json
libs/one/index.js
libs/two/package.json
libs/two/index.js
package.json
I am looking for a way to copy all libs/*/package.json files in one line. in addition I want to avoid reinstalling all packages again and reading them from the cache.
this is my docker file
FROM node:16.14-alpine as base
RUN npm install -g lerna#4.0.0
WORKDIR /usr/app
FROM base as builder
ARG SERVICE_DIR
ARG TSCONFIG_BUILD_FILE
COPY .npmrc .
COPY lerna.json .
COPY ${TSCONFIG_BUILD_FILE} ./tsconfig.build.json
COPY package*.json ./
# copy package dependencies
COPY libs/service-http-handler/package.json ./libs/service-http-handler/
COPY libs/utils/package.json ./libs/utils/
COPY libs/commons/package.json ./libs/commons/
COPY libs/cache/package.json ./libs/cache/
COPY libs/internal-dto/package.json ./libs/internal-dto/
COPY libs/clients/package.json ./libs/clients/
COPY ${SERVICE_DIR}/package.json ./${SERVICE_DIR}/
# install service and package dependencies
RUN lerna bootstrap -- --production --no-optional --ignore-scripts --include-dependencies --scope ${SERVICE_DIR}
# Copy all libs and build by dependency order
COPY libs/ ./libs
COPY ${SERVICE_DIR}/ ./${SERVICE_DIR}
RUN lerna run build --include-dependencies --scope $(echo ${SERVICE_DIR} | cut -d'/' -f 2)
# copy recursive and follow symbolic link
RUN cp -RL ./node_modules/ /tmp/node_modules/
# Runner
FROM base
ARG SERVICE_DIR
# copy runtime dependencies
COPY --from=builder /tmp/node_modules/ ./node_modules/
# Copy runtime service
COPY --from=builder /usr/app/${SERVICE_DIR}/dist/ ./dist/
COPY ./${SERVICE_DIR}/package.json ./
EXPOSE 5001
#TODO - modify start script in production mode
CMD ["npm", "run", "start"]
any idea how I can do it without the need to run the step below on any changes in libs sources files.
RUN lerna bootstrap -- --production --no-optional --ignore-scripts --include-dependencies --scope ${SERVICE_DIR}
I have a Dockerfile and when I run it locally, everything works fine, however my build through GitHub actions seems to fail, the error I am getting is:
error An unexpected error occurred: "ENOENT: no such file or directory, stat '/home/runner/work/akira/akira/README.md'".
I tried to remove the yarn.lock but without success, a full log of the build that fails can be found here, my Dockerfile is below:
Dockerfile:
FROM node:14.0.0 AS base
WORKDIR /usr/src/app
FROM base as builder
COPY ./lerna.json .
COPY ./package.json .
COPY ./tsconfig.json .
COPY ./yarn.lock .
COPY ./packages/akira/prisma ./packages/akira/prisma
COPY ./packages/akira/src ./packages/akira/src
COPY ./packages/akira/types ./packages/akira/types
COPY ./packages/akira/package*.json ./packages/akira/
COPY ./packages/akira/tsconfig.json ./packages/akira
RUN yarn install --frozen-lockfile
RUN yarn build
FROM builder as migrate
RUN yarn workspace akira prisma migrate up --experimental
FROM base AS app
COPY --from=builder /usr/src/app/yarn.lock .
COPY --from=builder /usr/src/app/packages/akira/dist ./dist
COPY --from=builder /usr/src/app/packages/akira/prisma ./prisma
COPY --from=builder /usr/src/app/packages/akira/package.json .
RUN yarn install --production
USER node
ENV NODE_ENV=production
EXPOSE 4000
CMD ["node", "dist/index.js"]
If you look at your GitHub Actions workflow,
or the log from the failing build that you linked, it seems to be running yarn commands outside of docker.
It looks like yarn is struggling with the README symlink, not sure why, but as it seems you want to build with docker, I would try the following:
Replace this part of the yaml
- name: Use Node.js
uses: actions/setup-node#master
with:
node-version: 14.4.0
- name: Install dependencies
run: yarn --frozen-lockfile
- name: Build packages
run: yarn build
with something like
- name: Build docker image
run: docker build .
Edit:
As pointed out in below comment, the Dockerfile includes a side-effect of deploying database migrations.
If you don't want to run everything from the Dockerfile in the Build pipeline,
you can leverage multi-stage builds and stop at a specific stage.
I.e., move the migrations into its own stage:
FROM node:14.0.0 AS base
WORKDIR /usr/src/app
FROM base as builder
COPY ./lerna.json .
<< lines omitted >>
RUN yarn install --frozen-lockfile
RUN yarn build
FROM builder AS migr
RUN yarn workspace akira prisma migrate up --experimental
FROM base AS app
COPY --from=builder /usr/src/app/yarn.lock .
<< lines omitted >>
Then you can stop after the builder stage with
docker build --target builder .
Edit 2:
Or you could keep the build pipeline and Dockerfile as it is, and instead fix the broken symlink, i.e. revert commit 0c87fa3
I have the following Dockerfile:
FROM node:12.4.0 as builder
COPY ./package.json /src/package.json
ENV PORT 3000
WORKDIR /src
RUN npm install
COPY ./lerna.json /src/lerna.json
COPY ./packages/package-api/package.json /src/packages/package-api/package.json
COPY ./packages/package-config/package.json /src/packages/package-config/package.json
COPY ./packages/package-sitemap/package.json /src/packages/package-sitemap/package.json
COPY ./packages/package-utils/package.json /src/packages/package-utils/package.json
RUN npm run clean
COPY . /src
WORKDIR /src/packages/package-sitemap
RUN npm run build
FROM node:12.4.0
RUN echo "starting 2nd stage"
WORKDIR /src/packages/package-sitemap
COPY --from=builder /src/packages/package-sitemap/build/bundle.js .
EXPOSE ${PORT}
CMD ["node" , "bundle.js"]
running the bundle built from such Dockerfile results in following error: Error: Cannot find module 'debug'.
However if I remove the second stage and build the bundle in one stage there's no error and the bundle runs successfully, that is with the following Dockerfile:
FROM node:12.4.0 as builder
COPY ./package.json /src/package.json
ENV PORT 3000
WORKDIR /src
RUN npm install
COPY ./lerna.json /src/lerna.json
COPY ./packages/package-api/package.json /src/packages/package-api/package.json
COPY ./packages/package-config/package.json /src/packages/package-config/package.json
COPY ./packages/package-sitemap/package.json /src/packages/package-sitemap/package.json
COPY ./packages/package-utils/package.json /src/packages/package-utils/package.json
RUN npm run clean
COPY . /src
WORKDIR /src/packages/leaf-sitemap
RUN npm run build
EXPOSE ${PORT}
CMD ["node" , "build/bundle.js"]
I'm wondering what the difference can be? Is it because in the 2nd Dockerfile node_modules exist as well and webpack somehow sets paths to some modules in the bundle to the node_modules?
I've just started to get deep inside dockerfile syntax.
Here is the one I use currently:
FROM node:12-alpine as install
WORKDIR /Backend-graphql
COPY ./src ./src
COPY ./index.js ./index.js
COPY ./schema.graphql ./schema.graphql
COPY ./package.json ./
COPY ./package-lock.json ./package-lock.json
RUN npm install
FROM node:12-alpine as prismawork
WORKDIR /PrismaWork
COPY --from=install /Backend-graphql .
COPY ./datamodel.prisma ./datamodel.prisma
COPY ./prisma.yml ./prisma.yml
RUN npx prisma deploy
RUN npx prisma generate
FROM node:12-alpine
#curl needed for healthcheck
RUN apk --update --no-cache add curl
WORKDIR /app
COPY --from=prismawork /PrismaWork .
ENTRYPOINT ["npm", "start"]
EXPOSE 4000
From personnals tests and documentations founds online, i've respected the following advice :
Use multi-stage builds
But I notice something, docker do not reuse cache after the first COPY layer different in current and followings build stages. And I think its a problem, because I use an automatic bump version git hook based on commit message semantic versionning syntax who modifies my package.json. So, at each commit docker build re-RUN npm install and subsequents layers.
First of all, have I understood docker cache layering system ?
Secondly, should I use an other file for automatic bumping version and COPY it at the very end of my Dockerfile ?
First of all, have I understood docker cache layering system?
Yes, you should.
If any change happens in any step, like changes in package.json, the docker will rebuild the rest of the steps.
No need to copy from the same image multiple times. We are also doing npm install after performing unrelated steps to catching other steps.
FROM node:12-alpine
#curl needed for healthcheck
RUN apk --update --no-cache add curl
WORKDIR /app
COPY ./src ./src
COPY ./index.js ./index.js
COPY ./schema.graphql ./schema.graphql
COPY ./datamodel.prisma ./datamodel.prisma
COPY ./prisma.yml ./prisma.yml
COPY ./package.json ./
COPY ./package-lock.json ./package-lock.json
RUN npm install
RUN npx prisma deploy
RUN npx prisma generate
ENTRYPOINT ["npm", "start"]
EXPOSE 4000
Multiple staging useful when you need to build between multiple images like this example:
FROM node:12-alpine
RUN npm install -g gzipper
WORKDIR /build
ADD . .
RUN npm install
ARG CONFIGURATION
RUN npm run build:${CONFIGURATION}
RUN gzipper --gzip-level=6 ./dist
FROM nginx:latest
WORKDIR /usr/share/nginx/html
COPY --from=0 /build/dist .
COPY nginx/default.conf /etc/nginx/conf.d/default.conf