I want to build a multi container docker app with docker compose. My project structure looks like this:
docker-compose.yml
...
webapp/
...
Dockerfile
api/
...
Dockerfile
Currently, I am just trying to build and run the webapp via docker compose up with the correct build context. When building the webapp container directly via docker build, everything runs smoothly.
However, with my current specifications in the docker-compose.yml the line COPY . /webapp/ in webapp/Dockerfile (see below) copies the whole parent project to the container, i.e. the directory which contains the docker-compose.yml, and not just the webapp/ sub directory.
For some reason the line COPY requirements.txt /webapp/ works as expected.
What is the correct way of specifying the build context in docker compose? Why is the . in the Dockerfile interpretet as relative to the docker-compose.yml, while the requirements.txt is relative to the Dockerfile as expected? What am I missing?
Here are the contents of the docker-compose.yml:
version: "3.8"
services:
frontend:
container_name: "pc-frontend"
volumes:
- .:/webapp
env_file:
- ./webapp/.env
build:
context: ./webapp
ports:
- 5000:5000
and webapp/Dockerfile:
FROM python:3.9-slim
# set environment variables
ENV PYTHONWRITEBYTECODE 1
ENV PYTHONBUFFERED 1
# set working directory
WORKDIR /webapp
# copy dependencies
COPY requirements.txt /webapp/
# install dependencies
RUN pip install -r requirements.txt
# copy project
COPY . /webapp/ # does not work as intended
# add entrypoint to app
# ENTRYPOINT ["start-gunicorn.sh"]
CMD [ "ls", "-la" ] # for debugging
# expose port
EXPOSE 5000
The COPY directive is (probably) working the way you expect. But, you have volumes: that are overwriting the image content with something else. Delete the volumes: block.
The image build sequence is working exactly the way you expect. build: { context: ./webapp } uses the webapp subdirectory as the build context and sends it to the Docker daemon. When the Dockerfile for example COPY requirements.txt . it comes out of this directory. If you, for example, docker-compose run frontend pip freeze, you should see the installed Python packages.
After the image is built, Compose starts a container, and at that point volumes: take effect. When you say volumes: ['.:/webapp'], here the . before the colon refers to the directory containing the docker-compose.yml file (and not the webapp subdirectory), and then it hides everything in the /webapp directory in the container. So you're replacing the image's /webapp (which had been built from the webapp subdirectory) with the current directory on the host (one directory higher).
You should usually be able to successfully combine an ordinary host-based development environment and a Docker deployment setup. Use a non-Docker Python virtual environment to build the application and run its unit tests, then use docker-compose up --build to run integration tests and the complete application. With a setup like this, you don't need to deal with the inconveniences of the Python runtime being "somewhere else" as you're developing, and you can safely remove the volumes: block.
I'm trying to run docker but it can't find modules
I have already have docker image after build step and here's some information.
docker run --rm -v $(pwd):/code -v /code/node_modules -w /code $dockerBuilderImage npm run dist-admin-web
package.json has a script of dist-admin-web with rm -rf bin %% tsc
My docker file looks like
FROM node:12.18.2
COPY package.json ./
COPY package-lock.json ./
RUN npm install
(... some global installations)
As I said, when I do commands docker run and it doesn't work! FYI, I have docker-compose for local development and it works with that image.
My docker compose is following... (I've deleted unneccessary information like env..)
webpack_dev_server:
build:
context: ./
dockerfile: Dockerfile
image: nodewebpack_ts:12.18.2-ts3.5.2
ports:
- "3000:3000"
volumes:
- ./:/ROOT/
- /ROOT/node_modules
working_dir: /ROOT
As I know, I have to add node_modules at volumes because of this. That's why docker compose works
The above code works just fine, the issue is that you forgot to set WORKDIR in your Dockerfile.
WORKDIR /code
which means that you are copying those package.json files into the root directory and node_modules will also be installed there (once the npm install is processed). Then you change workdir when you run the container and since you are using volumes, you will see some strange behavior (things are not exactly where they should be).
Also, while the trick with the additional volume works (node_modules inside of container are not obscured by the mount) there are drawbacks to this approach.
You are creating new unnamed volume each time you run the container. If you don't take care of the old volumes, soon your system will be filled with a lot of dangling volumes.
You are preventing node_modules syncing which is not exactly convenient during development. If you try to install some additional packages once the container is running, you will need to stop the container, build a new image and run it again because it is using the old node_modules which are created during build time.
I guess that this is a matter of taste but I prefer to sync local node_modules with the container via bind mount to avoid the above mentioned problems.
I'm going crazy here.
I've been working on a Dockerfile and docker-compose.yml file for my project. I recently updated my project's dependencies. When I build the project outside of a container using composer install, it builds with the correct dependencies. However, when I build the project inside a docker container, it downloads and installs the latest dependencies, but then somehow runs the application using obsolete dependencies!
First of all, this is what my Dockerfile looks like:
FROM composer
# Set the working directory within the docker container
WORKDIR /app
# Copy in the app, then install dependencies.
COPY . /app
RUN composer install
I have excluded the composer.lock file and the vendor directory in my .dockerignore:
vendor
composer.lock
Here's my docker-compose.yml:
version: "3"
services:
app:
build: .
volumes:
- app:/app
webserver:
image: richarvey/nginx-php-fpm
volumes:
- app:/var/www/html
volumes:
app:
Note that the build process occurs within the app volume. I don't think this should be part of the problem, as I run docker system prune each time, to purge all existing volumes.
This is what I do to run the container. While troubleshooting, I have been running these commands to eliminate any cached files before starting the container:
$ docker system prune
$ docker-compose build --no-cache
$ docker-compose up --force-recreate
As I watch the dependencies install and download, I can see that it is downloading and installing the right versions! So it must have the correct composer.json file at some point in the process.
Yet somehow, once the build is complete and the application starts, I get the same old warnings about obsolete dependencies, and sure enough, and the composer.json inside the container is obsolete!
So my questions are:
How TF is the composer.json file in the container obsolete?
WHERE is it getting the obsolete file from, since it no longer exists in any image or cache??
How TF is it managing to install the latest dependencies with this obsolete composer.json file, but then not using them, and in fact reverting the composer.json file and the dependencies??
I think the problem is, that you copy your local files into the app-container and run composer install on this copy. Since this will not affect your host system, your webserver, which will actually serve your project will still use the outdated local version, instead of the copy from your other image.
You could try using multi-stage builds or something like this:
COPY FROM app:latest /app /var/www/html
This will copy the artifact from your "build-container", i.e. your project with the installed dependency in app, into the actual container that is running the code, i.e. webserver. Unfortunately, I don't think this will work (well) with your setup, where you mount the volume into that location.
Well, I finally fixed this issue, although parts of my original problem still confuse me.
Here's what I learned:
The docker-compose up process goes in this order:
If an image already exists, use it, even if the Dockerfile (or files used by it) has changed. (This can be avoided with docker-compose up --build).
If there is no existing image, build the image from the Dockerfile.
Mount the volumes specified in the docker-compose file.
A huge part of my problem was that I thought that the volumes were mounted before the build process, and that my application would be installed into this volume as a result of these commands:
COPY . /app
RUN composer install
However, these files were later overwritten when the volume was mounted at the same location within the container (/app).
Now, since I was not mounting a host directory, just an ephemeral, named volume, the /app directory should have been empty. I still don't understand why it wasn't, considering I was clearing my existing Docker volumes with docker system prune before each build. Whatever.
In the end, I used #dbrumann's solution. This was simpler, did not require the use of any Docker volumes, and avoids having a live composer container after the build process has completed (this would be bad for production). My Dockerfile now looks like this:
Dockerfile:
# Install dependencies using the composer image
FROM composer AS composer
# Set the working directory within the docker container
WORKDIR /app
# Copy in the app, then install dependencies.
COPY . .
RUN composer install
# Start the nginx server
FROM richarvey/nginx-php-fpm
# Copy over files from the composer image, which is then discarded automatically
WORKDIR /var/www/html
COPY --from=composer /app .
And the new docker-compose.yml:
version: "3.7"
services:
webserver:
build: .
tty: true
ports:
- "80:80"
- "443:443"
I have the problem with installing node_modules inside the Docker container and synchronize them with the host. My Docker's version is 18.03.1-ce, build 9ee9f40 and Docker Compose's version is 1.21.2, build a133471.
My docker-compose.yml looks like:
# Frontend Container.
frontend:
build: ./app/frontend
volumes:
- ./app/frontend:/usr/src/app
- frontend-node-modules:/usr/src/app/node_modules
ports:
- 3000:3000
environment:
NODE_ENV: ${ENV}
command: npm start
# Define all the external volumes.
volumes:
frontend-node-modules: ~
My Dockerfile:
# Set the base image.
FROM node:10
# Create and define the working directory.
RUN mkdir /usr/src/app
WORKDIR /usr/src/app
# Install the application's dependencies.
COPY package.json ./
COPY package-lock.json ./
RUN npm install
The trick with the external volume is described in a lot of blog posts and Stack Overflow answers. For example, this one.
The application works great. The source code is synchronized. The hot reloading works great too.
The only problem that I have is that node_modules folder is empty on the host. Is it possible to synchronize the node_modules folder that is inside Docker container with the host?
I've already read these answers:
docker-compose volume on node_modules but is empty
Accessing node_modules after npm install inside Docker
Unfortunately, they didn't help me a lot. I don't like the first one, because I don't want to run npm install on my host because of the possible cross-platform issues (e.g. the host is Windows or Mac and the Docker container is Debian 8 or Ubuntu 16.04). The second one is not good for me too, because I'd like to run npm install in my Dockerfile instead of running it after the Docker container is started.
Also, I've found this blog post. The author tries to solve the same problem I am faced with. The problem is that node_modules won't be synchronized because we're just copying them from the Docker container to the host.
I'd like my node_modules inside the Docker container to be synchronized with the host. Please, take into account that I want:
to install node_modules automatically instead of manually
to install node_modules inside the Docker container instead of the host
to have node_modules synchronized with the host (if I install some new package inside the Docker container, it should be synchronized with the host automatically without any manual actions)
I need to have node_modules on the host, because:
possibility to read the source code when I need
the IDE needs node_modules to be installed locally so that it could have access to the devDependencies such as eslint or prettier. I don't want to install these devDependencies globally.
At first, I would like to thank David Maze and trust512 for posting their answers. Unfortunately, they didn't help me to solve my problem.
I would like to post my answer to this question.
My docker-compose.yml:
---
# Define Docker Compose version.
version: "3"
# Define all the containers.
services:
# Frontend Container.
frontend:
build: ./app/frontend
volumes:
- ./app/frontend:/usr/src/app
ports:
- 3000:3000
environment:
NODE_ENV: development
command: /usr/src/app/entrypoint.sh
My Dockerfile:
# Set the base image.
FROM node:10
# Create and define the node_modules's cache directory.
RUN mkdir /usr/src/cache
WORKDIR /usr/src/cache
# Install the application's dependencies into the node_modules's cache directory.
COPY package.json ./
COPY package-lock.json ./
RUN npm install
# Create and define the application's working directory.
RUN mkdir /usr/src/app
WORKDIR /usr/src/app
And last but not least entrypoint.sh:
#!/bin/bash
cp -r /usr/src/cache/node_modules/. /usr/src/app/node_modules/
exec npm start
The trickiest part here is to install the node_modules into the node_module's cache directory (/usr/src/cache) which is defined in our Dockerfile. After that, entrypoint.sh will move the node_modules from the cache directory (/usr/src/cache) to our application directory (/usr/src/app). Thanks to this the entire node_modules directory will appear on our host machine.
Looking at my question above I wanted:
to install node_modules automatically instead of manually
to install node_modules inside the Docker container instead of the host
to have node_modules synchronized with the host (if I install some new package inside the Docker container, it should be
synchronized with the host automatically without any manual actions
The first thing is done: node_modules are installed automatically. The second thing is done too: node_modules are installed inside the Docker container (so, there will be no cross-platform issues). And the third thing is done too: node_modules that were installed inside the Docker container will be visible on our host machine and they will be synchronized! If we install some new package inside the Docker container, it will be synchronized with our host machine at once.
The important thing to note: truly speaking, the new package installed inside the Docker container, will appear in /usr/src/app/node_modules. As this directory is synchronized with our host machine, this new package will appear on our host machine's node_modules directory too. But the /usr/src/cache/node_modules will have the old build at this point (without this new package). Anyway, it is not a problem for us. During next docker-compose up --build (--build is required) the Docker will re-install the node_modules (because package.json was changed) and the entrypoint.sh file will move them to our /usr/src/app/node_modules.
You should take into account one more important thing. If you git pull the code from the remote repository or git checkout your-teammate-branch when Docker is running, there may be some new packages added to the package.json file. In this case, you should stop the Docker with CTRL + C and up it again with docker-compose up --build (--build is required). If your containers are running as a daemon, you should just execute docker-compose stop to stop the containers and up it again with docker-compose up --build (--build is required).
If you have any questions, please let me know in the comments.
Having run into this issue and finding the accepted answer pretty slow to copy all node_modules to the host in every container run, I managed to solve it by installing the dependencies in the container, mirror the host volume, and skip installing again if a node_modules folder is present:
Dockerfile:
FROM node:12-alpine
WORKDIR /usr/src/app
CMD [ -d "node_modules" ] && npm run start || npm ci && npm run start
docker-compose.yml:
version: '3.8'
services:
service-1:
build: ./
volumes:
- ./:/usr/src/app
When you need to reinstall the dependencies just delete node_modules.
A Simple, Complete Solution
You can install node_modules in the container using the external named volume trick and synchronize it with the host by configuring the volume's storage location to point to your host's node_modules directory. This can be done with a named volume using the local driver and a bind mount, as seen in the example below.
The volume's data is stored on your host anyway, in something like /var/lib/docker/volumes/, so we're just storing it inside your project instead.
To do this in Docker Compose, just add your node_modules volume to your front-end service, and then configure the volume in the named volumes section, where "device" is the relative path (from the location of docker-compose.yml) to your local (host) node_modules directory.
docker-compose.yml
version: '3.9'
services:
ui:
# Your service options...
volumes:
- node_modules:/path/to/node_modules
volumes:
node_modules:
driver: local
driver_opts:
type: none
o: bind
device: ./local/path/to/node_modules
The key with this solution is to never make changes directly in your host node_modules, but always install, update, or remove Node packages in the container.
Version Control Tip:
When your Node package.json/package-lock.json files change, either when pulling, or switching branches, in addition to rebuilding the Image, you have to remove the Volume, and delete its contents:
docker volume rm example_node_modules
rm -rf local/path/to/node_modules
mkdir local/path/to/node_modules
Documentation:
https://docs.docker.com/storage/volumes/
https://docs.docker.com/storage/bind-mounts/
https://docs.docker.com/compose/compose-file/compose-file-v3/#driver_opts
There's three things going on here:
When you run docker build or docker-compose build, your Dockerfile builds a new image containing a /usr/src/app/node_modules directory and a Node installation, but nothing else. In particular, your application isn't in the built image.
When you docker-compose up, the volumes: ['./app/frontend:/usr/src/app'] directive hides whatever was in /usr/src/app and mounts host system content on top of it.
Then the volumes: ['frontend-node-modules:/usr/src/app/node_modules'] directive mounts the named volume on top of the node_modules tree, hiding the corresponding host system directory.
If you were to launch another container and attach the named volume to it, I expect you'd see the node_modules tree there. For what you're describing you just don't want the named volume: delete the second line from the volumes: block and the volumes: section at the end of the docker-compose.yml file.
No one has mentioned solution with actually using docker's entrypoint feature.
Here is my working solution:
Dockerfile (multistage build, so it is both production and local dev ready):
FROM node:10.15.3 as production
WORKDIR /app
COPY package*.json ./
RUN npm install && npm install --only=dev
COPY . .
RUN npm run build
EXPOSE 3000
CMD ["npm", "start"]
FROM production as dev
COPY docker/dev-entrypoint.sh /usr/local/bin/
ENTRYPOINT ["dev-entrypoint.sh"]
CMD ["npm", "run", "watch"]
docker/dev-entrypoint.sh:
#!/bin/sh
set -e
npm install && npm install --only=dev ## Note this line, rest is copy+paste from original entrypoint
if [ "${1#-}" != "${1}" ] || [ -z "$(command -v "${1}")" ]; then
set -- node "$#"
fi
exec "$#"
docker-compose.yml:
version: "3.7"
services:
web:
build:
target: dev
context: .
volumes:
- .:/app:delegated
ports:
- "3000:3000"
restart: always
environment:
NODE_ENV: dev
With this approach you achieve all 3 points you required and imho it is much cleaner way - not need to move files around.
Binding your host node_modules folder with your container node_modules is not a good practice as you mention. I have seen the solution of creating an internal volume for this folder quite often. Not doing so will cause problems during the building stage.
I ran into this problem when I was trying to build a docker development environment for an angular app, that shows tslib errors when I was editing the files within my host folder cause my host's node_modules folder was empty (as expected).
The cheap solution that helps me, in this case, was to use the Visual Studio Code Extension called "Remote-Containers".
This extension will allow you to attach your Visual Studio Code to your container and edit transparently your files within your container folders. To do so, it will install an internal vscode server within your development container. For more information check this link.
Ensure, however, that your volumes are still created in your docker-compose.yml file.
I hope it helps :D!
I wouldn't suggest overlapping volumes, although I haven't seen any official docs ban it, I've had some issues with it in the past. How I do it is:
Get rid of the external volume as you are not planning on actually using it how it's meant to be used - respawning the container with its data created specifically in the container after stopping+removing it.
The above might be achieved by shortening your compose file a bit:
frontend:
build: ./app/frontend
volumes:
- ./app/frontend:/usr/src/app
ports:
- 3000:3000
environment:
NODE_ENV: ${ENV}
command: npm start
Avoid overlapping volume data with Dockerfile instructions when not necessary.
That means you might need two Dockerfiles - one for local development and one for deploying a fat image with all the application dist files layered inside.
That said, consider a development Dockerfile:
FROM node:10
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
RUN npm install
The above makes the application create a full node_modules installation and map it to your host location, while the docker-compose specified command would start your application off.
I'm not sure to understand why you want your source code to live inside the container and host and bind mount each others during development. Usually, you want your source code to live inside the container for deployments, not development since the code is available on your host and bind mounted.
Your docker-compose.yml
frontend:
volumes:
- ./app/frontend:/usr/src/app
Your Dockerfile
FROM node:10
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
Of course you must run npm install first time and everytime package.json changes, but you run it inside the container so there is no cross-platform issue: docker-compose exec frontend npm install
Finally start your server docker-compose exec frontend npm start
And then later, usually in a CI pipeline targetting a deployment, you build your final image with the whole source code copied and node_modules reinstalled, but of course at this point you don't need anymore the bind mount and "synchronization", so your setup could look like :
docker-compose.yml
frontend:
build:
context: ./app/frontend
target: dev
volumes:
- ./app/frontend:/usr/src/app
Dockerfile
FROM node:10 as dev
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
FROM dev as build
COPY package.json package-lock.json ./
RUN npm install
COPY . ./
CMD ["npm", "start"]
And you target the build stage of your Dockerfile later, either manually or during a pipeline, to build your deployment-ready image.
I know it's not the exact answer to your questions since you have to run npm install and nothing lives inside the container during development, but it solves your node_modules issue, and I feel like your questions are mixing development and deployment considerations, so maybe you thought about this problem in the wrong way.
The best for development
docker-compose.yml
...
frontend:
build: ./app/frontend
ports:
- 3000:3000
volumes:
- ./app/frontend:/usr/src/app
...
./app/frontend/Dockerfile
FROM node:lts
WORKDIR /usr/src/app
RUN npm install -g react-scripts
RUN chown -Rh node:node /usr/src/app
USER node
EXPOSE 3000
CMD [ "sh", "-c", "npm install && npm run start" ]
#FOR PROD
# CMD [ "sh", "-c", "npm install && npm run build" ]
The user node will help you with the rights of host<->guest
The folder node_modules will be accessible from the host and synchronize host<->guest
Thanks Vladyslav Turak for answer with entrypoint.sh where we copy node_modules from container to host.
I implemented the similar thing but I run into the issue with husky, #commitlint, tslint npm packages.
I can't push anything into repository.
Reason: I copied node_modules from Linux to Windows. In my case <5% of files are different (.bin and most of package.json) and 95% are the same. example: image with diff
So I returned to solution with npm install of node_modules for Windows first (for IDE and debugging). And Docker image will contain Linux version of node_modules.
I know that this was resolved, but what about:
Dockerfile:
FROM node
# Create app directory
WORKDIR /usr/src/app
# Your other staffs
EXPOSE 3000
docker-composer.yml:
version: '3.2'
services:
api:
build: ./path/to/folder/with/a/dockerfile
volumes:
- "./volumes/app:/usr/src/app"
command: "npm start"
volumes/app/package.json
{
... ,
"scripts": {
"start": "npm install && node server.js"
},
"dependencies": {
....
}
}
After run, node_modules will be present in your volumes, but its contents are generated within the container so no cross platform problems.
My workaround is to install dependencies when the container is starting instead of during build-time.
Dockerfile:
# We're using a multi-stage build so that we can install dependencies during build-time only for production.
# dev-stage
FROM node:14-alpine AS dev-stage
WORKDIR /usr/src/app
COPY package.json ./
COPY . .
# `yarn install` will run every time we start the container. We're using yarn because it's much faster than npm when there's nothing new to install
CMD ["sh", "-c", "yarn install && yarn run start"]
# production-stage
FROM node:14-alpine AS production-stage
WORKDIR /usr/src/app
COPY package.json ./
RUN yarn install
COPY . .
.dockerignore
Add node_modules to .dockerignore to prevent it from being copied when the Dockerfile runs COPY . .. We use volumes to bring in node_modules.
**/node_modules
docker-compose.yml
node_app:
container_name: node_app
build:
context: ./node_app
target: dev-stage # `production-stage` for production
volumes:
# For development:
# If node_modules already exists on the host, they will be copied
# into the container here. Since `yarn install` runs after the
# container starts, this volume won't override the node_modules.
- ./node_app:/usr/src/app
# For production:
#
- ./node_app:/usr/src/app
- /usr/src/app/node_modules
You could also use dockerized npm install. This is the same as npm install but it runs on a docker container.
https://github.com/datastack-net/dockerized
The node_modules will be written to the host. It should work out of the box and you can specify which npm version to use. If needed, the container can be extended or customized.
Be aware that some npm packages may require compilation, and the generated binaries may not be compatible with your host machine. If you just need the source code or dist files, this is not an issue.
Disclaimer: I'm the author of Dockerized.
I am working on creating a docker container for a node.js microservice and am running into an issue with a local dependency from another folder.
I added the dependency to the node_modules folder using:
npm install -S ../dependency1(module name).
This also added an entry in the package.json as follows:
"dependency1": "file:../dependency1".
When I run the docker-compose up -d command, I receive an error indicating the folowing:
npm ERR! Could not install from "../dependency1" as it does not contain a package.json file.
Dockerfile:
FROM node:latest
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY . /usr/src/app
RUN npm install
CMD [ "npm", "start" ]
EXPOSE 3000
docker-compose.yml:
customer:
container_name: "app_customer"
build:
context: .
dockerfile: Dockerfile
volumes:
- .:/usr/src/app/
- /usr/src/app/node_modules
ports:
- "3000:3000"
depends_on:
- mongo
- rabbitmq
I found articles outlining an issue with symlinks in a node_modules folder and docker and a few outlining this issue but none seem to provide a solution to this problem. I am looking for a solution to this problem or a really good workaround.
A Docker build can't reference files outside of the build context, which is the . defined in the docker-compose.yml file.
docker build creates a tar bundle of all the files in a build context and sends that to the Docker daemon for the build. Anything outside the context directory doesn't exist to the build.
You could move your build context with context: ../ to the parent directory and shuffle all the paths you reference in the Dockerfile to match. Just be careful not to make the build context to large as it can slow down the build process.
The other option is to publish the private npm modules to a scope, possible on a private npm registry that you and the build server have access to and install the dependencies normally.