I have two local (custom) NPM packages that I've used before. When I install them with npm i parentFolder/package1 parentFolder/package2, they install just fine. However, when I add them to my package.json file, and then copy / npm install in my DockerFile, I get an error saying "Could not install from "x" as it does not contain a package.json file.". I honestly don't know what to make of this, since it does have a package.json file, and it installs just fine otherwise.
Is there a special step that I need to take for custom packages to work in docker, or am I just lost somehow? I've been staring at this error so long that my brain's a mess, so I probably forgot to add some details. If I forgot something you need to know in order to help, please let me know.
Docker-compose file:
volumes:
- ./sprinklers:/app
# - sprinklers_node_modules:/app/node_modules/
- ./sprinklers/node_modules_temp:/app/node_modules/
# - sprinklers_persistent:/app/.node-persist/
- ./sprinklers/.node-persist:/app/.node-persist/
As you can see, I was using named volumes, but tried going to mounted ones to see what happened.
DockerFile:
FROM node:14.16.0-slim
WORKDIR /app
RUN npm install
The COPY statements here were removed, since all files are mounted in the docker-compose file.
Related
I'm using poetry in my project and now working on a feature that will allow to run the app inside of a docker container. Now, my Dockerfile looks like this:
COPY pyproject.toml /
...
RUN poetry install
The last command takes around 4 minutes which is quite a lot so I thought of caching somehow this dependencies. I'm trying to convert my pyproject.toml to requirements.txt so I could feed it to Docker and it would cache it if the file hasn't been changed since the last run.
Now I'm trying:
poetry export -f requirements.txt --output requirements.txt
And it only writes dependencies from [tool.poetry.dependencies] section, but the problem is that I have other sections and would like to see dependencies from those in my requirements.txt file. How should I modify the command above so it would take dependencies from other sections as well.
P.S. Maybe you might know other ways of how to cache poetry install in docker, I'd really appreciate that!
I think you can do 2-step dependencies install with poetry to cache dependensies like in example here - https://pythonspeed.com/articles/poetry-vs-docker-caching/, no need to migrate to requirements.txt. The idea is to copy only toml, install dependencies (this way dependencies will cache and need to update only if toml changes), then copy you source files (which change more often, than toml) and do install again. More detailed explanation in the link above (https://pythonspeed.com/articles/poetry-vs-docker-caching/)
When I was building and trying to run my docker image with a rubyonrails app, I was getting this error:
warning Integrity check: System parameters don't match
error Integrity check failed
error Found 1 errors.
I tried changing my Docker file with
RUN yarn install --check-files
But that didn't do anything.
I then just deleted the yarn.lock file and my container now runs.
I am guessing the issue is that rails was run locally on my laptop, and now it is trying to run the same yarn.lock file on another computer and the integrity check is failing? Is this correct?
What should my dockerfile be doing? Should I exclude the yarn.lock file from getting into my docker container in the first place?
First of all, you will need to remove the node_modules folder and run the yarn install again. In your command line, follow these instructions:
Remove node_modules folder by typing rm -rf node_modules
Run yarn install
Run rails webpacker:install
Restart your command-line editor.
Be careful about the NodeJS version in your machine. It must be the same as the version in which the rails project was initialized. You can use nvm to manage the Node version.
Add "config.webpacker.check_yarn_integrity = false" in "development.rb" will solve the problem
I would suggest not copying yarn.lock in the container. You can add yarn.lock in the .dockerignore which Docker will ignore the yarn.lock while building the image.
I had faced similar issues because, locally I run on macOS and containers are alpine-based, so we end up ignoring the yarn.lock
I see that bundle install and yarn install are usually done in Dockerfile as:
RUN bundle install && yarn install
Which means that if I modify Gemfile or yarn.lock, I need to re-build the image again. I know that there is layer caching and the docker build will not rebuild other layers except bundle install && yarn install layer. But it means I have to do docker-compose up -d --build
But I was wondering if it is ok to put these commands inside an entry script of docker-compose or in command as:
command: bundle install && yarn install && rails s
In this way, I believe, whenever I do docker-compose up -d, bundle install and yarn install will be executed without having to build the image.
Not sure if it has any advantages over conventional bundle install in Dockerfile except not having to append --build in docker-compose up. Correct that if I do this, bundle install and yarn install will get executed even when there are no changes to Gemfile or Yarn files. I guess this is one of the bad sides.
Please correct me if it is not the ideal way to go.
New to docker world.
It wastes several minutes of time and uses up network bandwidth every time you start your application. When you're doing local development, it'd be the equivalent of doing this, every time you run the application:
rm -rf vendor node_modules
bundle install # from scratch
yarn install # from scratch
bundle exec rails s
A core part of Docker is rebuilding images (in the same way that languages like Go, Java, Typescript, etc. have a "build" phase). Trying to avoid image rebuilds isn't usually advisable. With a well-written Dockerfile, and particularly for an interpreted language, running docker build should be fairly efficient.
The one important trick is to separately copy the files that specify dependencies, and the rest of your application. As soon as a Dockerfile COPY instruction encounters a file that's changed it will disable layer caching for the rest of the application. Since dependencies change relatively infrequently, a sequence that first copies the dependency file, then installs the dependencies, then copies the application can jump straight to the last step if the dependency file hasn't changed.
COPY Gemfile Gemfile.lock ./
RUN bundle install
COPY package.json yarn.lock ./
RUN yarn install
COPY . ./
(Make sure to include the Bundler vendor directory and the node_modules directory in a .dockerignore file so the last COPY step doesn't overwrite what previously got installed.)
This question is opinion based. As you already found out yourself, it is a common practice to install dependencies (bundle, yarn, others) during the image build process, and not image run process.
The rationale is that you run more times than you build, and you want the run operation to start quickly.
In the same way that you do apt install... or yum install... in the build stage, you should normally do bundle install in the build stage as well.
That said, if it makes sense to you to bundle install as a part of the entrypoint, that is your choice. I suspect that after you do it, you will see that it is less common for a reason.
Another note about docker layers: If the Gemfile change, not only the layer that refers to it will change, but all subsequent layers as well. For that reason, it is often common to separate the copy of the dependencies manifest (Gemfile.*) from the copying of the app, like this:
# Pre-install gems
COPY Gemfile* ./
RUN gem install bundler && \
bundle install --jobs=3 --retry=3
# Copy the rest of the app
COPY . .
So this way, if your app files change, but not the dependencies, the build will be faster.
I am trying to setup docker-compose architecture for local development and production and I can't figure when in the containers life it's the best time to install library dependencies. In the same time I am not sure if these should be placed in the container or in external volume.
All my code is mounted in external volumes, so that changes are immidiately taken into without rebuilding the containers, but I am not sure about libraries that need to be installed by pip (I am running python backend) and npm/yarn (for webpack front-end).
Placing requirments.txt and package.json into the containers and running pip install and yarn install in the container build process means that I have to rebuild the container any time dependecies change - that is too much overhead.
Putting them in an external volume and running pip install and yarn install as part of the command of each container when it is started seems to solve the issue.
The build process of each container then contains only platform dependencies (eg. installing python, webpack or other platform tools), but libraries are installed after started (with CMD directive).
Is this the correct approach? I have seen lot of examples doing exactly the oposite and running npm install in the build process of the container - but I don't see any advantage for that, am I missing something?
Installing dependecies is usually part of the build process. Mounting code is a good trick when developing in order to get changes directly reflected.
Concerning adding requirements.txt or package.json. Installing dependecies takes time, and for that you need to take advantage of docker layer caching. In particular, you want to avoid cache invalidation.
For pip I suggest the following in development phase: For dependencies that you are unlikely to change, install these in separate RUN instuction. Your Docker file will look something like.
FROM ..
RUN pip install package1 package2 package3 ...
ADD requirements.txt requirements.txt
RUN RUN pip install -r requirements.txt
...
Keep only dependencies that might be changed in requirements.txt. Once you are done developing, add the packages back to the requirements.txt and build using the requirements file.
A similar approach would be adding two requirements files, and at the end combining them.
I haven’t been able to find an answer to this question anywhere, so I’m hoping someone here might have run into a similar issue.
I’m trying to containerize a NodeJS application, but run into this error during RUN npm install:
npm ERR! Could not install from "packages/example-package" as it does
not contain a package.json file
The package in question is listed as a local dependency in my app’s package.json:
"dependencies": {
"#example-name/example-package": "file:../example-package" },
There are no issues running npm install outside of docker with these local file paths.
I’ve tried copying the packages package.json file into the container, via the Dockerfile, like so:
COPY ./packages/example-package/package.json ./packages/example-package
The resulting error is:
npm ERR! Could not install "packages/example-packages" as it is not a
directory and is not a file with a name ending in .tgz, .tar.gz or
.tar
I’m at a loss as to where to go from here! Any suggestions?
You end up with a file called ./packages/example-package with the contents of the package.json rather than a directory.
You can confirm this by temporarily adding a cat of the file/dir to your Dockerfile
RUN cat ./packages/example-package
Use a / on the end of the destination path to make it a directory or specify the full path to the file.
COPY ./packages/example-package/package.json ./packages/example-package/
COPY ./packages/example-package/package.json ./packages/example-package/package.json