I'm trying to use bellow syntax:
cache:
bundler: true
directories:
- foldername/foldername/node_modules
After the build is done I see 11 MB of cache on the 'Cahes' page in Travis, but in real life node_modules is about 60 mb and I can't say that I have some perfomance improvment.
Is it ok or maybe I'm doing something wrong?
This is how I have cached my folders for nodejs.
# .travis.yml
language: node_js
node_js:
- "stable"
cache:
directories:
- node_modules
Note: Folders located under directories are relative to the .travis.yml file.
The official documentation has more information.
Related
I have currently a problem to build and to run my app using bitbucket pipeline:
I have a error about the cache folder which is not writable:
warning Skipping preferred cache folder "/home/node/.cache/yarn" because it is not writable.
warning Skipping preferred cache folder "/tmp/.yarn-cache-1000" because it is not writable.
warning Skipping preferred cache folder "/tmp/.yarn-cache" because it is not writable.
error Yarn hasn't been able to find a cache folder it can use. Please use the explicit --
cache-folder option to tell it what location to use, or make one of the preferred locations
writable.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
I'm using a bitbucket-pipelines.yml and a dockerfile.
Thank you
I came across the same problem (also using Docker and Bitbucket Pipelines). Except that I was using k8s to deploy the app.
What I did was change the next line in my deployment to false:
securityContext:
readOnlyRootFilesystem: false (it was `true`)
It solved it for me!
I am using Rails 6 in a VirtualBox virtual machine through Vagrant.
Vagrant mounts shared folder (with project) from the host system (Windows) to the Ubuntu 20LTS virtual machine.
And it happens that in my case symlinks don't work in this shared folder.
(There is a key with which the symlinks should work, but it did not happen)
And here's the problem.
When yarn installs packages in node_modules it creates symlinks as well.
And then I get an error because the symlinks don't work:
error An unexpected error occurred: "EPROTO: protocol error, symlink
Ok, I thought that folder node_modules can be moved outside of the project and the shared folder in the virtual machine, where the symlinks will work.
And indeed, you can create a .yarnrc file in the project folder with the contents:
"--modules-folder" "/home/user/custom_path/node_modules"
Now running yarn add or rake webpacker:install will install all packages in the specified directory.
It would seem that the problem is solved.
However, Webpacker doesn't know anything about .yarnrc and keeps looking for it in the root of the project.
And if for example you call rake webpacker:info:
Ruby: ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
Rails: 6.1.4.1
Webpacker: 5.4.3
Node: v16.13.0
Yarn: 1.22.17
#rails/webpacker:
project#0.1.0 /home/vagrant/data/shared/project
└── (empty)
Is bin/webpack present?: true
Is bin/webpack-dev-server present?: true
Is bin/yarn present?: true
empty - It doesn't see those #rails/webpacker.
However, its in custom node_modules.
I tried writing in config/webpacker.yml
additional_paths: ["/home/user/custom_path/node_modules"]
However, this doesn't have the desired effect.
And at the compile stage I get an error:
Error: Cannot find module '#rails/webpacker'
package.json is in the root of the project and this and other packages are prescribed there.
Question: how to correctly specify the path to node_modules to make everything work?
Yes it's all in the context of development/test environment for rails.
What seems to work for me:
Make sure the folder is still called node_modules, wherever it is.
Make sure it is in a parent folder of your Rails project
Webpack seems to keep stepping up a folder looking for node_modules
Tell Webpacker where to find the .bin folder with WEBPACKER_NODE_MODULES_BIN_PATH env variable
Webpacker tries to use yarn bin but that doesn't respect the --modules-folder setting in .yarnrc
Use the environmental variable to override: export WEBPACKER_NODE_MODULES_BIN_PATH="/node_modules/.bin"
My CI process, to ensure reliability of generated files, rm -rf all __generated__ dirs then generates again. It seems there is a problem (somewhere), as my results are occasionally different on OSX (local) vs Linux (CI). I'm wondering if it is a timing problem with file events and watchman.
Then I thought, do I need to wipe out these files or will relay-compiler delete old/unused generated files?
In my experience it deletes files. Here's an example from a terminal I have open:
Parsed default in 0.32s
Writing default
Writer time: 0.46s [0.09s compiling, 0.37s generating, 0.00s extra]
Created:
- deleteCustomsBondMutation.graphql.js
- putCustomsBondMutation.graphql.js
Updated:
- BondInformationQuery.graphql.js
Deleted:
- deleteCustomsBondMutation.graphql.js
- putCustomsBondMutation.graphql.js
Unchanged: 5 files
Written default in 0.51s
Watching for changes to default...
Notice that it deleted two files. This was because I moved my mutations into a different directory.
I'm used to building ng applications using just Angular and with this typical structure:
app
js
home
home.controller.js
home.directive.js
shared
usedByAll.controller.js
usedByAll.directive.js
templates
home.html
test
e2e
protractor tests
node_modules
index.html
package.json
So that when someone else coming in wants to catch up they just need to run
npm install
to install all the dependencies listed into their node_modules folder saving them time.
But recently I've started working on a Ruby-on-Rails project that is looking to angularize some of its components. This naturally means they're following their own MVC style and project structure and the angular stuff has been restricted to the following folders:
app/assets/javascripts/angular-components/component1
component2
component3
lib/angular.js
angular-mocks.js
etc
This is fine for development but when it comes down to testing I think this could be a problem. So my questions are really the following:
1 - When installing new modules via npm and attempting to save them as devDependencies
npm install karma --save-dev
to the package.json file, doesn't it mean you should have the node_modules folder at the root as well as the package.json file at the root?
2 - Should the karma.conf.js file, like the node_modules folder and package.json file, always be at the absolute root of the application?
Thanks
In many rails/angular applications you can see that Angular is tucked away inside the app/assets/javascripts folder. I personally don't like mixing and matching what I'm using when it comes to pages - i.e. go and get a Rails generated view or this one is an Angular one so go there.
Instead I broke out the two. Essentially the Rails part of this just functions as an API whereas Angular now handles the front end entirely. This allows you to treat the angular part nearly as a standalone app, following the typical angular structure. By doing this you can have the karma.conf.js file at the root of this folder.
ExampleAppName
- app (All Angular stuff is in here)
- src
- gulpfile.js
- karma.conf.js
- package.json
- api (All rails stuff is in here)
- rails stuff
I change some text in my application and deploy new version with
git push heroku master
But everytime I push, it increase a lot. I write here the increasement after each deploy.
168.2 KB
37.9 MB
178.6 KB
38.7 MB
187 KB
39.4 MB
194 KB
40.2 MB
205.3 KB
40.9 MB
232.8 KB
41.8 MB
277.9 KB
42.4 MB
286.5 KB
43 MB
If i select all my folders in my app project (including public/, tmp/ and log/ that I ignore with slugignore e gitignore) total size is only 198 KB.
Why all this?
There are a few steps that can be followed to reduce the slug size.
Purge build-cache
Buildpacks cache some content to speed up future builds. But sometimes, when your dependencies change (especially when you remove a dependency), it might not be removed from the cache. So if you are facing unexpectedly high slug sizes and you have removed some dependencies from your project, try this first (if you can live with a slow build on the next deploy).
$ heroku plugins:install heroku-repo
$ heroku repo:purge_cache -a appname
Ignore files with .slugignore
Your .sligignore should list directories and files that are in the repo (not ignored by .gitignore) but not needed for the app to run on prod. Some examples of files like these are the tests, documentation and design files. To remove these from the slug, create a .slugignore at the root of the project.
*.psd
/doc
/test
To check out what files are being put into the slug, you can also enter the heroku
bash and check the filesystem.
heroku run bash -a appname
du -sh .[^.]* * | sort -hr
Node modules
This part is specific to apps that include an asset pipeline that bundles all JavaScript code to single script. If you are using a node server, this might not apply to you.
If you are using Rails with webpacker, you might have a lot of dependencies in node_modules. But after the asset pipeline compiles the resources, you probably don't need the node_modules directory (unless you run custom node scripts at runtime that use those modules). To remove the node_modules directory in this case, we can use the heroku-buildpack-post-build-clean buildpack with a .slug-post-clean file. The format of this file is the same as .gitignore or .sligignore, so just list out directories that you want to ignore from the final slug and they will be removed. One thing to make sure with the buildpack is that this should be the last one in the order otherwise it will remove the files before your actual buildpack runs.
But what if you are using custom node scripts that access the node_modules at runtime? In that case, what we do is move those scripts to a separate directory with it's own package.json file. To install the dependencies into that directory along with the installation of root package.json, add a postinstall script to your root package.json. This will instruct yarn to change directory to your custom script and install dependencies there after the dependencies of your root project are installed.
"scripts": {
"postinstall": "yarn --cwd lib/custom-node-script"
}
With this, you can then safely remove your root node_modules directory even when using custom node scripts in a production Rails app.
Here is the detailed post addressing the same issue.
It's most likely due to build cache. To speed up build process, Heroku downloads dependencies, and over time it gets built up.
Use their repo plugin to purge build cache, and re-deploy. You will see the slug size being lowered.
https://github.com/heroku/heroku-repo
Here is more about their build cache: https://devcenter.heroku.com/articles/git#build-cache