My monorepo folder structure looks like this, but I haven't got cross app imports to work for yarn workspaces, that is, I can't import from shared to app1 or app2.
./
app1
app2
shared
server
package.json
Nearly all other workspaces I've seen includes a packages folder, is this required or just a convention? My folder structure would in that case look like:
./
packages
app1
app2
shared
server
package.json
Do I need to refactor for workspaces to work?
It's just a convention. You need to list your workspaces in package.json:
"workspaces": [
"app1",
"app2",
"shared",
"server"
],
The convention let's you use a glob (and presumably streamlines some other npm/yarn commands that you want to run across all packages):
"workspaces": ["packages/*"],
Related
I want to setup the bazel build system to build a monorepo with multiple JavaScript packages with the new rules_js rule set. The documentation of rules_js says that it supports "nested npm packages in a monorepo" via "workspaces", but I don't understand how to wire everything up so that:
each package in the monorepo can have different external dependencies
packages in the monorepo can also depend on each other
I tried using #npm//$DEPENDENCY in the deps of rules like js_library to refer to my npm dependencies (like with the older rules_nodejs ruleset), but I just get errors about non-existing targets.
TL;DR
Set up a pnpm workspace and use npm_link_all_packages multiple times to set up separate node_modules folders in your bin tree. Then refer to the dependencies in these node_modules folders by :node_modules/$DEPENDENCY in your deps.
How does rules_js create node_modules folders?
rules_js uses the npm_link_all_packages rule to set up node_modules folders based on a pnpm-lock.yaml lockfile. This is similar to the pnpm install command of the pnpm package manager. The difference is that pnpm install creates the node_modules folders in the source tree and npm_link_all_packages creates them in the bin tree.
Since you want that each package can have different external dependencies, they each need to have their own node_modules. pnpm supports a single lock file for multiple packages if you set up a pnpm workspace. The resulting lock file can be used by rules_js to set up a Bazel workspace with multiple packages that each get their own node_modules folder in the bin tree. The npm_link_all_packages rule will automatically set up the correct node_modules folder based on the Bazel package name and the pnpm-lock.yaml.
So you have to set up a pnpm workspace and use npm_link_all_packages multiple times to set up separate node_modules folders in your bin tree.
How exactly do I set this up?
To achieve this, you can put the following pieces in your workspace root:
pnpm-workspace.yaml to configure the list of pnpm workspace packages
pnpm-lock.yaml to record how all all direct and indirect npm dependencies are resolved
WORKSPACE.bazel to setup rules_js
package.json to avoid confusing pnpm, and in case you want global scripts or settings for your IDE or similar
BUILD.bazel to create the virtual store with npm_link_all_packages
And in each pnpm workspace package $PACKAGE you put:
$PACKAGE/package.json to configure the direct npm dependencies of $PACKAGE
$PACKAGE/BUILD.bazel to create the node_modules folder in the bin tree with npm_link_all_packages, and for your js_library or similar
So every pnpm workspace package is also a bazel package (since it has a BUILD.bazel).
How can I refer to my npm dependencies in deps?
In the deps of js_library and similar, you can point to the dependencies in the node_modules folders generated by npm_link_all_packages. Typically, the js_library is in the same Bazel package as the npm_link_all_package, so you can just use :node_modules/$DEPENDENCY. If the js_library is in a sub-package of $PACKAGE, you can use //$PACKAGE:node_modules/$DEPENDENCY instead. Then in your JavaScript files, you can import from "$DEPENDENCY" or require("$DEPENDENCY"). This will work out at runtime because the node_modules folder will be at an appropriate place in the runfiles for node to find it.
How can the packages in my workspace refer to each other?
If you want one of your packages $PACKAGE to depend on another one of your packages $OTHER, you put a "$OTHER": "workspace:*" into the $PACKAGE/package.json as usual with pnpm. You also have to make sure that the default target of $OTHER (for example, //some/path/to/other:other) is an npm_package rule. Then you can use :node_modules/$OTHER or //$PACKAGE:node_modules/$OTHER in the deps of a js_library or similar to refer to $OTHER just like you would refer to an npm dependency.
How can I create the pnpm-lock.yaml?
Note that this is a working pnpm setup with some additional BUILD.bazel files, so you can:
use pnpm install --lockfile-only to create the pnpm-lock.yaml
use pnpm install to create the pnpm-lock.yaml and also create node_modules folders in the source tree. These folders are not needed for rules_js but can be helpful for your IDE.
My repo has some gitignored paths that are sometimes populated during non-Bazel development and contain symlinks to other code paths that can sometimes cause the Bazel loader to too aggressively look for packages, which can cause bazel query to wrongly match packages through symlinks. Workspace root .bazelignore allows me to hide these directories but is there an option to tell Bazel to ignore a directory using a more local config rather than a single global file?
Specifically, I'm using Yarn Workspaces in Node.js. The source tree looks like the following:
BUILD
package.json
yarn.lock
myapp
├─ BUILD
└─ package.json
Running yarn install outside of Bazel creates a directory called node_modules, which includes third-party dependencies as well as a symlink farm to local packages:
BUILD
package.json
yarn.lock
node_modules
├─ myapp -> ../myapp
└─ some_3p_dep
myapp
├─ BUILD
└─ package.json
Expressing this pattern through Bazel is straightforward, and I've written a custom rule that just uses ctx.actions.declare_directory to create node_modules in the bindir (I have to tar/untar the symlink to conceal it from Bazel to prevent it from complaining, but the rule works great).
However, this results in a problem if a user runs the yarn install command locally without Bazel then runs something like bazel query //... | xargs bazel test to run all tests. In general bazel query //... appears to wrongly list not just targets in //myapp but also targets in //node_modules/myapp... after all node_modules/myapp/BUILD technically exists. This is problematic at the very least because it duplicates targets, but also in practice it causes ArtifactPrefixConflictException for any outputs that //node_modules/myapp declares because the paths overlap with the //:node_modules target/output. If I wanted to, I could technically work around the latter by messing with output names, but the duplication would still exist, wasting compute time and leading to different Bazel results depending on if the user is also doing a non-Bazel devloop.
.bazelignore appears to work fine to get Bazel not to traverse node_modules, but it only supports full paths (https://github.com/bazelbuild/bazel/issues/8106) and requires every new node_modules that may come into existence, which is cumbersome because it's a global file. Is there an option that we can declare the ignore more globally or another workaround I'm missing?
In serverless.yml file, how can I include the modules from parent folder?
package:
include:
- ../node_modules/**
but the packaged zip file doesn't include the dependencies from parent node_modules folder. I have tried the plugin: https://www.npmjs.com/package/serverless-plugin-include-dependencies but it still doesn't work. None of dependencies from parent folder is packaged.
How can I change the dependencies folder directory when packaging my application? I also want to apply excluding devDependencies logic for the parent folder.
I'd suggest to package your functions individually in some side script & then run the serverless deploy.
Then you can easily fine-tune your packaging process and include the node_modules from your parent, excluding the dev dependencies.
package:
individually: true
functions:
sample:
package:
artifact: ../functions/sample/deploy/sample.zip
Another small benefit on this is, if you only want to update some function you can explicitly just repackage that one and run the serverless deploy much faster.
I use 'electron-packager . --asar' command to package my project folder. but there are some folders which must not be included while packaging.
Is it possible to exclude some of folders?
If so, I would like to include those folders in exe file path so that the user can manage their files. Is there any way to do this?
Thank you in advance!
Is it possible to exclude some of folders?
Yes, you can do it with ignore
You can use --ignore to ignore files and folders via a regular
expression (not a glob pattern). Examples include --ignore=.gitignore
or --ignore=".git(ignore|modules)".
Take a look at API document here
If so, I would like to include those folders in exe file path so that the user can manage their files. Is there any way to do this?
You can use extraResource API to include them to resources directory (document here):
extraResource
String or Array of Strings
One or more files to be copied directly into the app's
Contents/Resources directory for OS X target platforms, and the
resources directory for other target platforms.
Hope this help.
I had similar problem. I wanted to exclude qpdf folder containing (*.exe, *.dll) from asar package. My qpdf.exe could not run without needed linked .dll directly from asar package.
I just added --extra-resource parameter into electron packager syntax where I am excluding whole qpdf folder. This folder and all files inside are then in resource folder with asar package.
Part of my package.json
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1",
"start": "electron .",
"package-win": "electron-packager . hiss_xread --overwrite --extra-resource=\"./qpdf\" --asar --platform=win32 --arch=x64 --icon=src/xRead.ico --prune=true --out=release-builds --version-string.CompanyName=CE --version-string.FileDescription=CE --version-string.ProductName=\"Hiss xRead\""
},
Resource folder in builded app look like this.
I use NPM to maintain front-end dependencies in my MVC 5 web project. One of the key uses I have for NPM is to get Typescript definitions for things.
The project is organised like this:
MySolution
|
|-- MySolution.Web
|
|-- Controllers
|-- node_modules
|-- Views
|-- (etc...)
What I want to know is if I should be including "node_modules" into the project, and if I should be including "node_modules" into source control.
The reason I ask is that everything seems happy with it being excluded (even the build agent, which does a "NPM restore" before compiling).
However, NCrunch is complaining that it can't find any reference to various types which my project's TypeScript files make reference to.
E.g. Admin.ts makes reference to "/node_modules/#types/knockout/index.d.ts"
I think that bringing in "node_modules" into the web project would fix it, but I'm not sure if it's right.
no, you shouldn't include this in source control or the project.
assume you do a full clone, from scratch, the first you would do is an npm install which would restore all packages and populate that folder anyway. It's exactly the same reason why you do not add things like NuGet packages either.
that folder can easily get very big and delay your CI/CD pipeline. As you said yourself, running npm install is something that can be done automatically anyway. If you download a git ignore file for Visual Studio for example, it will come with stuff like that ignored by default and for good reason.
As far as your typescript files are concerned, those should be fine as well. You'll do the restore first anyway before doing any work and you can use something like Visual Studio Code to work with your front end files and it doesn't matter if the node modules are part of your solution or not.