7

Related: single node_modules folder for multiple projects

If npm install -g everything is not recommended, and I do not want to link individual modules, can I possibly symlink <some project>/node_modules to a common directory to be shared by multiple projects?

Community
  • 1
  • 1
prusswan
  • 6,853
  • 4
  • 40
  • 61

2 Answers2

8

Node can handle the symlinks perfectly fine. How to achieve this is going to depend on some of your objectives. The most important being: what experience do you want to have for other developers who download your project(s) from version control?

When designing this experience, it is super helpful to read about the Node module loading algorithm, to get insight on what is possible.

In general, my recommendation is to not be concerned with duplicated dependencies between projects. "Fixing" this is not worth the maintenance cost, which includes dependency gridlock (conflicting needs of the subprojects) and needing custom tooling in some cases to account for your custom structure.

With that warning out of the way, how do we do it? The simplest way is to create a superproject that encapsulates various subprojects. The subprojects will effectively inherit the dependencies of the superproject.

superproject/
|-- node_modules/
|   +-- socket.io/
|-- package.json
|-- subprojectA/
|   |-- node_modules/
|   |   +-- browserify/
|   |-- package.json
|   +-- app/
|       +-- client.js
+-- subprojectB/
    |-- node_modules/
    |   +-- express/
    |-- package.json
    +-- lib/
        +-- server.js

This structure works how you might expect, the files within the subprojects can require() their own modules and any of those in superproject/node_modules, but they will not easily require() the modules within their sibling subprojects (it is still possible to do so via explicit paths). In other words, client.js can require() browserify and socket.io without a path, but it would need to use a path to require() express.

An important aspect of this is that npm does a "find up" search for a package.json and deals with modules in a node_modules directory as a sibling to that file when installing, etc. This means that your current working directory needs to be superproject in order to install modules in it, unless your subproject does not have a package.json file.

Seth Holladay
  • 8,951
  • 3
  • 34
  • 43
  • 2
    The "duplicated dependencies" is a huge concern, because the files take up quite a lot of space even for very small projects, and also lead to indirect waste of bandwidth when setting up new projects that are using (mostly) the same dependencies. – prusswan Mar 27 '16 at 01:43
  • 1
    I can relate to those desires. But I think you will find that tightly coupling various projects in this manner creates a different sort of burden. One where you can't update a dependency because project x needs a certain behavior. But you really want/need to update it because project y is using it differently and needs something new. In the old days, all npm modules were global, this practice stopped because the community realized it isn't worth it. https://nodejs.org/en/blog/npm/npm-1-0-global-vs-local-installation/ All that said, there are still ways to do pseudo-global modules. – Seth Holladay Mar 27 '16 at 02:21
  • 1
    The superproject is a brilliant solution! Thanks a lot. In my case I'm working on several Laravel PHP projects, which use node for development-only tasks (gulp). Gulp and require() only work when installed locally, but duplicating 200MB of node_modules for every 5MB PHP project is completely unreasonable, especially since I'm not doing any node development myself. The superproject is the only sensible solution I've found: I can have a laravel-5.2 folder with a single copy of the tools required by 5.2, alongside all 5.2 projects, and so on. Brilliant! – Tobia Jul 06 '16 at 10:50
  • 1
    @SethHolladay People have already faced this problem and it has been successfully solved in the history of OS development. There are literally hundreds (if not thousands) of shared libraries in any OS and multiple different versions of the same library can happily coexist just by using a simple rule - include the version in the name of the library. So a clever developer would have used the convention of `node_modules/package_name@MajorVersion.MinorVersion.BuildNumber/library.js` and then we would not have to fight against the walls .... – IVO GELOV Sep 11 '17 at 13:03
  • @IVOGELOV "multiple different versions of the same library can happily coexist" - right, and npm can do this, too. The only difference is that npm chooses to nest dependencies in a tree format when that becomes necessary, rather than always putting everything flat and duplicating version information in the module ID. That works better for local dependencies, which is recommended. To make it work for global shared dependencies, the module loader would need version information in the module ID. But no one wants to `require('express@4.0.0')`, because that belongs in only one place: package.json – Seth Holladay Sep 30 '17 at 00:48
  • @SethHolladay I admit that my experience with NPM and Node is not enough so please forgive me if my words sound stupid. In Linux when you **install** new version of a library the old version is not removed but a symlink (which does not contain version information in its name) is created to point to the latest version of the library. However, if you **upgrade** a library - current version is not kept. Thus consumers can use the latest version (whatever it is) through the symlink - or explicitly point to the desired version of the library. And it is the loader's job to check `package.json` – IVO GELOV Oct 01 '17 at 13:40
  • @SethHolladay and determine if `require` or `import` needs a specific version or whichever is the currently installed **latest** one. It also looks counter-intuitive to me that NPM refuses to work when `node_modules` in the project folder is a symlink. – IVO GELOV Oct 01 '17 at 13:46
  • 1
    `node_modules` being a symlink _should_ work just fine. I haven't specifically tested this, but if it doesn't work, that sounds like a bug. I'm not aware of any design choice or limitation that would impose a constraint where `node_modules` must not be a symlink. That said, maybe npm is naively clobbering it. As for, "it is the loader's job to check package.json", that would be nice. Unfortunately, it doesn't. Sounds like a good solution, though, and one that could potentially be implemented in userland. – Seth Holladay Oct 02 '17 at 05:52
  • Not to hijack the post but I'm facing a similar issue, see my SO post [here](https://stackoverflow.com/questions/65417495/how-can-i-require-node-modules-across-disk-partitions), and have tried the solution of creating a symlink from an external `node-modules` directory that puts it in the root of my project directory. My point is not to share the `node_modules` directory across different projects but to deal with a space constrained environment of a disk partitioned Raspberry Pi. – Brad W Dec 23 '20 at 17:14
  • Yeah this is why Deno goes another way, and stores all dependencies centrally... – Jerry Green Oct 10 '21 at 02:36
  • Accepting this because I finally resign to the fact that I don't have time for "fighting" the tools (npm, bundlers etc) any more than I already do. Trying get some random Node project with a well-specified package.json to work is already hard enough... – prusswan Jul 21 '22 at 09:31
7

In npm >= 7.21.0 you can't. npm will delete a symlink called node_modules when you do npm install. I'm using yarn as a workaround, which handles a symlinked node_modules folder fine.

user1169629
  • 441
  • 3
  • 12
  • 2
    Still looking into it, but this does seem to be working for me. The steps to try it would be 1) remove `node_modules` and `package-lock.json`, 2) create `node_modules` as a symlink: `ln -s /tmp/node_modules node_modules`, 3) install yarn globally: `npm install -g yarn`, 4) install packages using yarn: `yarn install`. The result, as far as I've seen so far, is that your `node_modules` folder continues to be a symlink and your packages are installed in the intended directory. Now to remember to always type `yarn install` instead of `npm install`... – jdgregson Nov 28 '22 at 23:05
  • Unfortunately, didn't work for me: `error An unexpected error occurred: "EEXIST: file already exists, mkdir '/home/[...]/node_modules'".` yarn 1.22.19 Too bad because my VPS is quite strapped for storage. – GuiRitter Apr 08 '23 at 14:43