92

I have several apps in node that all share a few modules that I've written. These modules are not available via npm. I would like to be able to share freely between apps, but I don't want to copy directories around, nor rely on Git to do so. And I'm not really big on using symlinks to do this either.

I would like to arrange directories something like this:

app1
 server.js
 node_modules
  (public modules from npm needed for app1)
 lib
  (my own modules specific to app1)

app2
 server.js
 node_modules
  (public modules from npm needed for app2)
 lib
  (my own modules specific to app2)

shared_lib
 (my own modules that are used in both app1 and app2)

The problem I'm seeing is that the modules in shared_lib seem to get confused as to where to find the modules that will be in the node_modules directory of whichever app they are running in. At least I think that is the problem.

So....what is a good way to do this that avoids having duplicates of files? (note that I don't care about duplicates of things in node_modules, since those aren't my code, I don't check them into Git, etc)

rob
  • 9,933
  • 7
  • 42
  • 73

6 Answers6

53

The npm documentation recommends using npm-link to create your own Node.js packages locally, and then making them available to other Node.js applications. It's a simple four-step process.

A typical procedure would be to first create a package with the following structure:

  hello
  | index.js
  | package.json

A typical implementation of these files would be:

index.js

  exports.world = function() {
     return('Hello World');
  }

package.json

  {
    "name": "hello",
    "version": "0.0.1",
    "private": true,
    "main": "index.js",
    "dependencies": {
    },
    "engines": {
    "node": "v0.6.x"
    }
  }

"private:true" ensures that npm will refuse to publish the package. This is a way to prevent accidental publication of private packages.

Next, navigate to the root of your Node.js package folder and run npm link to link the package globally so it can be used in other applications.

To use this package in another application, e.g., "hello-world", with the following directory structure:

 hello-world
 | app.js

Navigate to the hello-world folder and run:

 npm link hello

Now you can use it like any other npm package like so:

app.js

  var http = require('http');
  var hello = require('hello');

  var server = http.createServer(function(req, res) {
     res.writeHead(200);
     res.end(hello.world());
  });
  server.listen(8080);
kas
  • 857
  • 1
  • 15
  • 21
almypal
  • 6,612
  • 4
  • 25
  • 25
  • 8
    How does this work when deploying to a PaaS environment such as Heroku or Nodejitsu? – Adam C May 04 '13 at 21:21
  • npm link won't work in a PaaS environment. Heroku honours node modules that are pushed in with your code. So that might be an option. – almypal May 05 '13 at 03:05
  • 2
    That's what I don't get - if you deploy to Heroku from git, you don't check your node_modules in. Also, that would imply that you copy you shared code into the node_modules folder before you check in. Which seems cumbersome and error prone. Or am I missing something? Thanks! – Adam C May 06 '13 at 04:49
  • That seems the only way out. The npm link option is only available if you have access to the remote terminal over ssh... as in AWS or Rackspace – almypal May 06 '13 at 07:08
  • Ok, I think the way to go is using dependencies defined as git repos or tarballs, then use npm link to make life easier during development... Thanks for your thoughts! – Adam C May 07 '13 at 06:30
  • 1
    I think you can use webpack to package the whole thing together for that purpose – Georgii Oleinikov Jun 21 '17 at 04:45
25

I've got this working by having node_modules folders at different levels - node then automatically traverses upwards until it finds the module.

Note you don't have to publish to npm to have a module inside of node_modules - just use:

"private": true

Inside each of your private package.json files - for your project I would have the following:

app1
 server.js
 node_modules
  (public modules from npm needed for app1)
  (private modules locally needed for app1)

app2
 server.js
 node_modules
  (public modules from npm needed for app2)
  (private modules locally needed for app2)

node_modules
  (public modules from npm needed for app1 & app2)
  (private modules locally for app1 & app2)

The point is node.js has a mechanism for dealing with this already and it's awesome. Just combine it with the 'private not on NPM' trick and you are good to go.

In short a:

require('somemodule')

From app A or B would cascade upwards until it found the module - regardless if it lived lower down or higher up. Indeed - this lets you hot-swap the location without changing any of the require(...) statements.

node.js module documentation

Bino Carlos
  • 1,357
  • 8
  • 10
  • 9
    When I tried this, any modules required by the shared modules could not be resolved. In your structure, it looks like the highest node_modules folder has both private (checked into source control) and public (npm install'd - not checked in) modules. I don't normall check public modules into source control - how did you get around this? – Adam C May 04 '13 at 21:19
6

Just use the correct path in your require call

For example in server.js that would be:

var moduleName = require('../shared_lib/moduleName/module.js');

Its Important to know that as soon as your path is prefixed with '/', '../', or './' the path is relative to the calling file.

For further information about nodes module loading visit: http://nodejs.org/docs/latest/api/modules.html

Taner Topal
  • 921
  • 8
  • 14
  • 9
    But if I do that, and then that module tries to require a module that comes from npm, it gets confused and gives errors. – rob Jul 01 '12 at 02:29
  • 3
    no answer? that's because you tried to build something more complicated than a demo with Node – Toolkit Jul 03 '18 at 18:16
  • this works great with esbuild targeting to lambda :thumbsup – ir0h Jun 10 '23 at 22:19
5

Yes, you can reference shared_lib from app1, but then you run into a problem if you want to package and deploy app1 to some other environment, such as a web server on AWS.

In this case, you're better off installing your modules in shared_lib to app1 and app2 using "npm install shared_lib/module". It will also install all the dependencies of the shared_lib modules in app1 and app2 and deal with conflicts/duplicates.

See this: How to install a private NPM module without my own registry?

Community
  • 1
  • 1
Mohamed Fakhreddine
  • 2,326
  • 2
  • 13
  • 10
3

If you check out the node.js docs, you'll see that Node.js understands the package.json file format as well, at least cursorily.

Basically, if you have a directory named foo, and in that directory is a package.json file with the key-value pair: "main": "myCode.js", then if you try to require("foo") and it finds this directory with a package.json file inside, it will then use foo/myCode.js for the foo module.

So, with your directory structure, if each shared lib has it's own directory with such a simple package.json file inside, then your apps can get the shared libs by:

var lib1 = require('../shared_lib/lib1');
var lib2 = require('../shared_lib/lib2');

And that should work for both of these apps.

0

Another solution can be cloning files from the other places into this repo:

clone.js:

const path = require('path')
const fs = require('fs')

const shared = [
  {
    type: 'file',
    source: '../app1',
    files: [
      'src/file1',
      'src/file2',
      '...'
    ],
  },
]

function cloneFiles(source, files) {
  const Reset = '\x1b[0m'
  const FgGreen = '\x1b[32m'
  console.log(`---------- Cloning ${files.length} files from ${source} ----------`)

  for (const file of files) {
    const sourceFile = path.join(__dirname, '..', source, file)
    const targetFile = path.join(__dirname, '..', file)

    process.stdout.write(` ${file} ... `)
    fs.copyFileSync(sourceFile, targetFile)
    console.log(`${FgGreen}Done!${Reset}`)
  }

  console.log(`---------- All done successfully ----------\n`)
}

;(() => {
  for (const item of shared) {
    switch (item.type) {
      case 'file':
        cloneFiles(item.source, item.files)
        break
    }
  }
})()

Then, in the package.json you can add this script and call it when you want to clone / sync files:

"clone": "node clone.js"
Farhad
  • 67
  • 1
  • 3