Using a shared node module for common classes

Question

Goal

So I am having a project with this structure:

• ionic-app
• firebase-functions
• shared

The goal is to define common interfaces and classes in the shared module.

Restrictions

I don't want to upload my code to npm to use it locally and am not planning on uploading the code at all. It should 100% work offline.

While the development process should work offline, the ionic-app and firebase-functions modules are going to be deployed to firebase (hosting & functions). Therefore, the code from the shared module should be available there.

What I have tried so far

• I have tried using Project References in typescript, but I have not gotten it close to working
• I tried it with installing it as an npm module like in the second answer of this question
  • It seems to be working fine at first, but during the build, I get an error like this when running firebase deploy:

Function failed on loading user code. Error message: Code in file lib/index.js can't be loaded.
Did you list all required modules in the package.json dependencies?
Detailed stack trace: Error: Cannot find module 'shared'
    at Function.Module._resolveFilename (module.js:548:15)
    at Function.Module._load (module.js:475:25)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/srv/lib/index.js:5:18)

Question

Do you have a solution for making a shared module using either typescripts config, or NPM?

Please do not mark this as a duplicate → I have tried any solution I have found on StackOverflow.

Additional Info

Config for shared:

// package.json
{
  "name": "shared",
  "version": "1.0.0",
  "description": "",
  "main": "dist/src/index.js",
  "types": "dist/src/index.d.ts",
  "files": [
    "dist/src/**/*"
  ],
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "publishConfig": {
    "access": "private"
  }
}

// tsconfig.json
{
  "compilerOptions": {
    "module": "commonjs",
    "rootDir": ".",
    "sourceRoot": "src",
    "outDir": "dist",
    "sourceMap": true,
    "declaration": true,
    "target": "es2017"
  }
}

Config for functions:

// package.json
{
  "name": "functions",
  "scripts": {
    "lint": "tslint --project tsconfig.json",
    "build": "tsc",
    "serve": "npm run build && firebase serve --only functions",
    "shell": "npm run build && firebase functions:shell",
    "start": "npm run shell",
    "deploy": "firebase deploy --only functions",
    "logs": "firebase functions:log"
  },
  "engines": {
    "node": "8"
  },
  "main": "lib/index.js",
  "dependencies": {
    "firebase-admin": "^8.0.0",
    "firebase-functions": "^3.1.0",
    "shared": "file:../../shared"
  },
  "devDependencies": {
    "@types/braintree": "^2.20.0",
    "tslint": "^5.12.0",
    "typescript": "^3.2.2"
  },
  "private": true
}


// tsconfig.json
{
  "compilerOptions": {
    "baseUrl": "./",
    "module": "commonjs",
    "noImplicitReturns": true,
    "noUnusedLocals": false,
    "rootDir": "src",
    "outDir": "lib",
    "sourceMap": true,
    "strict": true,
    "target": "es2017"
  }
}

Current soution

I have added a npm script to the shared module, which copies all files (without the index.js) to the other modules. This has the problem, that I check in duplicate code into SCM, and that I need to run that command on every change. Also, the IDE just treats it as different files.

Answer 1

Preface: I'm not too familiar with how Typescript compilation works and how package.json in such a module should be defined. This solution, although it works, could be considered a hacky way of achieving the task at hand.

Assuming the following directory structure:

project/
  ionic-app/
    package.json
  functions/
    src/
      index.ts
    lib/
      index.js
    package.json
  shared/
    src/
      shared.ts
    lib/
      shared.js
    package.json

When deploying a Firebase service, you can attach commands to the predeploy and postdeploy hooks. This is done in firebase.json via the properties predeploy and postdeploy on the desired service. These properties contain an array of sequential commands to run before and after deploying your code respectively. Furthermore, these commands are called with the environment variables RESOURCE_DIR (the directory path of ./functions or ./ionic-app, whichever is applicable) and PROJECT_DIR (the directory path containing firebase.json).

Using the predeploy array for functions inside firebase.json, we can copy the shared library's code into the folder that is deployed to the Cloud Functions instance. By doing this, you can simply include the shared code as if it were a library located in a subfolder or you can map it's name using Typescript's path mapping in tsconfig.json to a named module (so you can use import { hiThere } from 'shared';).

The predeploy hook definition (uses global install of shx for Windows compatibility):

// firebase.json
{
  "functions": {
    "predeploy": [
      "shx rm -rf \"$RESOURCE_DIR/src/shared\"", // delete existing files
      "shx cp -R \"$PROJECT_DIR/shared/.\" \"$RESOURCE_DIR/src/shared\"", // copy latest version
      "npm --prefix \"$RESOURCE_DIR\" run lint", // lint & compile
      "npm --prefix \"$RESOURCE_DIR\" run build"
    ]
  },
  "hosting": {
    "public": "ionic-app",
    ...
  }
}

Linking the copied library's typescript source to the functions typescript compiler config:

// functions/tsconfig.json
{
  "compilerOptions": {
    ...,
    "baseUrl": "./src",
    "paths": {
      "shared": ["shared/src"]
    }
  },
  "include": [
    "src"
  ],
  ...
}

Associating the module name, "shared", to the copied library's package folder.

// functions/package.json
{
  "name": "functions",
  "scripts": {
    ...
  },
  "engines": {
    "node": "8"
  },
  "main": "lib/index.js",
  "dependencies": {
    "firebase-admin": "^8.6.0",
    "firebase-functions": "^3.3.0",
    "shared": "file:./src/shared",
    ...
  },
  "devDependencies": {
    "tslint": "^5.12.0",
    "typescript": "^3.2.2",
    "firebase-functions-test": "^0.1.6"
  },
  "private": true
}

The same approach can be used with the hosting folder.

---

Hopefully this inspires someone who is more familiar with Typescript compilation to come up with a cleaner solution that makes use of these hooks.

Answer 2

You might want to try Lerna, a tool for managing JavaScript (and TypeScript) projects with multiple packages.

Setup

Assuming that your project has the following directory structure:

packages
  ionic-app
    package.json
  firebase-functions
    package.json
  shared
    package.json

Make sure to specify the correct access level (private and config/access keys) in all the modules you don't wish to have published, as well as the typings entry in your shared module:

Shared:

{
  "name": "shared",
  "version": "1.0.0",
  "private": true,
  "config": {
    "access": "private"
  },
  "main": "lib/index.js",
  "typings": "lib/index.d.ts",
  "scripts": {
    "compile": "tsc --project tsconfig.json"
  }
}

Ionic-app:

{
  "name": "ionic-app",
  "version": "1.0.0",
  "private": true,
  "config": {
    "access": "private"
  },
  "main": "lib/index.js",
  "scripts": {
    "compile": "tsc --project tsconfig.json"
  },
  "dependencies": {
    "shared": "1.0.0"
  }
}

With the above changes in place, you can create a root-level package.json where you can specify any devDependencies that you wish all your project modules to have access to, such as your unit testing framework, tslint, etc.

packages
  ionic-app
    package.json
  firebase-functions
    package.json
  shared
    package.json
package.json         // root-level, same as the `packages` dir

You can also use this root-level package.json to define npm scripts that will invoke the corresponding scripts in your project's modules (via lerna):

{
  "name": "my-project",
  "version": "1.0.0",
  "private": true,
  "scripts": {
    "compile": "lerna run compile --stream",
    "postinstall": "lerna bootstrap",
  },
  "devDependencies": {
    "lerna": "^3.18.4",
    "tslint": "^5.20.1",
    "typescript": "^3.7.2"
  },
}

With that in place, add the lerna config file in your root directory:

packages
  ionic-app
    package.json
  firebase-functions
    package.json
  shared
    package.json
package.json
lerna.json

with following contents:

{
  "lerna": "3.18.4",
  "loglevel": "info",
  "packages": [
    "packages/*"
  ],
  "version": "1.0.0"
}

Now when you run npm install in the root directory, the postinstall script defined in your root-level package.json will invoke lerna bootstrap.

What lerna bootstrap does is that it will symlink your shared module to ionic-app/node_modules/shared and firebase-functions/node_modules/shared, so from the point of those two modules, shared looks just like any other npm module.

Compilation

Of course, symlinking the modules is not enough as you still need to compile them from TypeScript to JavaScript.

That's where the root-level package.json compile script comes into play.

When you run npm run compile in your project root, npm will invoke lerna run compile --stream, and lerna run compile --stream invokes the script called compile in each of your modules' package.json file.

Since each of your modules now has its own compile script, you should can have a tsonfig.json file per module. If you don't like the duplication you could get away with a root-level tsconfig, or a combination of a root-level tsconfig and module-level tsconfig files inheriting from the root one.

If you'd like to see how this setup works on a real-world project, have a look at Serenity/JS where I've been using it quite extensively.

Deployment

The nice thing about having the shared module symlinked under node_modules under firebase-functions and ionic-app, and your devDepedencies under node_modules under project root is that if you need to deploy the consumer module anywhere (so the ionic-app for example), you could just zip it all up together with its node_modules and not worry about having to remove the dev depedencies prior to deployment.

Hope this helps!

Jan

Answer 3

Another possible solution, if you are using git to manage your code, is using git submodule. Using git submodule you are able to include another git repository into your project.

Applied to your use case:

1. Push the current version of your shared-git-repository
2. Use git submodule add <shared-git-repository-link> inside your main project(s) to link the shared repository.

Here is a link to the documentation: https://git-scm.com/docs/git-submodule

Answer 4

Here's my two cents on this matter, adding to samthecodingman's answer:

I achieved the same results on linux with a simple symlink to the shared module build folder using an additional bootstrap step which would 'replace' the call to npm install:

    // ./functions/package.json
    {
    ...
      "scripts": {
        "bootstrap": "npm install && ln -s ../shared/lib shared",
    ...
      "dependencies": {
        "shared": "file:shared",
    ...
    }

Remove the symlink from versioning:

    // ./functions/.gitignore
    shared

Shared module tsconfig:

    // ./shared/tsconfig.json
    {
    ...
      "dependencies": {
        "outDir": "lib",
        // outputs .d.ts files as well, allowing other modules to point directly
        // to build folder while enjoying type safety
        "declaration": true,
    ...
    }

Since the symlink is present inside the functions folder, it's contents are copied during firebase deploy.

Obs.: The hosting module, on the other hand, can depend on the shared module build folder directly, as long as you are using webpack or similar:

    // ./hosting/package.json
    {
      "dependencies": {
        "shared": "file:../shared/lib",
    ...
    }

Answer 5

The tool you are looking for is npm link. npm link provides symlinks to a local npm package. That way you can link a package and use it in your main project without publishing it to the npm package library.

Applied to your use case:

1. Use npm link inside your shared package. This will set the symlink destination for future installs.
2. Navigate to your main project(s). Inside your functions package and use npm link shared to link the shared package and add it to the node_modules directory.

Here is a link to the documentation: https://docs.npmjs.com/cli/link.html

Answer 6

If I understand your problem correctly, the solution is more complex than a single answer and it partly depends on your preference.

Approach 1: Local copies

You can use Gulp to automate the working solution you described already, but IMO it is not very easy to maintain and drastically increases the complexity if at some point another developer comes in.

Approach 2: Monorepo

You can create a single repository that contains all three folders and connect them so they behave as a single project. As already answered above, you can use Lerna. It requires a bit of a configuration, but once done, those folders will behave as a single project.

Approach 3: Components

Treat each one of these folders as a stand-alone component. Take a look at Bit. It will allow you to set up the folders as smaller parts of a bigger project and you can create a private account that will scope those components only to you. Once initially set up, it will allow you to even apply updates to the separate folders and the parent one that uses them will automatically get the updates.

Approach 4: Packages

You specifically said that you don't want to use npm, but i want to share it, because I am currently working with a set up as described below and is doing a perfect job for me:

1. Use npm or yarn to create a package for each folder(you can create scoped packages for both of them so the code will only available to you, if this is your concern).
2. In the parent folder(that uses all of these folders), the created packages are connected as dependencies.
3. I use webpack to bundle all the code, using webpack path aliases in combination with typescript paths.

Works like a charm and when the packages are symlinked for local development it works entirely offline and in my experience - each folder is scalable separately and very easy to maintain.

Note

The 'child' packages are already precompiled in my case as they are pretty big and I have created separate tsconfigs for each package, but the beautiful thing is that you can change it easily. In the past I've both used typescript in the module and compiled files, and also raw js files, so the entire thing is very, very versatile.

Hope this helps

*****UPDATE**** To continue on point 4: I apologize, my bad. Maybe I got it wrong because as far as I know, you cannot symlink a module if it is not uploaded. Nevertheless, here it is:

1. You have a separate npm module, let's use firebase-functions for that. You compile it, or use raw ts, depending on your preference.
2. In your parent project add firebase-functions as a dependency.
3. In tsconfig.json, add "paths": {"firebase-functions: ['node_modules/firebase-functions']"}
4. In webpack - resolve: {extensions: ['ts', 'js'], alias: 'firebase-functions': }

This way, you reference all your exported functions from the firebase-functions module simply by using import { Something } from 'firebase-functions'. Webpack and TypeScript will link it to the node modules folder. With this configuration, the parent project will not care if the firebase-functions module is written in TypeScript or vanilla javascript.

Once set up, it will work perfectly for production. Then, to link and work offline:

1. Navigate to firebase-functions project and write npm link. It will create a symlink, local to your machine and will map the link the name you set in package.json.
2. Navigate to parent project and write npm link firebase-functions, which will create the symlink and map the dependency of firebase-functions to the folder in you have created it.

Answer 7

I don't want to upload my code to npm to use it locally and am not planning on uploading the code at all. It should 100% work offline.

All npm modules are installed locally and always work offline, but if you do not want to publish your packages publicly so people can see it, you can install private npm registry.

ProGet is NuGet/Npm Private repository server available for windows which you can use in your private development/production environment to host, access and publish your private packages. Though it is on windows but I am sure there are various alternatives available on linux.

1. Git Submodules is bad idea, it is really an old fashion way to share code which is not versioned like packages are, changing and committing submodules are real pain.
2. Source import folder is also a bad idea, again versioning is issue, because if someone modifies dependent folder in dependent repository, again tracking it is nightmare.
3. Any third party scripted tools to emulate package separation is waste of time as npm already provides range of tools to manage packages so well.

Here is our build/deployment scenario.

1. Every private package has .npmrc which contains registry=https://private-npm-repository.
2. We publish all our private packages onto our privately hosted ProGet repository.
3. Every private package contains dependent private packages on ProGet.
4. Our build server accesses ProGet via npm authentication set by us. Nobody outside our network has access to this repository.
5. Our build server creates npm package with bundled dependencies which contains all packages inside node_modules and production server never needs to access NPM or private NPM packages as all necessary packages are already bundled.

Using private npm repository has various advantages,

1. No need of custom script
2. Fits into node buid/publish pipeline
3. Every private npm package will contain direct link to your private git source control, easy to debug and investigate errors in future
4. Every package is a readonly snapshot, so once published cannot be modified, and while you are making new features, existing code base with older version of dependent packages will not be affected.
5. You can easily make some packages public and move to some other repository in future
6. If your private npm provider software changes, for example you decide to move your code into node's private npm package registry cloud, you will not need to make any changes into your code.