Node can't find certain modules after synchronous install

Question

I've got a script that synchronously installs non-built-in modules at startup that looks like this

const cp = require('child_process')

function requireOrInstall (module) {
  try {
    require.resolve(module)
  } catch (e) {
    console.log(`Could not resolve "${module}"\nInstalling`)
    cp.execSync(`npm install ${module}`)
    console.log(`"${module}" has been installed`)
  }
  console.log(`Requiring "${module}"`)
  try {
    return require(module)
  } catch (e) {
    console.log(require.cache)
    console.log(e)
  }
}

const http    = require('http')
const path    = require('path')
const fs      = require('fs')
const ffp     = requireOrInstall('find-free-port')
const express = requireOrInstall('express')
const socket  = requireOrInstall('socket.io')
// List goes on...

When I uninstall modules, they get installed successfully when I start the server again, which is what I want. However, the script starts throwing Cannot find module errors when I uninstall the first or first two modules of the list that use the function requireOrInstall. That's right, the errors only occur when the script has to install either the first or the first two modules, not when only the second module needs installing.

In this example, the error will be thrown when I uninstall find-free-port, unless I move its require at least one spot down ¯\_(• _ •)_/¯

I've also tried adding a delay directly after the synchronous install to give it a little more breathing time with the following two lines:

var until = new Date().getTime() + 1000
while (new Date().getTime() < until) {}

The pause was there. It didn't fix anything.

@velocityzen came with the idea to check the cache, which I've now added to the script. It doesn't show anything out of the ordinary.

@vaughan's comment on another question noted that this exact error occurs when requiring a module twice. I've changed the script to use require.resolve(), but the error still remains.

Does anybody know what could be causing this?

Edit

Since the question has been answered, I'm posting the one-liner (139 characters!). It doesn't globally define child_modules, has no last try-catch and doesn't log anything in the console:

const req=async m=>{let r=require;try{r.resolve(m)}catch(e){r('child_process').execSync('npm i '+m);await setImmediate(()=>{})}return r(m)}

The name of the function is req() and can be used like in @alex-rokabilis' answer.

I sort of wonder what problem you're _actually_ trying to solve with this. — robertklep, Nov 12 '18 at 21:50
I simply want the modules to be installed and found in one go, without errors. That's all. — Gust van de Wal, Nov 12 '18 at 22:01
That's what `package.json` and `npm install` are for, though. — robertklep, Nov 12 '18 at 22:02
I know. Only having to add `const package = require('package')` because of a line of 156 characters (minified) at the top of the script is a nice perk, though. — Gust van de Wal, Nov 12 '18 at 22:11
Node.js `require` has cache, try to check for the problem there? `require.cache` — velocityzen, Nov 15 '18 at 22:12
`require` is more than just a function (it has [various properties attached to it](https://nodejs.org/api/modules.html#modules_require)). So it's not entirely surprising that monkeypatching it leads to unexpected behaviour. — robertklep, Nov 16 '18 at 13:06
you can't solve this without messing with node's internals, not an easy problem — mihai, Nov 16 '18 at 13:36
cp.execSync is an async call so try check if the module is installed in it's call back function. I have tried it, installation is clean now: — Arulmozhi Manikandan, Nov 19 '18 at 16:26

score 1 · Answer 1 · answered Nov 19 '18 at 14:22

I think your best option is either:

(ugly) to install package globally, instead of locally
(best solution ?) to define YOUR new 'package repository installation', when installing, AND when requiring

First, you may consider using the npm-programmatic package.

Then, you may define your repository path with something like:

const PATH='/tmp/myNodeModuleRepository';

Then, replace your installation instruction with something like:

const npm = require('npm-programmatic');
npm.install(`${module}`, {
        cwd: PATH,
        save:true
}

Eventually, replace your failback require instruction, with something like:

return require(module, { paths: [ PATH ] });

If it is still not working, you may update the require.cache variable, for instance to invalide a module, you can do something like:

delete require.cache[process.cwd() + 'node_modules/bluebird/js/release/bluebird.js'];

You may need to update it manually, to add information about your new module, before loading it.

Alex Michailidis · Accepted Answer · 2018-11-20T09:43:52.997

It seems that the require operation after an npm install needs a certain delay. Also the problem is worse in windows, it will always fail if the module needs to be npm installed. It's like at a specific event snapshot is already known what modules can be required and what cannot. Probably that's why require.cache was mentioned in the comments. Nevertheless I suggest you to check the 2 following solutions.

1) Use a delay

const cp = require("child_process");

const requireOrInstall = async module => {
  try {
    require.resolve(module);
  } catch (e) {
    console.log(`Could not resolve "${module}"\nInstalling`);
    cp.execSync(`npm install ${module}`);
    // Use one of the two awaits below
    // The first one waits 1000 milliseconds
    // The other waits until the next event cycle
    // Both work
    await new Promise(resolve => setTimeout(() => resolve(), 1000));
    await new Promise(resolve => setImmediate(() => resolve()));
    console.log(`"${module}" has been installed`);
  }
  console.log(`Requiring "${module}"`);
  try {
    return require(module);
  } catch (e) {
    console.log(require.cache);
    console.log(e);
  }
}

const main = async() => {
  const http = require("http");
  const path = require("path");
  const fs = require("fs");
  const ffp = await requireOrInstall("find-free-port");
  const express = await requireOrInstall("express");
  const socket = await requireOrInstall("socket.io");
}

main();

await always needs a promise to work with, but it's not needed to explicitly create one as await will wrap whatever it is waiting for in a promise if it isn't handed one.

2) Use a cluster

const cp = require("child_process");

function requireOrInstall(module) {
  try {
    require.resolve(module);
  } catch (e) {
    console.log(`Could not resolve "${module}"\nInstalling`);
    cp.execSync(`npm install ${module}`);
    console.log(`"${module}" has been installed`);
  }
  console.log(`Requiring "${module}"`);
  try {
    return require(module);
  } catch (e) {
    console.log(require.cache);
    console.log(e);
    process.exit(1007);
  }
}

const cluster = require("cluster");

if (cluster.isMaster) {
  cluster.fork();
  cluster.on("exit", (worker, code, signal) => {
    if (code === 1007) {
      cluster.fork();
    }
  });
} else if (cluster.isWorker) {
  // The real work here for the worker

  const http = require("http");
  const path = require("path");
  const fs = require("fs");
  const ffp = requireOrInstall("find-free-port");
  const express = requireOrInstall("express");
  const socket = requireOrInstall("socket.io");

  process.exit(0);
}

The idea here is to re-run the process in case of a missing module. This way we fully reproduce a manual npm install so as you guess it works! Also it seems more synchronous rather the first option, but a bit more complex.

I used your first solution. It's elegant ES6 code and it did the trick with only a couple of extra lines and without the use of third party packages (which was kind of the whole point for me). Thanks! — Gust van de Wal, Nov 19 '18 at 20:52
I am glad that you used/liked my answer! My only concern is about `await setTimeout(() => {}, 1000);` Are you sure it can work like this? Without a promise? I cannot reproduce the delay that way, in chrome at least. — Alex Michailidis, Nov 20 '18 at 07:00
I seem to have blindly pasted the `setTimeout` option in there after it worked in a test. I hadn't noticed it didn't actually wait 1 second. Good catch! — Gust van de Wal, Nov 20 '18 at 09:07

Arulmozhi Manikandan · Answer 3 · 2018-11-19T20:14:06.680

cp.execSync is an async call so try check if the module is installed in it's call back function. I have tried it, installation is clean now:

const cp = require('child_process')

function requireOrInstall (module) {
    try {
        require.resolve(module)
    } catch (e) {
        console.log(`Could not resolve "${module}"\nInstalling`)
        cp.execSync(`npm install ${module}`, () => {
            console.log(`"${module}" has been installed`)
            try {
                return require(module)
            } catch (e) {
                console.log(require.cache)
                console.log(e)
            }
        })

    }
    console.log(`Requiring "${module}"`)

}

const http    = require('http')
const path    = require('path')
const fs      = require('fs')
const ffp     = requireOrInstall('find-free-port')
const express = requireOrInstall('express')
const socket  = requireOrInstall('socket.io')

When node_modules not available yet :

When node_modules available already:

`.exec()` is asynchronous, not `.execSync()`. it therefore doesn't have a callback function. Returning the required module also won't work here, because you're returning the require from inside the callback, not `requireOrInstall()`. It's not the installation that fails, but the modules are unfindable. — Gust van de Wal, Nov 19 '18 at 20:18

Node can't find certain modules after synchronous install

3 Answers3

1) Use a delay

2) Use a cluster

Linked