How to use dynamic import from a dependency in Node.js?

Question

I'm using Node.js (v16) dynamic imports in a project to load plugins using a function loadJsPlugin shown here:

import { pathToFileURL } from 'url';

async function loadJsPlugin(pluginPath) {
  const pluginURL = pathToFileURL(pluginPath).toString();
  const result = await import(pluginURL);
  return result.default;
}

My main program provides absolute paths to the loadJsPlugin function, such as /home/sparky/example/plugins/plugin1.js (Linux) or C:\Users\sparky\example\plugins\plugin1.js (Windows). The pathToFileURL function then converts these absolute paths to URLs like file:///home/sparky/example/plugins/plugin1.js (Linux) or file:///C:/Users/sparky/example/plugins/plugin1.js (Windows).

Loading the plugins this way works fine when the loadJsPlugin function is in the same package as the main program, like this:

import { loadJsPlugin } from './plugin-loader.js';

async function doSomething() {
  const plugin = await loadJsPlugin('...'); // works
  // use plugin
}

However, if I try to move loadJsPlugin to a separate library and use it from there, it fails with Error: Cannot find module '<url here>'

import { loadJsPlugin } from '@example/plugin-loader';

async function doSomething() {
  const plugin = await loadJsPlugin('...'); // error
  // use plugin
}

NOTE: the dependency name here is not on NPM, it's on a private repository and there's no problem loading the dependency itself. Also, static ES6 imports in general are working fine in this system.

I looked through Node.js documentation, MDN documentation, and other StackOverflow questions for information about what is allowed or not, or whether dynamic import works differently when in the same package or a dependency, and didn't find anything about this. As far as I can tell, if a relative path or file URL is provided, and the file is found, it should work.

Ruling out file not found:

1. I can switch back and forth between the two import lines to load the loadJsPlugin function from either ./plugin-loader.js or @example/plugin-loader, give it the same input, and the one in the same package works while the one from the dependency doesn't.

2. When I test in VS Code, I can hover the mouse over the URL in the Error: Cannot find module 'file:///...' message and the file opens just fine

3. I can also copy the 'file:///...' URL to a curl command (Linux) or paste it into the address bar of Windows Explorer and it works.

4. If I try a path that actually doesn't exist, I get a slightly different message Error [ERR_MODULE_NOT_FOUND]: Cannot find module '<path here>', and it shows the absolute path to the file that wasn't found instead of the file URL I provided.

Checking different file locations:

1. I tried loading plugins that are located in a directory outside the program (the paths shown above like /home/sparky/example/plugins/...); got the results described above

2. I tried loading plugins that are located in the same directory (or subdirectory) as the main program; same result

3. I tried loading plugins that are packaged with the dependency in node_modules/@example/plugin-loader; same result (obviously this is not a useful set up but I just wanted to check it)

I'd like to put the plugin loader in a separate library instead of having the same code in every project, but it seems that dynamic import only works from the main package and not from its dependencies.

I'm hoping someone here can explain what is going on, or give me a pointer to what might make this work.

Answer 1

If the plugin loader is in another package, the plugin loader does not have the plugin in its node_modules. You are trying to import a non-existing package.

In that case, you must import a file instead of the module name or directory:

Import from Parent	Does Work?
await import("my-plugin")	NO
await import("/parent/node_modules/my-plugin")	NO
await import("/parent/node_modules/my-plugin/dist/index.js")	YES

Now, we need to find the path of the entry file of the plugin. Below is a solution using a plugin name (not path):

import parentModule from "parent-module";
import { resolve } from "import-meta-resolve";

async function loadJsPlugin(pluginName) {
  // Find the path of the parent. (The file calling this function)
  const parentModulePath = parentModule();
  
  if (parentModulePath === undefined) throw new Error("NO_PLUGIN_FOUND");
  
  // Find the entry point of the plugin resolved for the parent module.
  const pluginUrl = await resolve(pluginName, parentModulePath);
  
  const plugin = await import(pluginUrl);
  return plugin.default;
}

Usage:

await loadJsPlugin('my-plugin');

Important Note

The loadJsPlugin function must be called from the parent module directly. Otherwise the parentModule() part does not return the expected value. If you need to divide the loadJsPlugin into multiple functions, then use parentModule() in the first function called from the parent and pass its value to other functions:

async function loadJsPlugin(pluginName) {
  // Find the path of the parent. (The file calling this function)
  const parentModulePath = parentModule();
  ...
  return someMoreWork(pluginName, parentModulePath)
}


async function someMoreWork(pluginName, parentModulePath) {
  const WRONG = parentModule(); <---- This does not work, use from parameter.
  ...
}

Answer 2

I haven't found the reason why await import('file:///C:/Path/To/Script.js') says module not found when fs.readFileSync(fileURLToPath('file:///C:/Path/To/Script.js')) is able to read the script, but that gave me an idea to try a different approach.

I found the solution to my problem in another question.

In my module @example/plugin-loader (not the real name), I replaced this line:

const result = await import(moduleURL);

With this code adapted from the answers to that question:

const moduleText = fs.readFileSync(fileURLToPath(moduleURL), 'utf-8').toString();
const moduleBase64 = Buffer.from(moduleText).toString('base64');
const moduleDataURL = `data:text/javascript;base64,${moduleBase64}`;
const result = await import(moduleDataURL);

This allows my original code to work when it didn't before:

import { loadJsPlugin } from '@example/plugin-loader';

async function doSomething() {
  const plugin = await loadJsPlugin('...'); // works now
  // use plugin
}