How to integrate own library with WebPack which needs to span WebWorkers and AudioWorklets

Question

Goal I am the author of a JavaScript library which can be consumed through AMD or ESM within various runtime environments (Browser, Node.js, Dev Servers). My library needs to spawn WebWorkers and AudioWorklets using the file it is contained. The library detects in which context it is running and sets up the required stuff for the execution context.

This works fine as long users (user=integrator of my library) do not bring bundlers like WebPack into the game. To spawn the WebWorker and AudioWorklet I need the URL to file in which my library is contained in and I need to ensure the global initialization routines of my library are called.

I would prefer to do as much as possible of the heavy lifting within my library, and not require users to do a very specialized custom setup only for using my library. Offloading this work to them typically backfires instantly and people open issues asking for help on integrating my library into their project.

Problem 1: I am advising my users to ensure my library is put into an own chunk. Users might setup the chunks based on their own setup as long the other libs don't cause any troubles or side effects in the workers. Especially modern web frameworks like React, Angular and Vue.js are typical problem children here, but also people tried to bundle my library with jQuery and Bootstrap. All these libraries cause runtime errors when included in Workers/Worklets.

The chunking is usually done with some WebPack config like:

config.optimization.splitChunks.cacheGroups.alphatab = {
  chunks: 'all',
  name: 'chunk-mylib',
  priority: config.optimization.splitChunks.cacheGroups.defaultVendors.priority + 10,
  test: /.*node_modules.*mylib.*/
};

The big question mylib now has: What is the absolute URL of the generated chunk-mylib.js as this is now the quasi-entrypoint to my library now with bundling and code splitting in place:

• document.currentScript points usually to some entry point like an app.js and not the chunks.
• __webpack_public_path__ is pointing to whatever the user sets it to in the webpack config.
• __webpack_get_script_filename__ could be used if the chunk name would be known but I haven't found a way to get the name of the chunk my library is contained in.
• import.meta.url is pointing to some absolute file:// url of the original .mjs of my library.
• new URL(import.meta.url, import.meta.url) causes WebPack to generate an additional .mjs file with some hash. This additional file is not desired and also the generated .mjs contains some additional code breaking its usage in browsers.

I was already thinking to maybe create a custom WebPack plugin which can resolve the chunk my library is contained in so I can use it during runtime. I would prefer to use as much built-in features as possible.

Problem 2: Assuming that problem 1 is solved I could now spawn a new WebWorker and AudioWorklet with the right file. But as my library is wrapped into a WebPack module my initialization code will not be executed. My library only lives in a "chunk" and is not an entry and I wouldn't know that this splitting would allow mylib to run some code after the chunk was loaded by the browser.

Here I am rather clueless. Maybe chunks are not the right way of splitting for this purpose. Maybe some other setup is needed I am not yet aware of that its possible?

Maybe also this could be done best with a custom WebPack plugin.

Visual Representation of the problem: With the proposed chunking rule we get an output as shown in the blocks. Problem 1 is the red part (how to get this URL) and Problem 2 is the orange part (how to ensure my startup logic is called when the background worker/worklet starts)

Actual Project I want to share my actual project for better understanding of my use case. I am talking about my project alphaTab, a music notation rendering and playback library. On the Browser UI thread (app.js) people integrate the component into the UI and they get an API object to interact with the component. One WebWorker does the layouting and rendering of the music sheet, a second one synthesizes the audio samples for playback and the AudioWorklet sends the buffered samples to the audio context for playback.

Answer 1

I think the worker code should be handled as an assets instead of a source code. Maybe you could add a simple CLI to generate a ".alphaTab" folder on the root of the project and add instructions for your user to copy that to the "dist"or "public folder". Even if come up with a Webpack specific solution, you would have to work your way around other bundlers/setups (Vite, rollup, CRA, etc).

EDIT: You would also need to add an optional parameter to the initialization for passing the script path. Not fully automated, but simpler that having to setup complex bundler configs

Answer 2

Disabling import.meta

Regarding import.meta.url, this link might help. It looks like you'd disable it in your webpack config by setting module.parser.javascript.importMeta to false.

Reworking Overall Architecture

For the rest, it sounds like a bit of a mess. You probably shouldn't be trying to import the same exact chunk code into your workers/worklets, since this is highly dependent on how webpack generates and consumes chunks. Even if you manage to get it to work today, it might break in the future if the webpack team changes how they internally represent chunks.

Also from a user's perspective, they just want to import the library and have it just work without fiddling with all of the different build steps.

Instead, a cleaner way would to be to generate separate files for the main library, the AudioWorklet, and the Web Worker. And since you already designed the worklet and web worker to use your library, you can just use the prebuilt, non-module library for them, and have a separate file for the entry point for webpack/other bundlers.

The most straightforward way would be to have users add your original non-module js library in with the bundle that they build, and have the es module load Web Workers and Audio Worklets using that non-module library's url.

Of course, from a user's perspective, it'd be easier if they didn't have to copy over additional files and put them in the right directory (or configure a scripts directory). The straightforward way would be to load the web worker or worklet from a CDN (like https://unpkg.com/@coderline/alphatab@1.2.2/dist/alphaTab.js), but there are restrictions from loading web workers cross origin, so you'd have to use a work around like fetching it and then loading it from a blob url (like that found here). This unfortunately makes initializing the Worker/Worklet asynchronous.

Bundling Worker code

If this isn't an option, you can bundle a library, Web Worker/Worklet code into one file by stringifying the Worker/Worklet code and loading it via a blob or data url. In your particular use case, it's a little painful from an efficiency standpoint considering how much code will be duplicated in the bundled output.

For this approach, you'd have multiple build steps:

1. Build the library that's used by your Web Worker and/or Audio Worklet.
2. Build the single library by stringifying the previous libraries/library.

This is all complicated by there being only one entry file for the library, web worker, and audio worklet. In the long term, you'd probably benefit by rewriting entry points for these different targets, but for now, we could reuse the current workflow and change the build steps by using different plugins. For the first build, we'll make a plugin that returns a dummy string when it tries to import the worker library, for the second, we'll have it return the stringified contents of that library. I'll use rollup, since that's what your project uses. The code below is mostly for illustrative purposes (which saves the worker library as dist/worker-library.js); I haven't actually tested it.

First plugin:

var firstBuildPlugin = {
  load(id) {
    if (id.includes('worker-library.js')) {
      return 'export default "";';
    }
    return null;
  }
}

Second plugin:

var secondBuildPlugin = {
  transform(code, id) {
    if (id.includes('worker-library.js')) {
      return {
        code: 'export default ' + JSON.stringify(code) + ';',
        map: { mappings: '' }
      };
    }
    return null;
  }
}

Using these plugins, we can import the web worker/audio worklet library via import rawCode from './path/to/worker-library.js';. For your case, since you'd be reusing the same library, you may want to create a new file with an export, so the to prevent multiple bundling of the same code:

libraryObjectURL.js:

import rawCode from '../dist/worker-library.js'; // may need to tweak  the path here
export default URL.createObjectURL(
  new Blob([rawCode], { type: 'application/javascript' })
);

And to actually use it:

import libraryObjectURL from './libraryObjectURL.js'; // may need to tweak the path here
//...
var worker = new Worker(libraryObjectURL);

To then actually build it, your rollup.config.js would look something like:

module.exports = [
    {
        input: `dist/lib/alphatab.js`,
        output: {
            file: `dist/worker-library.js`,
            format: 'iife', // or maybe umd
            //...
            plugins: [
                firstBuildPlugin,
                //...
            ]
        }
    },
    {
        input: `dist/lib/alphatab.js`,
        output: {
            file: `dist/complete-library.mjs`,
            format: 'es',
            //...
            plugins: [
                secondBuildPlugin,
               //...
            ]
        }
    },
    // ...     

Preserving old code

Finally, for your other builds, you may still want to preserve the old paths. You can use @rollup/plugin-replace for this, by using a placeholder that will be replaced in the build process.

In your files, you could replace:

var worker = new Worker(libraryObjectURL);

with:

var worker = new Worker(__workerLibraryURL__);

and in the build process use:

  // ...
  // for the first build:
  plugins: [
    firstBuildPlugin,
    replace({ __workerLibraryURL__: 'libraryObjectURL')
    // ...
  ],
  // ...
  // for the second build:
  plugins: [
    secondBuildPlugin,
    replace({ __workerLibraryURL__: 'libraryObjectURL')
    // ...
  ],
  // ...
  // for all other builds:
  plugins: [
    firstBuildPlugin,
    replace({ __workerLibraryURL__: 'new URL(import.meta.url)') // or whatever the old code was
    // ...
  ],

You may need to use another replacement for your AudioWorklet url if it's different. In cases where the worker-library file isn't used, the imported libraryObjectURL will be tree shook out.

Future work:

You may want to look into having multiple outputs for your different targets: web worker, audio worklet, and library code. They really aren't supposed load the same exact file. This would negate the need for the first plugin (that ignores certain files), and it might make things more manageable and efficient.

More Reading:

• Loading files as raw strings (you can see/use TrySound's plugin here; it's a simple plugin)
• Loading strings as blob or data URLs (see https://stackoverflow.com/a/10372280/)

Answer 3

I found a way to solve the described problem but there are still some open pain points because the WebPack devs are rather trying to avoid vendor specific expressions and prefer to rely on "recognizable syntax constructs" to rewrite the code as they see fit.

The solution does not work in a fully local environment, but it works together with NPM:

I am launching my worker now with /* webpackChunkName: "alphatab.worker" */ new Worker(new URL('@coderline/alphatab', import.meta.url))) where @coderline/alphatab is the name of the library installed through NPM. This syntax construct is detected by WebPack and will trigger generation of a new special JS file containing some WebPack bootstrapper/entry-point which loads the library for startup. So effectively it looks after compilation like this:

For this to work, users should configure the WebPack to place the library in an own chunk. Otherwise it can happen that the library is maybe inlined into the webpack generated worker file instead of also loaded from a common chunk. It would work also without a common chunk, but it would defy the benefits of even using webpack because it duplicates the code of the library to spawn it as worker (double loading time and double disk usage).

Unfortunately this currently only works for Web Workers for now because WebPack has no support for Audio Worklet at this point.

Also there are some warnings due to cyclic dependencies produced by WebPack because there seem to be a a cycle between chunk-alphatab.js and alphatab.worker.js. In this setup it should not be a problem.

In my case there is no difference between the UI thread code and the one running in the worker. If users decide to render to an HTML5 canvas through a setting, rendering happens in the UI thread, and if SVG rendering is used it is off-loaded to a worker. The whole layout and rendering pipeline is the same on both sides.