Why does webpack --watch invoke my custom loader on unrelated files?

Question

I have a simple custom Webpack loader which generates TypeScript code from a .txt file:

txt-loader.js

module.exports = function TxtLoader(txt) {
  console.log(`TxtLoader invoked on ${this.resourcePath} with content ${JSON.stringify(txt)}`)
  if (txt.indexOf('Hello') < 0) {
    throw new Error(`No "Hello" found`)
  }
  return `export const TEXT: string = ${JSON.stringify(txt)}`
}

In real life, I'm of doing some parsing on the input; in this example, let's assume that a file must contain the text Hello to be valid.

This loader lets me import the text file like this:

index.ts

import { TEXT } from './hello.txt'

console.log(TEXT)

It all works fine, except for one thing: webpack watch (and its cousin webpack serve). The first compilation is fine:

$ /tmp/webpack-loader-repro/node_modules/.bin/webpack watch
TxtLoader invoked on /tmp/webpack-loader-repro/hello.txt with content "Hello world!\n"
asset main.js 250 bytes [compared for emit] [minimized] (name: main)
./index.ts 114 bytes [built] [code generated]
./hello.txt 97 bytes [built] [code generated]
webpack 5.64.3 compiled successfully in 3952 ms

But then I change the hello.txt file:

$ touch hello.txt

And suddenly weird stuff happens:

TxtLoader invoked on /tmp/webpack-loader-repro/index.ts with content "import { TEXT } from './hello.txt'\n\nconsole.log(TEXT)\n"
TxtLoader invoked on /tmp/webpack-loader-repro/custom.d.ts with content "declare module '*.txt'\n"
[webpack-cli] Error: The loaded module contains errors
    at /tmp/webpack-loader-repro/node_modules/webpack/lib/dependencies/LoaderPlugin.js:108:11
    at /tmp/webpack-loader-repro/node_modules/webpack/lib/Compilation.js:1930:5
    at /tmp/webpack-loader-repro/node_modules/webpack/lib/util/AsyncQueue.js:352:5
    at Hook.eval [as callAsync] (eval at create (/tmp/webpack-loader-repro/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at AsyncQueue._handleResult (/tmp/webpack-loader-repro/node_modules/webpack/lib/util/AsyncQueue.js:322:21)
    at /tmp/webpack-loader-repro/node_modules/webpack/lib/util/AsyncQueue.js:305:11
    at /tmp/webpack-loader-repro/node_modules/webpack/lib/Compilation.js:1392:15
    at /tmp/webpack-loader-repro/node_modules/webpack/lib/HookWebpackError.js:68:3
    at Hook.eval [as callAsync] (eval at create (/tmp/webpack-loader-repro/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at Cache.store (/tmp/webpack-loader-repro/node_modules/webpack/lib/Cache.js:107:20)
error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

It seems that Webpack decided to throw more files at my loader than specified in the configuration.

If I remove the exception throwing in the loader and return some arbitrary valid TypeScript code, the generated main.js looks exactly the same. So it seems that these extra operations are entirely redundant. But I don't believe that the right solution is to make my loader swallow those exceptions.

The loader is configured like this:

webpack.config.js

const path = require('path')

module.exports = {
  mode: 'production',
  entry: './index.ts',
  module: {
    rules: [
      {
        test: /\.ts$/,
        use: 'ts-loader',
      },
      {
        test: /\.txt$/,
        use: [
          {
            loader: 'ts-loader',
            // Tell TypeScript that the input should be parsed as TypeScript,
            // not JavaScript: <https://stackoverflow.com/a/47343106/14637>
            options: { appendTsSuffixTo: [/\.txt$/] },
          },
          path.resolve('txt-loader.js'),
        ],
      },
    ],
  },
}

Finally, these are the necessary bits to put it all together:

custom.d.ts

declare module '*.txt'

tsconfig.json

{}

package.json

{
  "name": "webpack-loader-repro",
  "license": "MIT",
  "private": true,
  "devDependencies": {
    "ts-loader": "9.2.6",
    "typescript": "4.5.2",
    "webpack": "5.64.3",
    "webpack-cli": "4.9.1"
  },
  "dependencies": {}
}

For those who want to try this at home, clone this minimal repro project.

Is this a bug in Webpack? In ts-loader? In my configuration?

Answer 1

1. The Problem

The main problem is that ts-loader will load additional files and manually call your loader on them.

In your current webpack configuration you will end up with 2 independent ts-loader instances:

• One for .ts files
• And one for .txt files

1.1. first compilation

During the initial compilation the following will happen:

• index.ts will be handled by the first ts-loader instance, which will try to compile it.
• The first ts-loader doesn't know how to load a .txt file, so it looks around for some module declarations and finds custom.d.ts and loads it.
• Now that the first ts-loader knows how to deal with .txt files, it will register index.ts and custom.d.ts as dependent on hello.txt (addDependency call here)
• After that the first ts-loader instance will ask webpack to please compile hello.txt.
• hello.txt will be loaded by the second ts-loader instance, through your custom loader (like one would expect)

2.1. second compilation

Once you touch (or modify) hello.txt, webpack will dutifully notify all watchers that hello.txt has changed. But because index.ts & custom.d.ts are dependent on hello.txt, all watchers will be notified as well that those two have changes.

• The first ts-loader will get all 3 change events, ignore the hello.txt one since it didn't compile that one and do nothing for the index.ts & custom.d.ts events since it sees that there are no changes.

• The second ts-loader will also get all 3 change events, it'll ignore the hello.txt change if you just touched it or recompile it in case you edited it. After that it sees the custom.d.ts change, realizes that it hasn't yet compiled that one and will try to compile it as well, while invoking all loaders specified after it. The same thing happens with the index.ts change.

• The reason why the second ts-loader even tries to load those files are the following:

  • For index.ts: Your .tsconfig doesn't specify include or exclude or files, so ts-loader will use the default of ["**"] for include, i.e. everything it can find. So once it gets the change notification for index.ts it'll try to load it.
    • This also explains why you don't get it with onlyCompileBundledFiles: true - because in that case ts-loader realizes that it should ignore that file.
  • For custom.d.ts it's mostly the same, but they will still be included even with onlyCompileBundledFiles: true:

    The default behavior of ts-loader is to act as a drop-in replacement for the tsc command, so it respects the include, files, and exclude options in your tsconfig.json, loading any files specified by those options. The onlyCompileBundledFiles option modifies this behavior, loading only those files that are actually bundled by webpack, as well as any .d.ts files included by the tsconfig.json settings. .d.ts files are still included because they may be needed for compilation without being explicitly imported, and therefore not picked up by webpack.

1.3. any compilation after that

If you modify your txt-loader.js to not throw but rather return the contents unchanged, i.e.:

if (txt.indexOf('Hello') < 0) {
    return txt;
}

We can see what happens on the third, fourth, etc... compilation.

Since both index.ts & custom.d.ts are now in the caches of both ts-loaders, your custom loader will only be called if there is an actual change in any of those files.

---

2. Similar issues

You aren't the only one that ran into this "feature", there's even an open github issue for it:

• custom loader in chain with ts-loader consumes ts files
• ts-loader breaks webpack serve due to loaderContext reuse
• Proposal to make ts-loader a better webpack citizen

---

3. Potential solutions

There are a few ways you can avoid this problem:

3.1. make the .txt ts-loader transpile-only

In transpileOnly: true-mode ts-loader will ignore all other files and only handle those that webpack explicitly asked to compile.

So this would work:

/* ... */
    rules: [
      {
        test: /\.ts$/,
        use: 'ts-loader',
      },
      {
        test: /\.txt$/,
        use: [
          {
            loader: 'ts-loader',
            options: { appendTsSuffixTo: [/\.txt$/], transpileOnly: true },
          },
          path.resolve('txt-loader.js'),
        ],
      },
    ],
/* ... */

You'll loose type-checking for your .txt files though with this approach.

3.2. make sure there's only one ts-loader instance

As long as you specify exactly the same options to each loader, ts-loader will reuse the loader instance.

That way you have a shared cache for *.ts files and *.txt files, so ts-loader doesn't try to pass *.ts files through your *.txt webpack rule.

So the following definition would work as well:

/* ... */
    rules: [
      {
        test: /\.ts$/,
        use: [
          {
            loader: 'ts-loader',
            options: { appendTsSuffixTo: [/\.txt$/] },
          }
        ],
      },
      {
        test: /\.txt$/,
        use: [
          {
            loader: 'ts-loader',
            options: { appendTsSuffixTo: [/\.txt$/] },
          },
          path.resolve('txt-loader.js'),
        ],
      },
    ],
/* ... */

3.2.1 using ts-loader's instance option

ts-loader has a (rather hidden) instance option.

Normally this would be used to segregate two ts-loader instances which have the same options - but it can also be used to forcefully merge two ts-loader instances.

So this would work as well:

/* ... */
    rules: [
      {
        test: /\.ts$/,
        use: [
          {
            loader: 'ts-loader',
            options: { appendTsSuffixTo: [/\.txt$/], instance: "foobar" },
          }
        ],
      },
      {
        test: /\.txt$/,
        use: [
          {
            loader: 'ts-loader',
            options: { instance: "foobar", /* OTHER OPTIONS SILENTLY IGNORED */ },
          },
          path.resolve('txt-loader.js'),
        ],
      },
    ],
/* ... */

You need to be careful with this one though, since the first loader that gets instanced by webpack gets to decide the options. The options you passed to all other ts-loader's with the same instance option get silently ignored.

3.3 Make your loader ignore *.ts files

The simplest option would be to just change your txt-loader.js to not modify *.ts files in case it gets called with one. It's not a clean solution but it works nonetheless :D

txt-loader.js:

module.exports = function TxtLoader(txt) {
  // ignore .ts files
  if(this.resourcePath.endsWith('.ts'))
    return txt;

  // handle .txt files:
  return `export const TEXT: string = ${JSON.stringify(txt)}`
}

Answer 2

In your minimal repro, I found that commenting out these lines removed the problem:

...
{
  test: /\.txt$/,
  use: [
    // remove ts-loader from this pipeline, and you don't get the unexpected watch behavior
    path.resolve('txt-loader.js'),
  ],
},
...

I think what is happening is that when you chain the ts-loader in your use array for /\.txt$/ pipeline, it is setting watches on what it thinks is the entire typescript project, and then re-invoking the pipeline (including your custom txt-loader) whenever anything changes. Normally this is a good thing, because it will re-compile your project if, for example a .d.ts file changes that is only included implicitly through tsconfig.json, not through an explicit webpack-processed import statement.

At least in the simple repro you provided, the bundle seems to generate and run without ts-loader in the /\.txt$/ pipeline at all, which might be enough to solve your problem.

But on the off chance that it's necessary for some reason in your real-world case to include ts-loader in this pipeline, you should be able to tell ts-loader to only look at/watch the explicitly bundled files by using the onlyCompileBundledFiles option (see docs):

...
{
  test: /\.txt$/,
  use: [
    { 
      loader: 'ts-loader',
      options: { appendTsSuffixTo: [/\.txt$/], onlyCompileBundledFiles: true },
    }
    path.resolve('txt-loader.js'),
  ],
},
...