Intermittent EMFILE, ENOTDIR, ENOENT errors during generation

Question

I'm running npx eleventy in a docker container without changes to the source and see intermittant errors.

I see these errors every now and then:

EMFILE: too many open files
ENOTDIR: not a directory
ENOENT: no such file or directory

Here's an example to reproduce the output:

$ while true ; do npx eleventy ; done
[11ty] Wrote 95 files in 1.42 seconds (14.9ms each, v2.0.1)
[11ty] Wrote 95 files in 1.71 seconds (18.0ms each, v2.0.1)
[11ty] Wrote 95 files in 1.55 seconds (16.3ms each, v2.0.1)
[11ty] Wrote 95 files in 2.01 seconds (21.2ms each, v2.0.1)
[11ty] Wrote 95 files in 1.47 seconds (15.5ms each, v2.0.1)
[11ty] Wrote 95 files in 1.45 seconds (15.3ms each, v2.0.1)
[11ty] Wrote 95 files in 1.62 seconds (17.1ms each, v2.0.1)
[11ty] Wrote 95 files in 1.73 seconds (18.2ms each, v2.0.1)
[11ty] Wrote 95 files in 1.78 seconds (18.7ms each, v2.0.1)
[11ty] Wrote 95 files in 1.52 seconds (16.0ms each, v2.0.1)
[11ty] Problem writing Eleventy templates: (more in DEBUG output)
[11ty] EMFILE: too many open files, stat './src/DIR/index.md' (via Error)
[11ty]
[11ty] Original error stack trace: Error: EMFILE: too many open files, stat './src/DIR/index.md'
[11ty] Wrote 0 files in 1.53 seconds (v2.0.1)
[11ty] Wrote 95 files in 1.56 seconds (16.4ms each, v2.0.1)
[11ty] Wrote 95 files in 1.70 seconds (17.9ms each, v2.0.1)
[11ty] Wrote 95 files in 1.86 seconds (19.6ms each, v2.0.1)
[11ty] Wrote 95 files in 1.48 seconds (15.6ms each, v2.0.1)
[11ty] Wrote 95 files in 2.06 seconds (21.7ms each, v2.0.1)
[11ty] Wrote 95 files in 1.64 seconds (17.3ms each, v2.0.1)
[11ty] Wrote 95 files in 1.58 seconds (16.6ms each, v2.0.1)
[11ty] Wrote 95 files in 1.98 seconds (20.8ms each, v2.0.1)
[11ty] Problem writing Eleventy templates: (more in DEBUG output)
[11ty] EMFILE: too many open files, stat './src/DIR/index.md' (via Error)
[11ty]
[11ty] Original error stack trace: Error: EMFILE: too many open files, stat './src/DIR/index.md'
[11ty] Wrote 0 files in 1.02 seconds (v2.0.1)
[11ty] Wrote 95 files in 1.52 seconds (16.0ms each, v2.0.1)
[11ty] Wrote 95 files in 1.60 seconds (16.8ms each, v2.0.1)
[11ty] Wrote 95 files in 1.45 seconds (15.3ms each, v2.0.1)
[11ty] Wrote 95 files in 1.53 seconds (16.1ms each, v2.0.1)
[11ty] Wrote 95 files in 1.48 seconds (15.6ms each, v2.0.1)
[11ty] Wrote 95 files in 1.57 seconds (16.5ms each, v2.0.1)
[11ty] Wrote 95 files in 1.57 seconds (16.5ms each, v2.0.1)
[11ty] Wrote 95 files in 2.01 seconds (21.2ms each, v2.0.1)
[11ty] Wrote 95 files in 1.39 seconds (14.6ms each, v2.0.1)
[11ty] Wrote 95 files in 2.00 seconds (21.1ms each, v2.0.1)
[11ty] Wrote 95 files in 1.55 seconds (16.3ms each, v2.0.1)
[11ty] Wrote 95 files in 1.52 seconds (16.0ms each, v2.0.1)
[11ty] Wrote 95 files in 1.49 seconds (15.7ms each, v2.0.1)
[11ty] Wrote 95 files in 1.56 seconds (16.4ms each, v2.0.1)
[11ty] Wrote 95 files in 2.07 seconds (21.8ms each, v2.0.1)
[11ty] Wrote 95 files in 1.62 seconds (17.1ms each, v2.0.1)
[11ty] Wrote 95 files in 1.65 seconds (17.4ms each, v2.0.1)
[11ty] Wrote 95 files in 1.76 seconds (18.5ms each, v2.0.1)
[11ty] Wrote 95 files in 1.48 seconds (15.6ms each, v2.0.1)
[11ty] Wrote 95 files in 1.86 seconds (19.6ms each, v2.0.1)
[11ty] Problem writing Eleventy templates: (more in DEBUG output)
[11ty] EMFILE: too many open files, stat './src/DIR/index.md' (via Error)
[11ty]
[11ty] Original error stack trace: Error: EMFILE: too many open files, stat './src/DIR/index.md'
[11ty] Wrote 0 files in 1.12 seconds (v2.0.1)
[11ty] Wrote 95 files in 1.53 seconds (16.1ms each, v2.0.1)
[11ty] Wrote 95 files in 1.51 seconds (15.9ms each, v2.0.1)
[11ty] Wrote 95 files in 1.49 seconds (15.7ms each, v2.0.1)
[11ty] Wrote 95 files in 1.96 seconds (20.6ms each, v2.0.1)
[11ty] Wrote 95 files in 1.59 seconds (16.7ms each, v2.0.1)
[11ty] Wrote 95 files in 1.48 seconds (15.6ms each, v2.0.1)

The content is confidential, so I can't provide additional information with the current config. Before I start to create a reproducible image, I'd like to ask if there are any known root causes for this kind of non-deterministic behavior?

Thanks in advance!

score 0 · Answer 1 · answered Jun 10 '23 at 12:04

Solved. The root cause was the default number of open files of the (macos) host:

$ ulimit -n
256

The limit can be increased per shell session by adding ulimit -n NNNNNN to your shell resource file (.zshrc, .bashrc, .profile, etc).

For a persistent fix, see Max open files per process.

Intermittent EMFILE, ENOTDIR, ENOENT errors during generation

1 Answers1