1

I'm using fake-words module (npm install fake-words) with the following simple code:

#!/usr/bin/env node
const fake = require("fake-words");
while (true) {
  console.log(fake.sentence());
}

When I run ./genwords.js, everything works as expected.

However when I pipe into external program (on Ubuntu shell), the generation of words stops after a second.

$ ./genwords.js | cat
...
(output generation stops after a second)
$ ./genwords.js | tee
...
(stuck as well)
$ ./genwords.js | pv -l
...
4.64k 0:00:13 [0.00 /s]

Same happening when assigning a value to variable to avoid any caching (as precaution after reading this post, probably not relevant to Node.js):

while (true) {
  words = fake.sentence();
  console.log(words);
}

What I'm doing wrong?

I'm using Node v16 on Ubuntu:

$ node --version
v16.13.1
kenorb
  • 155,785
  • 88
  • 678
  • 743
  • How would assigning to a variable prevent cache? And what cache exactly? – Konrad Sep 07 '22 at 16:47
  • I did some tests and running `./genwords.js | tee test.txt | pv -l` still seems like it stopped, but words are still generated and saved to the file. It seems like `pv` would be bugged, not node. – Konrad Sep 07 '22 at 16:54
  • @KonradLinkowski Same with `./genwords.js | tee`, printing on the terminal stops, so not only `pv`. – kenorb Sep 07 '22 at 16:56
  • 1
    @KonradLinkowski I've seen some [post](https://stackoverflow.com/a/51574036/55075) about JVM cache optimizing code (infinite loop), but I doubt it applies to Node.js. So I've tested different syntax as precaution. – kenorb Sep 07 '22 at 17:01
  • Ok, so it's definitely clogging at some point. Changing `while (true)` to `setInterval` fixes the issue, but it's slower. Also, `./genwords.js | pv > test.txt` works as intended. It's either too much data or too fast or both. – Konrad Sep 07 '22 at 17:02
  • Displaying any long sentence in a loop will stop after a few seconds in node. I tried to replicate it in c++, but it runs without a problem both with `pv -l` and `tee`. – Konrad Sep 07 '22 at 17:09
  • I also tried running it on Windows and it works fine. Seems like the issue is with Ubuntu. – Konrad Sep 07 '22 at 17:12
  • 1
    @KonradLinkowski also happens on macOS. My guess is that Node.js does something special when connected to a tty. – robertklep Sep 07 '22 at 17:12
  • 1
    [This](https://nodejs.org/api/process.html#a-note-on-process-io) might be relevant: writes to pipes on POSIX systems (Linux, macOS) are asynchronous. By using a `while (true)` loop, you're blocking the event loop and the output pipe/socket may not get flushed because of that. – robertklep Sep 07 '22 at 17:28
  • 1
    I searched a little more and I didn't find a full answer, but [it's possible to know if output buffer is full] and I checked that and it's full, that's why it's not printing. [There is also a`drain` event](https://stackoverflow.com/a/35801921/5089567) which should happen when buffer is emptied. But the `drain` event never fires in a while loop. Maybe because node is single-threaded. – Konrad Sep 07 '22 at 21:30
  • Reported a bug at https://github.com/nodejs/node/issues/44577 – kenorb Sep 08 '22 at 19:43

1 Answers1

1

The behavior of console.log() in code (such as the while loop in your example) that never relinquishes control to the event loop (especially when piped to another process) in Node.js is a longstanding...uh...quirk. It's a lot harder to fix than you might think. Here's the relevant issue in the tracker: https://github.com/nodejs/node/issues/6379

Or more specifically on the whole handling-when-piped-to-another-process issue: https://github.com/nodejs/node/issues/1741

You can work around the issue by restructuring the code to relinquish control to the event loop. Here is one possibility.

#!/usr/bin/env node
const fake = require("fake-words");

function getWords() {
  console.log(fake.sentence());
  setImmediate(getWords);
}

getWords()
Trott
  • 66,479
  • 23
  • 173
  • 212