5

How can I cause the data to be written to the file as WriteStream.write() is called?

Edit: As it turns out, this works when I use the REPL and call the function. However, this doesn't work in my program:

import * as FS from "fs";
import { LetterGroup } from "./LetterGroup";
import { Dictionary } from "./Dictionary";
import { Word } from "./Word";
import * as OS from "os";

const groups: Array<LetterGroup> = LetterGroup.letterGroupsFromString(FS.readFileSync("./letterGroups.txt").toString());
const dictionary = Dictionary.create(FS.readFileSync("./dictionary.txt").toString());

const inputStr: string = FS.readFileSync("./input.txt").toString();
const inputWords = new Array<Word>();
const fileStream = FS.createWriteStream("./output.txt");

for (const line of inputStr.trim().split(OS.EOL))
{
    inputWords.push(new Word(line));
}

function permute(index: number)
{
    index = Math.floor(index);

    if (!(index >= 0 && index < inputWords.length))
    {
        return;
    }

    const word: Word = inputWords[index];

    let outputString: string = "";

    outputString += `Valid permutations of '${word.wordString}':` + OS.EOL;

    console.log(`Testing '${ word.wordString }'...`);

    for (const permutation of word.generatePermutations(groups, dictionary, true))
    {
        outputString += permutation.wordString + OS.EOL;
    }

    outputString += OS.EOL;
    console.log();

    if (!fileStream.write(outputString))
    {
        console.log("Wrote to file too fast! Will resume writing once the system catches up...");
        fileStream.once("drain", () => permute(index));
        return;
    }

    permute(index + 1);
}

permute(0);

It should be noted that word.generatePermutations is an extremely intensive function and often can take several minutes (sometimes over an hour) to return. My issue is that the data it returns should be written to the output file immediately so that the entire program doesn't need to finish running for the results to be read. Hopefully someone can make sense of all this.

To clarify, data written to a WriteStream in the REPL is immediately written to the file, while no data is written to the file in my program until the entire program finishes running.

John Leuenhagen
  • 576
  • 7
  • 23
  • 1
    Possible duplicate of [Writing files in Node.js](https://stackoverflow.com/questions/2496710/writing-files-in-node-js) – CDspace Jan 23 '18 at 22:23
  • 1
    @CDspace I actually decided to use file descriptors and `fs.write()` to solve to problem, but that doesn't change the fact that my original question hasn't been answered. I specifically want to know if there's a way to have a WriteStream sync its data to disk as data is written to it. This seems to work in the REPL, but not in my program. I'll edit the original question. – John Leuenhagen Jan 24 '18 at 13:24
  • I have the same problem here. Any solution found? – yue you Oct 26 '18 at 12:25

1 Answers1

2

Try these packages:

https://www.npmjs.com/package/flushwritable

https://www.npmjs.com/package/flush-write-stream

And I think you question is duplicate of How to flush Node.js file writeStream properly?

Aleksei Semidotskii
  • 1,385
  • 1
  • 10
  • 18
  • I'll check those packages, but the thread you suggested is different from mine as he was trying to flush a file after closing the stream, while I want to flush to the file while the stream is active. – John Leuenhagen Jan 23 '18 at 20:51