2

I've got a file of records (one per line):

Record1
Record2
Record3

I'm using the fs and the readline modules for that:

        const rs = fs.createReadStream('input.txt');
        const ws = fs.createWriteStream('output.json', { encoding: "utf8" });
        const rl = readline.createInterface({
            input: rs,
            crlfDelay: Infinity,
        });
        ws.write('[');
        for await (const line of rl) {
            ws.write(`${line},\n`);
        }
        ws.end(']');

But this loop will also toss in a trailing comma at the end of the file which is bad cos trailing commas are not allowed in JSON.

The problem is that I don't want to read the whole file into memory and I don't know the number of records in advance.

Edit:

This is a simple example, but in the real code, the record gets transformed to a json record, but it's irrelevant.

one_tit_shark
  • 89
  • 1
  • 6
  • If the file is too large for memory, is it practical to output one huge JSON file? Assuming the consumer must repeat the streaming ingestion process, why not use [JSON Lines](https://jsonlines.org/)/[ndjson](http://ndjson.org/) instead? – jsejcksn Jan 16 '22 at 08:52
  • Does this answer your question? https://stackoverflow.com/a/36749787/11134160 – Leyiang Jan 16 '22 at 08:56
  • @jsejcksn Thank you. The requirement is to use an ordinary json file. =( – one_tit_shark Jan 16 '22 at 09:11

1 Answers1

3

By creating a local variable to track whether you've already encountered the first record, you can prepend the comma to all entries after the first one, achieving the same effect:

Note that, because whitespace is insignificant in JSON, you will decrease the size of the resulting file by omitting the newlines

ws.write('[');

let comma = false;

for await (const line of rl) {
  if (!line.trim()) continue;
  if (comma) ws.write(',');
  ws.write(line);
  comma = true;
}

ws.end(']');
jsejcksn
  • 27,667
  • 4
  • 38
  • 62