0

I cannot append data from multiple Read Streams using createWriteStreams() in Node. It just creates a new file after each iteration.

chunkNo = 4;

for (i = 0; i < chunkNo; i++) { 
    const response = await this.downloadChunk(downloadUrl, i); // response.data is a ReadStream
    await new Promise<void>((resolve, reject) => {
          const stream = response.data.pipe(fs.createWriteStream('C:\\big_file.mkv'), { flags: 'a'});
          stream.on('finish', () => {
            resolve();
          });
        });
}

What am I missing? Even though the append flag is there. Thank you in advance

Mike Me
  • 2,192
  • 3
  • 20
  • 32

1 Answers1

1

To better consolidate your intuition about what the a (append) flag purpose is supposed to indicate, first have a look at this answer: Difference between modes a, a+, w, w+, and r+ in built-in open function?

It details the different modes in which a file can be open. As many other languages, JavaScript reuses the same principle that in C.

From this man page, the relevant section in our case is:

 ``a''   Open for writing.  The file is created if it does not exist.  The
         stream is positioned at the end of the file.  Subsequent writes
         to the file will always end up at the then current end of file,
         irrespective of any intervening fseek(3) or similar.

So in your code example, a new file would indeed be created at each iteration of the loop. Therefore, what you can do is move the fs.createWriteStream('C:\\big_file.mkv', { flags: 'a'}) outside of the for loop, assigned a name to this readable stream and use it the pipe(), something along the way of :

chunkNo = 4;
const dest = fs.createWriteStream('C:\\big_file.mkv', { flags: 'a'});

for (i = 0; i < chunkNo; i++) { 
    const response = await this.downloadChunk(downloadUrl, i); // response.data is a ReadStream
    await new Promise<void>((resolve, reject) => {
          const stream = response.data.pipe(dest);
          stream.on('finish', () => {
            resolve();
          });
        });
}
Delapouite
  • 9,469
  • 5
  • 36
  • 41
  • Thank you @Delapouite for the suggestion, however I tried this and I receive an error when trying to pipe in the second iteration: "[ERR_STREAM_WRITE_AFTER_END]: write after end". It seems that it closes the file. I also tried adding {autoClose: false} in the createWriteStream options and close it after the loop but still the same. – Mike Me Apr 18 '21 at 10:47