2

I found this question more or less asked elsewhere, in less-than-clear language, but the answer was

"I doubt you have any reason to."

which doesn't really seem like a valid answer, considering if it's fundamentally possible (which it very well may not be), it would allow us to generate large (100GB+) files in parallel threads much faster. Now, if your data structure is something that could really be written in such a parallel way, you can probably just write 16 different files on your 16 threads and call it a day. That's probably what I'll be doing.

But just out of curiosity, let's say we really, really want all that data stored in one file. Can multiple

fs.createWriteStream(filePath)

be called from separate threads and successfully write to locations within the same file in parallel?

I couldn't find any info on this in the documentation, although I suspect the answer is simply "no."

J.Todd
  • 707
  • 1
  • 12
  • 34

2 Answers2

5

You cannot safely append to the same file from two separate threads at the same time. Each thread will have its own file handle and its own write position and there are concurrency issues with where each file is writing the data to.

Can multiple fs.createWriteStream(filePath)be called from separate threads and successfully write to locations within the same file in parallel?

No. You cannot do that safely. Each thread will have concurrency issues keeping track of where the write position should be set to. You would need some concurrency management to sequence the writes and keep track of where each one was being written to.

You could have just one thread doing all the writing and have each thread that is generating data send its data to the one thread that is doing the writing as a means of safely sequencing the writes.

jfriend00
  • 683,504
  • 96
  • 985
  • 979
  • Might also be worth mentioning that even if the OP devised a safe way for multiple threads to _concurrently_ write the same file (e.g., by memory mapping it), the writes could never happen in _parallel_, because any given disk drive (or other mass storage device) has only a single interface through which the data can be written. – Solomon Slow Sep 21 '20 at 10:28
0

If you really need this, I would suggest using a SQLite database, it's much safer (and faster) =)

edu.nicastro
  • 43
  • 5