4

How can I implement a system where multiple Node.js processes write to the same file with fs.createWriteStream, such that they don't overwrite data? It looks like the default setup for fs.createWriteStream is that the file is cleared out when that method is called. My goal is to clear out the file once, and then have all other subsequent writers only append data.

Should I use fs.createWriteStream and then fs.appendFile? Or is there a way to open up a stream for each process, not just for the first process to open the file?

Alexander Mills
  • 90,741
  • 139
  • 482
  • 817
  • Have you considered using an intermediary to delegate log writing to? Such as syslog, a third party logging service, or even a dedicated node process that writes incoming log requests (e.g. over udp or unix socket or whatever) to file. – mscdex Apr 16 '16 at 03:40
  • Yeah thanks that is a good observation, actually I am probably going to have to do that. I figure it would be more performant for each child process to pipe directly to the file; however, then the file data might get corrupted and the data out of order? So I pipe stderr (which is the data stream in question) to a single parent node.js process, which then writes the data to the file. Less performant as parent proc is a bottleneck, but probably fine. – Alexander Mills Apr 16 '16 at 05:17
  • You need just file append - you should change the topic. – xmedeko Apr 14 '21 at 07:25
  • The concurrent file append is OS specific (and maybe the buffer size, too), e.g. see https://stackoverflow.com/a/57229218/254109 – xmedeko Apr 14 '21 at 07:27

2 Answers2

3

Should I use fs.createWriteStream and then fs.appendFile?

you can use either.

with fs.createWriteStream you have to change the flag like this:

fs.createWriteStream('your_file',{
  flags: 'a+', // default is 'w' (just 'a' might be enough here, i'm not sure)
})

this should create the file if it doesn't exist or open it with write access if it exists and set the pointer to end. (append mode)

How to use fs.appendFile should be clear and it does pretty much the same.

Now the problem with multiple processes accessing the same file. Obviously only one process can open the same file with write access at the same time. Therefore you need to wait for the file to be released if another process has the write access. You will probably need a library for that.

this one for example: https://www.npmjs.com/package/lockup
or this one: https://github.com/Perennials/mutex-node
you can also find alot more here: https://www.npmjs.com/browse/keyword/lock
or here: https://www.npmjs.com/browse/keyword/mutex

I have not tried any of those libraries but the one I posted and several others on the list should do exactly what you need.

x4rf41
  • 5,184
  • 2
  • 22
  • 33
0

Writing on a single file from multiple processes, ensuring data integrity, it is a fairly complex operation that you can orchestrate using File locking.

However, you have two simpler approaches:

  1. Writing on a temporary file for each process, and then concatenate the files at the end of the operations.
  2. Transmitting what you need to write to a dedicated, single process and delegate the writing execution to it. Keep in mind that sending messages among processes can be expensive.
Luca Fagioli
  • 12,722
  • 5
  • 59
  • 57