5

If I have 4, 8 or more threads & clusters on one box running this code, can there be resource contention over log files / file writers? If so, could you post an example that illustrates? So far all the testing I've done seems to indicate that write data won't get interwoven and won't get thrown away, but I am not 100% convinced

Thanks!

var errLog = fs.createWriteStream(... + '/error.log');
GLOBAL.dbLog = fs.createWriteStream(... + '/db.log');
loganfsmyth
  • 156,129
  • 30
  • 331
  • 251
ControlAltDel
  • 33,923
  • 10
  • 53
  • 80

1 Answers1

4

There is no coordinating of threads or clustered processes, so the question comes down to whether the OS will allow multiple open write streams to a file. The default C flags for CreateWriteStream are O_WRONLY || O_CREAT || O_TRUNC. This indicates a first problem - all processes will be creating a new truncated file (when probably you wanted appending).

So assuming you're specifying w+ as a flag when creating the stream...

For Linux, this related question indicates that you will be ok as long as number of bytes is less than PIPE_BUF bytes (4096 on Linux, 512 on some other Unixes).

I'm not sure what the answer is on Windows.

Community
  • 1
  • 1
Steve Campbell
  • 3,385
  • 1
  • 31
  • 43
  • How do you specifiy the w+ when creating the stream? I am calling fs.createWriteStream(path, [options]) is this a possible option? It isn't listed in the doc. Also, is there a way to change the PIPE_BUF on my systems? I'm using 3 different flavors of unix - macOS, centOS, and ubuntu – ControlAltDel Apr 24 '12 at 19:49
  • The [docs](http://nodejs.org/api/fs.html#fs_fs_createwritestream_path_options) show an example of using 'w'. You can just replace that with 'w+'. Short of editing Linux source code and recompiling, I don't believe there is a way to change PIPE_BUF (it is a compiled-in constant) – Steve Campbell Apr 25 '12 at 14:22
  • How would you distribute in Node several different files (writing) amongst several threads or processes? I can't find anything after searching around. – João Pimentel Ferreira Sep 01 '22 at 15:32
  • @JoãoPimentelFerreira did you find anything? – No_Name Oct 26 '22 at 21:08
  • 1
    @No_Name yes I did, check [here](https://stackoverflow.com/a/73585229/1243247) – João Pimentel Ferreira Oct 26 '22 at 22:10