0

This may already be answered somewhere on the site, but if it is I couldn't find it. I also couldn't find an exact answer to my question (or at least couldn't make sense of how to implement a solution) based on any of the official Node.js documentation.

Question: Is it possible to customize the length (in bytes) of each disk write that occurs while piping the input of a readable stream into a file?

I will be uploading large files (~50Gb) and it's possible that there could be many clients doing so at the same time. In order to accomplish this I'll be slicing files at the client side and then uploading a chunk at a time. Ideally I want physical writes to disk on the server side to occur in 1Mb portions - but is this possible? And if it is possible then how can it be implemented?

Bergi
  • 630,263
  • 148
  • 957
  • 1,375
pooley1994
  • 723
  • 4
  • 16
  • "*Ideally I want physical writes to disk on the server side to occur in 1Mb portions*" - why? – Bergi Aug 13 '14 at 01:15
  • I don't have my heart set on the exact 1Mb value, but what I want to avoid is thousands of tiny little writes overwhelming the server. Instead I'd rather fewer writes of a larger size. One of my co-workers mentioned 1Mb would be a good "server friendly" sized write. – pooley1994 Aug 13 '14 at 01:16

1 Answers1

0

You will probably use a WriteStream. While it is not documented in the fs api, any Writable does take a highWaterMark option for when to flush its buffer. See also the details on buffering.

So it's just

var writeToDisk = fs.createWriteStream(path, {highWaterMark: 1024*1024});
req.pipe(writeToDisk);

Disclaimer: I would not believe in cargo-cult "server-friendly" chunk sizes. I'd go with the default (which is 16kb), and when performance becomes a problem test other sizes to find the optimal value for the current setup.

Community
  • 1
  • 1
Bergi
  • 630,263
  • 148
  • 957
  • 1,375
  • See, I did read that particular portion detailing the [highWaterMark](http://nodejs.org/api/stream.html#stream_new_stream_writable_options) but that's not how I interpreted it. Just because write would return false if the buffer grew to that size, does that necessarily mean that writes are by default that size when piping from a readable stream? I took it to mean instead that if the buffer grew to that size, then it wouldn't accept anymore data until it had been flushed to disk (not necessarily that it would only flush to disk once it reached that size...) – pooley1994 Aug 13 '14 at 02:08
  • Hm, that might be true, but why wouldn't you want your server to flush faster *when he is capable* to do it? – Bergi Aug 13 '14 at 13:18
  • 1
    Frankly that's a good question. I think I may move forward with not worrying too much about getting bigger writes and then revisit if it does present itself as an issue. – pooley1994 Aug 13 '14 at 13:59