Changing readstream chunksize

Question

I'm trying to create a socket based filetransfer system and because of that I came up with some trouble. I want to read files in chunks using node's fs.createReadStream() but I was unable to change the chunk size from 665536. I couldn't find anything about this.

Could you help with a related question: In this situation (socket-based filesharing), what's an ideal chunk size? And in general (aside from sockets), what's an ideal chunk size? — Morris, Sep 13 '20 at 20:15

score 72 · Accepted Answer · answered Dec 24 '14 at 19:50

72

According to the ReadStream code, you should be able to increase the highWaterMark by explicitly setting it in the ReadStream options:

var rs = fs.createReadStream('/foo/bar', { highWaterMark: 128 * 1024 });

answered Dec 24 '14 at 19:50

mscdex

104,356
15
192
153

5

@mscdex This was a super helpful answer, how on earth would you know that was there? :-) – stackunderflow Aug 22 '16 at 10:20
@KarlBateman I was already familiar with the code base, but the docs (at least currently) hint at it. – mscdex Aug 22 '16 at 10:26
@mscdex Well I salute you, that saved me a lot of time. Thank you! – stackunderflow Aug 22 '16 at 10:37
3

'bufferSize' would be the better name for 'highWaterMark' in my opinion – Combine Sep 25 '17 at 11:38
1

here's the param in docs, if anyone's interested https://nodejs.org/api/stream.html#stream_constructor_new_stream_writable_options – spencercooly May 07 '18 at 20:12
@mscdex What is an ideal (safe and fast) size? – Morris Sep 13 '20 at 20:09

score 1 · Answer 2 · answered Nov 02 '20 at 10:38

This is how you are going to change the change the size of the chunk with highWaterMark : 8 , you can set it to 8,16,24 all are in bytes, be sure if you make chunk size to much high then you are by passing the benefit of using readerstream1. Because benefit of readerstream or stream is that they work efficiently while consuming less memory.

let readerstream1 = fs.createReadStream('shamoon.txt', { highWaterMark: 8 });

Changing readstream chunksize

2 Answers2

Linked