1

Is it possible to know the amount of data a chunk (buffer) carries in ReadStreaming from a file (a text file) in node js ???

I created a ReadStream and displayed the data of a text file in chunks, but is it possible to find the amount of data a buffer could have ?

var myReadStream = fs.createReadStream( __dirname + '/readme.txt', 'utf8');

myReadStream.on('data', function(chunk){

    console.log('new chunk received');
    console.log(chunk);
Neil Lunn
  • 148,042
  • 36
  • 346
  • 317
Nihara Nelligahawatta
  • 23
  • 1
  • 4
  • [Changing readstream chunksize](https://stackoverflow.com/questions/27641571/changing-readstream-chunksize)? Otherwise `chunk.length`. Not really sure why you are asking or whether you are sure why you are asking. Usually if you want to "expect" a certain size then you would have set this on creation. – Neil Lunn May 06 '18 at 01:53
  • A data chunk can be any size up to the internal buffer size. Even when you are reading lots of data, there's no guarantee that a chunk will even be a full buffer size. So, basically, you need to design your code to handle whatever length piece of data you get. You can set the buffer size as an option when you create the read stream if you want, but that just enables the possibility of larger chunks - it does not guarantee it. – jfriend00 May 06 '18 at 02:07

1 Answers1

3

Is it possible to know the amount of data a chunk (buffer) carries in ReadStreaming from a file (a text file) in node js ???

Yes, in your example, chunk.length will be the amount of data in the chunk.

You cannot control what the chunk length will be. You have to code to whatever it ends up being when the data event occurs. This is how streams work. The size of a given chunk depends upon how much data was sent, depends upon the transport (slow transports are more likely to give you smaller chunks) and the max buffer size in the stream.

You can control the max buffer size in the stream at the time you create the stream with the highWaterMark option as shown in the doc.

var myReadStream = fs.createReadStream( __dirname + '/readme.txt', 'utf8', { highWaterMark: 60000 });

But setting this to a larger value only makes it possible to get larger chunks. It does not guarantee it.

I created a ReadStream and displayed the data of a text file in chunks, but is it possible to find the amount of data a buffer could have ?

If you set the highWaterMark when you create the stream, then that will be the max size chunk you can get. If you want to see what the current highWaterMark is set to, you can look in the stream instance data at:

myReadStream._readableState.highWaterMark

That is not a documented API so it is subject to change. If you need to know, you should probably just set it to a desired value when you create the stream.

jfriend00
  • 683,504
  • 96
  • 985
  • 979