0

I'm doing a bulk indexing in AWS OpenSearch within my node application and it's failing because the chunk size are too big. Below is the error

Request size exceeded 104857600 bytes

So I need to calculate the actual chunk size in bytes. So I tried with below 2 ways and not sure which one is correct.

// with Buffer
const str = 'tree'
const obj = [{a:'sas'},{a:'e'}]
console.log(Buffer.from(str).length); ---> 4
console.log(Buffer.from(obj).length); ---> 2

// with object-sizeof
var sizeof = require('object-sizeof')
const str = 'tree'
const obj = [{a:'sas'},{a:'e'}]
console.log(sizeof(str)); ---> 8
console.log(sizeof(obj)); ---> 12

It seems 'Buffer' takes 1 Byte for a char while 'object-sizeof' takes 2 bytes for a char. So what is correct here? And which is the correct way to check the byte size of actual object array? Thanks in advance

ahkam
  • 607
  • 7
  • 24

2 Answers2

2

For any type of value:

const { serialize } = require('v8')

// i.e
const variable = true 

console.log(serialize(variable).byteLength) // 3

3 = 0x03 0x01 0x00

Header byte indicating the type of the value (which is 0x03 for a boolean), followed by a byte indicating the value of the boolean (which is 0x01 for true and 0x00 for false)

Using: Buffer.byteLength(string, 'utf8'), would force one to use Buffer.byteLength(string.toString(), 'utf8'), which would be 4 for true and 5 for false, and for objects, would always be 15 due to [Object, Object], one could also do Buffer.byteLength(JSON.stringify(object), 'utf8'), which would also return the incorrect value of used memory.

Lawrence Cherone
  • 46,049
  • 7
  • 62
  • 106
0

According to answers on How many bytes in a JavaScript string? you can use Buffer.byteLength(string, 'utf8') to get the byte size of a string.

Anyway, the size of a char in javascript is 2 bytes, so i guess it's the same with typescript/node.js : https://developer.mozilla.org/en-US/docs/Web/API/DOMString/Binary

liguepk
  • 191
  • 6