Convert a large ArrayBuffer into hexadecimal string representation in JavaScript

Question

I have an ArrayBuffer with content that needs to be converted to a hex-string, that should be sent in a JSON message. The data can be up to 2 GB. For this I would like to have a string that works as a dynamic array, so I efficiently could push_back individual characters. In C++, we have std::string (or std::vector), I think Java has a StringBuilder. What is the way of doing it in JavaScript?

I tried += but that starts to take minutes after reaching 128 MB. I guess this is because it results in an O(n^2) algorithm.

Note: I also need to do the inverse operation.

Answer 1

Well, why not use a temporary Uint8Array, like so:

function to_hex_string(bytes)
{
    let tmp = new Uint8Array(2*bytes.length);
    for(let k = 0; k != bytes.length; ++k)
    {
        let value = bytes[k];
        let msb = value >> 4;
        let lsb = value & 0xf;
        tmp[2*k] = msb < 10 ? msb + 48 : (msb - 10) + 65;
        tmp[2*k + 1] = lsb < 10 ? lsb + 48 : (lsb - 10) + 65;
    }

    return new TextDecoder().decode(tmp);
}

This outperforms joining an array. Though it fails when trying to convert the array to a string when trying 512 MiB. I guess there is some hardcoded limit. Maybe a JS process may not use more than 4 GiB because of a 32-bit VM.

Answer 2

To efficiently convert a large ArrayBuffer into a hexadecimal string representation in JavaScript , you can use the Uint8Array view to access the individual bytes of the buffer and then build the hex string dynamically. Instead of using string concatenation (+=), which can be slow for large strings due to JavaScript's immutable string nature, you can use an array to efficiently push individual characters and then join them into a string.

function arrayBufferToHexString(buffer) {
  const bytes = new Uint8Array(buffer);
  const hexChars = [];

  for (let i = 0; i < bytes.length; i++) {
    const hex = bytes[i].toString(16).padStart(2 , '0');
    hexChars.push(hex);
  }

  return hexChars.join('');
}

In this code, we iterate over each byte of the ArrayBuffer using a for loop. We convert each byte to its hexadecimal representation using toString(16). The padStart(2, '0') ensures that each byte is represented by two characters, even if the value is less than 16 (e.g. , '0F' instead of just 'F'). We push each hex value into the hexChars array.

Finally , we use join('') to concatenate all the hex characters into a single string and return it.

To perform the inverse operation and convert the hexadecimal string back to an ArrayBuffer ,

function hexStringToArrayBuffer(hexString) {
  const bytes = [];

  for (let i = 0; i < hexString.length; i += 2) {
    const hexByte = hexString.substr(i , 2);
    const byte = parseInt(hexByte , 16);
    bytes.push(byte);
  }

  return new Uint8Array(bytes).buffer;
}

In this code , we iterate over the input hex string in steps of 2 to extract each byte's hexadecimal representation. We convert each hex byte to its decimal value using parseInt(hexByte, 16), and push it into the bytes array.

Finally , we create a new Uint8Array from the bytes array and obtain its underlying buffer using .buffer. This buffer represents the reconstructed ArrayBuffer.

These functions should handle large ArrayBuffers efficiently , as they avoid the performance issues associated with string concatenation for large strings.