How to encode integer to Uint8Array and back to integer in JavaScript?

Question

I need to add compression to my project and I decided to use the LZJB algorithm that is fast and the code is small. Found this library https://github.com/copy/jslzjb-k

But the API is not very nice because to decompress the file you need input buffer length (because Uint8Array is not dynamic you need to allocate some data). So I want to save the length of the input buffer as the first few bytes of Uint8Array so I can extract that value and create output Uint8Array based on that integer value.

I want the function that returns Uint8Array from integer to be generic, maybe save the length of the bytes into the first byte so you know how much data you need to extract to read the integer. I guess I need to extract those bytes and use some bit shifting to get the original number. But I'm not exactly sure how to do this.

So how can I write a generic function that converts an integer into Uint8Array that can be embedded into a bigger array and then extract that number?

Answer 1

General answer

These functions allow any integer (it uses BigInts internally, but can accept Number arguments) to be encoded into, and decoded from, any part of a Uint8Array. It is somewhat overkill, but I wanted to learn how to work with arbitrary-sized integers in JS.

// n can be a bigint or a number
// bs is an optional Uint8Array of sufficient size
//   if unspecified, a large-enough Uint8Array will be allocated
// start (optional) is the offset 
//   where the length-prefixed number will be written
// returns the resulting Uint8Array
function writePrefixedNum(n, bs, start) {
  start = start || 0;
  let len = start+2; // start, length, and 1 byte min
  for (let i=0x100n; i<n; i<<=8n, len ++) /* increment length */;
  if (bs === undefined) {  
    bs = new Uint8Array(len);
  } else if (bs.length < len) {
        throw `byte array too small; ${bs.length} < ${len}`;
  }
  let r = BigInt(n);
  for (let pos = start+1; pos < len; pos++) {
    bs[pos] = Number(r & 0xffn); 
        r >>= 8n;
  }
  bs[start] = len-start-1; // write byte-count to start byte
  return bs;
}

// bs must be a Uint8Array from where the number will be read
// start (optional, defaults to 0)
//    is where the length-prefixed number can be found
// returns a bigint, which can be coerced to int using Number()
function readPrefixedNum(bs, start) {
  start = start || 0;
  let size = bs[start]; // read byte-count from start byte
  let n = 0n;
  if (bs.length < start+size) {
        throw `byte array too small; ${bs.length} < ${start+size}`;
  }    
  for (let pos = start+size; pos >= start+1; pos --) {
    n <<= 8n;
    n |= BigInt(bs[pos])
  }
  return n;
}

function test(n) {
  const array = undefined;
  const offset = 2;
  let bs = writePrefixedNum(n, undefined, offset);
  console.log(bs);
  let result = readPrefixedNum(bs, offset);
  console.log(n, result, "correct?", n == result)
}

test(0)
test(0x1020304050607080n)
test(0x0807060504030201n)

---

Simple 4-byte answer

This answer encodes 4-byte integers to and from Uint8Arrays.

function intToArray(i) {
    return Uint8Array.of(
      (i&0xff000000)>>24,
      (i&0x00ff0000)>>16,
      (i&0x0000ff00)>> 8,
      (i&0x000000ff)>> 0);
}

function arrayToInt(bs, start) {
    start = start || 0;
    const bytes = bs.subarray(start, start+4); 
    let n = 0;
    for (const byte of bytes.values()) {       
            n = (n<<8)|byte;
    }
    return n;
}

for (let v of [123, 123<<8, 123<<16, 123<<24]) {
  let a = intToArray(v);
  let r = arrayToInt(a, 0);
  console.log(v, a, r);
}

Answer 2

Here are working functions (based on Converting javascript Integer to byte array and back)

function numberToBytes(number) {
    // you can use constant number of bytes by using 8 or 4
    const len = Math.ceil(Math.log2(number) / 8);
    const byteArray = new Uint8Array(len);

    for (let index = 0; index < byteArray.length; index++) {
        const byte = number & 0xff;
        byteArray[index] = byte;
        number = (number - byte) / 256;
    }

    return byteArray;
}

function bytesToNumber(byteArray) {
    let result = 0;
    for (let i = byteArray.length - 1; i >= 0; i--) {
        result = (result * 256) + byteArray[i];
    }

    return result;
}

by using const len = Math.ceil(Math.log2(number) / 8); the array have only bytes needed. If you want a fixed size you can use a constant 8 or 4. In my case, I just saved the length of the bytes in the first byte.

Answer 3

Posting this one-liner in case it is useful to anyone who is looking to work with numbers below 2^53. This strictly uses bitwise operations and has no need for constants or values other than the input to be defined.

export const encodeUvarint = (n: number): Uint8Array => n >= 0x80 
    ? Uint8Array.from([(n & 0x7f) | 0x80, ...encodeUvarint(n >> 7)]) 
    : Uint8Array.from([n & 0xff]);