Javascript combine ArrayBuffer parts

Question

I need to combine 2 parts of 2 existing arrayBuffers into a new one.

I am building a parser and the data comes in arraybuffers of random sizes, the data will spill over the end of one, into the beginning of the other. So I need to create a new output buffer and copy in a portion of the end of one buffer and a portion of the beginning of the other. The output will just be an Arraybuffer.

Starting out with this demo, I was going to make Uint8Arrays with some offsets then use set, the problem is certain combinations throw Invalid typed array length. I will not know the length of each array or offsets beforehand.

var buffer1 = new ArrayBuffer(8);
var buffer2 = new ArrayBuffer(8);
var buffer3 = new ArrayBuffer(8);

var uint8_1 = new Uint8Array(buffer1);
var uint8_2 = new Uint8Array(buffer2);
var uint8_3 = new Uint8Array(buffer3);

uint8_1.fill(1);
uint8_2.fill(2);

var uint8_1_slice = new Uint8Array(buffer1 , 0 , 3);
var uint8_2_slice = new Uint8Array(buffer2 , 4, 7);

For this demo need to get buffer3 to be 1,1,1,1,2,2,2,2.

Cannot Use Slice

Endless · Answer 1 · 2016-10-18T12:45:25.577

I have seen some people only use array.length. It's fine as long as the array is only 1 byte per element. It's also fine if the other typed arrays are filled up but in this example a2 isn't. That is why it's better to use byteLength this is also how Blob constructor concatenate the parts.

// Concatenate a mix of typed arrays
function concatenate(...arrays) {
  // Calculate byteSize from all arrays
  let size = arrays.reduce((a,b) => a + b.byteLength, 0)
  // Allcolate a new buffer
  let result = new Uint8Array(size)

  // Build the new array
  let offset = 0
  for (let arr of arrays) {
    result.set(arr, offset)
    offset += arr.byteLength
  }

  return result
}

// the total length of 1-3 = 5
// the total byteLength of 1-3 = 6
let a1 = Uint8Array.of(1, 2) // [1, 2]
let a2 = Uint16Array.of(3) // [3] just for the fun of it 16 takes up 2 bytes
let a3 = Uint8Array.of(4, 5) // [4, 5]

concatenate(a1, a2, a3) // [1, 2, 3, 0, 4, 5]

/********/
var blob = new Blob([a1, a2, a3])
var res = new Response(blob)
res.arrayBuffer().then(buffer => console.log(new Uint8Array(buffer)))
// [1, 2, 3, 0, 4, 5]

Note: This `concatenate` function only supports Uint8Array arguments. For example, `concatenate(new Uint16Array([1,2,3]))` returns [ 1, 2, 3, 0, 0, 0 ] rather than [ 1, 0, 2, 0, 3, 0 ]. The problem is that Uint8Array.set() iterates over the *elements* of the given typed array rather than the bytes. The solution is to change `result.set(arr, offset)` to `result.set(new Uint8Array(arr.buffer, arr.byteOffset, arr.byteLength), offset)`. (Note that support for ArrayBuffer arguments would require additional logic.) — tom, Jul 29 '22 at 12:13

guest271314 · Answer 2 · 2016-09-15T03:36:17.043

1

For this demo need to get buffer3 to be 1,1,1,1,2,2,2,2.

You can use for loop, set uint8_3 to uint8_1 value if variable n is less than uint8_1.byteLength / 2 else set uint8_3 to value at uint8_2 .

var len = 8;

var buffer1 = new ArrayBuffer(len);
var buffer2 = new ArrayBuffer(len);
var buffer3 = new ArrayBuffer(len);

var uint8_1 = new Uint8Array(buffer1);
var uint8_2 = new Uint8Array(buffer2);
var uint8_3 = new Uint8Array(buffer3);

uint8_1.fill(1);
uint8_2.fill(2);
// `len` : uint8_1.byteLength / 2 + uint8_2.byteLength / 2
for (var n = 0; n < len; n++) {
  uint8_3[n] = n < len / 2 ? uint8_1[n] : uint8_2[n];
}

console.log(uint8_3);

edited Sep 15 '16 at 03:36

answered Sep 15 '16 at 03:08

guest271314

1
15
104
177

This is on the right track, It should use set though because if not you're copying 1 byte at a time. – brianxautumn Sep 15 '16 at 03:19
Actually sorry, can't use slice, it will return a new array. – brianxautumn Sep 15 '16 at 03:20
@brianxautumn _"Actually sorry, can't use slice, it will return a new array"_ Not sure what you mean? Is requirement to not create a new array? – guest271314 Sep 15 '16 at 03:20
Yes, speed and memory usage are very, very important in this case. – brianxautumn Sep 15 '16 at 03:21
Better, but the problem is that it is copying 1 byte at a time, when it could be copying up to 8 at a time. – brianxautumn Sep 16 '16 at 20:05
@brianxautumn _"Better, but the problem is that it is copying 1 byte at a time, when it could be copying up to 8 at a time."_ Yes, requirement was to not use `.slice()` "can't use slice it will return a new array"? [`.set()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/TypedArray/set) does not expect a third parameter. [`.subarray`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/TypedArray/subarray) also returns a new array. – guest271314 Sep 17 '16 at 01:19
@brianxautumn Do you want to remove the values from the two existing typed arrays? – guest271314 Sep 17 '16 at 01:40

Javascript combine ArrayBuffer parts

2 Answers2