Converting non-unicode binary strings to byte array in Javascript

Question

I read a binary file using ajax from a web server. The HTTP response is of contentType: 'application/octet-stream' and contains a binary string that is just a string of bytes (not unicode), for example (in hex):

0x00 0x08 0x17 0xA1 0x01

Note: in C this would be represented as 5 bytes in memory:

char buf[5] = {0, 8, 23, 161, 1}

...but in Javascript that is a string which ASCII representation is something like " � " (I can't actually paste it properly as not all characters have a printable representation).

I now need to convert this into an array of characters or integers so that I can access the numerical value of each character in the string. However iterating through the example string using the charCodeAt() function returns:

[0] 0
[1] 8
[2] 23
[3] 65533
[4] 1

because charCodeAt() decodes unicode characters, and 0xA1 is not recognised as valid unicode character so a Replacement Character (65533) is used instead.

I would like to get the following:

[0] 0
[1] 8
[2] 23
[3] 161
[4] 1

How to achieve this?

Here is the code snippet:

$.ajax({
url: url,
type: "get",
success: function(data) { // data contains binary representation of 0x00 0x08 0x17 0xA1 0x01
    var byteTab = []
    for (var n = 0; n < data.length; ++n) {
        byteTab.push(data.charCodeAt(n))
    }
})

Haus · Answer 1 · 2018-08-08T11:55:23.427

Update: I'm not so sure [default] Ajax is the right tool for the job in this case. Regardless of file size, it is generally a good practice to use streams instead to accommodate future scalability, as your current method loads everything at once.

This article goes over how to create a custom ajax transport that wraps an XmlHttpRequest to load the data into an array buffer rather. You could do this, sequentially push the bytes into your main array, and go from there.

Old: I'm not sure if I properly understood your data representation, but I believe you can just use parseInt() to turn a hex string into decimal:

var data = ['0x00', '0x08', '0x17', '0xA1', '0x01'];
var parsed = [];

for(var i = 0; i < data.length; i++) {
 parsed.push(parseInt(data[i], 16));
}

console.log(parsed);
console.log(parseInt('0xA1', 16))

If this is not what you mean, please comment and I will try to update my answer with a more specific implementation.

Apologies if my question was not clear in that respect - I have now update the question. The content of the "data" buffer returned from the web server in my code snippet is a string corresponding to 5 bytes. In C, this could be represented as as: char buf[5] = {0, 8, 23, 161, 1}. In Javascript this is returned as a string of length 5. Sine Javascript does not operate on "bytes" or "chars" like C, I can't find a way to get the numerical value of individual characters in this string since some like 0xA1 (161) are not unicode characters. — Kane, Aug 08 '18 at 10:32

score 1 · Answer 2 · answered Aug 08 '18 at 19:23

Following the suggestion from Haus 's answer I found some more solutions to the problem so I will share my findings here.

My preferred solution is to use ArrayBuffer to access raw data. Unfortunately Ajax does not seem to support it as per this article, although there is an option to add support (which I could not get to work), and there is no straightforward way to create an ArrayBuffer out of string. However XMLHttpRequest supports ArrayBuffer as responseType as explained here so I changed my code to the following, which does what I want:
```
var xhr = new XMLHttpRequest();
xhr.open('GET', url);
xhr.responseType = 'arraybuffer';  
xhr.onload = function() {
    if (this.status == 200) {
        var byteTab=[]
        var uint8View = new Uint8Array(this.response)
        for (var n = 0; n < uint8View.byteLength; n++) {
            byteTab.push(uint8View[n])
        }

    } 
};
xhr.send();
```
Another solution is to get to ArrayBuffer via using Blob, which is explained with an example in this answer
Another option is to use Blob in conjunction with FileReader.readAsBinaryString() along the lines of the example here. Using readAsBinaryString() converts the blob to a unicode string which can then be parsed using charCodeAt()

Converting non-unicode binary strings to byte array in Javascript

2 Answers2