Javascript: convert CSV string into a) UTF-8 and b) a 2D array

Question

Two questions in one, not sure if that's allowed, but they're directly related to the same code. I retrieve a CSV string as a HTTP response in Javascript - this string comes in UTF-16 encoding it seems, as it has for example ' € ' instead of '€'.

a) How can I convert this to UTF-8 in vanilla Javascript?

Once that's done, how do I b) transform the multi-line CSV into a 2D array in vanilla Javascript?

Thanks!

---

[UPDATE]

Based on anqooqie's pointers, I take the following approach to re-encode the string:

OK, clear - so to be honest, I went a slightly different way (as the reencode function didn't work for me and it threw a generic error code) and now do the below;

var O = new ActiveXObject('ADODB.Stream');
O.Type = 2; 
O.Open;
O.Charset = 'ISO-8859-1';
O.LineSeparator = 10;
O.WriteText (csvStr);
O.Position = 0;
O.Charset = 'UTF-8';

And this works fine and in pretty much a split second (even though it's a 35K row CSV). Now if I want to put it back into the csvStr, I would do

csvStr = O.ReadText

but this takes ages - is that expected or am I doing something wrong?

For putting it into a 2D array, I split on the LineSeparator and then loop using a regex, which seems to work.

var A = new Array
A.push(csvStr[0].match(/"[^"]*"|[^,]+/g))

The vast delay on the readText is bothering me though, especially as the WriteText is so quick. Any help is appreciated.

Answer 1

Looks like you are confused about the terms of character encoding, so let's reconfirm that.

String is just a string. There is no "UTF-16 string", nor "UTF-8 string".

Character encoding is a protocol which converts between a string and a byte array. UTF-16 is one of the character encodings. Also, both of UTF-8 and ISO-8859-1 are character encodings. In UTF-16, the string '€' can be encoded to a byte array 20 AC. In UTF-8, the string '€' can be encoded to a byte array E2 82 AC. In ISO-8859-1, the byte array E2 82 AC can be decoded to a string 'â¬'.

Now, you may find that 'â¬' is not a "UTF-16 string". It is '€' encoded as UTF-8 and mistakenly decoded as ISO-8859-1.

a) How can I convert this to UTF-8 in vanilla Javascript?

What you should do is to fix the code to retrieve a CSV file. I cannot tell you how to fix it since I do not know your code, but I believe that it now decodes a CSV file as ISO-8859-1. You should fix the character encoding from ISO-8859-1 to UTF-8.

If the code is not yours and you cannot fix it, you can use a workaround. In other words, you can 1) re-encode a mistakenly decoded string as ISO-8859-1, and 2) re-decode it as UTF-8.

1)

// Note: This code requires ES5 or later.
function reencode(inputString) {
  return Array.apply(null, Array(inputString.length)).map(function (x, i) { return inputString.charCodeAt(i); });
}

2)

See this answer.

b) How do I transform the multi-line CSV into a 2D array in vanilla Javascript?

See this answer.