Chrome extension ajax sending malformed accented characters

Question

I am sending an AJAX POST request using jQuery on a chrome extension but the data doesn't arrive as expected, accented characters turn out malformed.

The text "HÄGERSTEN" becomes "HÄGERSTEN".

The text appears fine in the console etc, only via AJAX to this other page it appears as mentioned. My AJAX call is basic, I send a data-object via jQuery $.ajax. I've tried both with and without contentType, UTF-8 and ISO-8859-1. No difference.

This is how I make my AJAX call:

var newValues = {name: 'HÄGERSTEN'}

$.ajax({
    url: POST_URL,
    type: 'POST',
    data: newValues,
    success: function() ...
});

The newValues object has more values but I retrieve them from a form. However, I have tried to specify these values manually as newValues['name'] = 'ÄÄÄÄ'; and still would cause the same problem.

The original form element of the page that I am sending the AJAX to contains attribute accept-charset="iso-8859-1". Maybe this matters.

The target website is using Servlet/2.5 JSP/2.1. Just incase it might make a difference.

I assume this is an encoding issue and as I've understood it should be because Chrome extensions require the script files to be UTF-8 encoded which probably conflicts with the website the plugin is running on and the target AJAX page (same website) which is using an ISO-8859-1 encoding, however I have no idea how to deal with it. I have tried several methods of decoding/encoding it to and from UTF-8 to ISO-8859-1 and other tricks with no success.

I have tried using encodeURIComponent on my values which only makes them show that way exactly on the form that displays the values I have sent via POST, as e.g. H%C3%84GERSTEN.

I have no access to the websites server and cannot tell you whether this is a problem from their side, however I would not suppose so.

UPDATE

Now I have understood POST data must be sent as UTF-8! So a conversion is not the issue?

Answer 1

Seems like the data is UTF-8 encoded when it is sent and not properly decoded on the server side. It has to be decoded on the server side. Test it out with the following encode and decode functions:

function encode_utf8(s) {
  return unescape(encodeURIComponent(s));
}

function decode_utf8(s) {
  return decodeURIComponent(escape(s));
}

var encoded_string = encode_utf8("HÄGERSTEN");
var decoded_string = decode_utf8(encoded_string);
document.getElementById("encoded").innerText = encoded_string;
document.getElementById("decoded").innerText = decoded_string;

<div>
Encoded string: <span id="encoded"></span>
</div>
<div>
Decoded string: <span id="decoded"></span>
</div>

Answer 2

We too faced the same situation but in our case we always sent the parameters using JSON.stringify.
For this you have to make changes,
1) While making call to the page via AJAX you have to add content-type tag defining in which encoding data is sent

$.ajax
    ({
        type: "POST",
        url: POST_URL,
        dataType: 'json',//In our case the datatype is JSON
        contentType: "application/json; charset=utf-8",
        data: JSON.stringify(newValues),//I always use parameters to be sent in JSON format

EDIT
After reading your question more clearly I came to know that your server side JSP uses ISO-8859-1 encoding and reading some posts, I came to know that all POST method data will be transmitted using UTF-8 as mentioned.

POST data will always be transmitted to the server using UTF-8 charset, per the W3C XMLHTTPRequest standard

But while reading post jquery-ignores-encoding-iso-8859-1 there was a workaround posted by iappwebdev which might be useful and help you,

 $.ajax({
    url: '...',
    contentType: 'Content-type: text/plain; charset=iso-8859-1',
    // This is the imporant part!!!
    beforeSend: function(jqXHR) {
        jqXHR.overrideMimeType('text/html;charset=iso-8859-1');
    }
});

Above code is taken from Code Posted by iappwebdev

Answer 3

I don't know if it could have been solved using POST-data and AJAX. Perhaps if I made a pure JavaScript XHR AJAX call, I might be able to send POST-data encoded the way I like. I have no idea.

However, in my desperation I tried my final option (or what seemed like it); send the request as GET-data. I was lucky and the target page accepted GET-data.

Obviously the problem still persisted as I was sending data the same way, being UTF-8 encoded. So instead of sending the data as an object I parsed the data into a URL friendly string with my own function using escape, making sure they are ISO-8859-1 friendly (as encodeURIComponent encodes the URI as UTF-8 while escape encodes strings making them compatible with ISO-8859-1).

The simple function that cured my headaches:

function URLEncodeISO(values) {
   var params = [];
   for(var k in values) params[params.length] = escape(k) + '=' + escape(values[k]);
   return params.join('&');
}

Answer 4

The client side character coding is not completely up to you (immagine the usage of the page from different users all around the world: chinese, italian...) while on the server side you need to handle the coding for your purposes.

So, the data in the Ajax-POST can continue to be UTF8-encoded, but in your server side you coan to do the following:

PHP:

$name = utf8_decode($_POST['name']);

JSP:

request.setCharacterEncoding("UTF-8");
String name = request.getParameter("name");