Ajax autocompletion lost in character encoding

Question

The scheme is a text input field in a html form to be autocompleted using jQuery.autocomplete and getting the appropriate server response (e;g. a city name json list). The whole package works well... except that the client does not get data returned from the server when typing accented characters (éèà..). Same as many, it looks like I'm facing a char encoding issue but can not manage to figure out where and how to solve it despite many tries (iconv, utf8_encode, urldecode...) and readings like this one for example.

Therefore I'd need some help/hints to understand where to act (before prototyping jQuery autocomplete code ... ?)

EDIT: might be also a jQuery accent folding issue, I'll try also that way.

Configuration:
server: Apache2.2 (debian lenny)
php : compiled 5.3.3 (so the option JSON_UNESCAPED_UNICODE is not available for json_encode)
mysql: 5.1.49 with MySQL charset: UTF-8 Unicode (utf8),
class: using a modified PFBC2.x version for the php form building
meta
The website is mostly for french users so it's all designed with ISO-8859-1 (bad initial choice I guess) :
```
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
```

jQuery autocomplete code (applied to the city input field)

// DEBUG Testing (tested w/ and w/o the $charset_attr: no change)
$charset_attr = 'contentType: "application/x-www-form-urlencoded;charset=ISO-8859-1"';
echo 'jQuery("#' . $this->attributes["id"] . '").autocomplete({source:"' , $this->xhr_path . '", minLength:2, ' . $charset_attr .'});';

The generated code for that input field is matching the above expection.

Converting mysql rows into utf8 using this function :
I convert the msyql returned array into utf8 prior to sending json back to the client. Actually I tested and wrote also other functions, but this does not change anything so I guess the point is not there.
```
$encoded_arr = utf8json($returnData);   
echo json_encode($encoded_arr);   
flush();   
```

Encoding control 1 (client side)
A embed control in the html form in order to check which char encoding is actually passed to jQuery.autocomplete :

jQuery(document).ready(function() {   
<?php   
$test_str ="foobar";
$check_encoding = "'" . mb_detect_encoding($test_str) . "'";
?>
alert('Check charset server encoding: ' + <?php echo $check_encoding;?> ); // output : ASCII
});

Encoding control 2 (server side)

$inputData = (isset($_GET))? htmlspecialchars($_GET['term'],ENT_COMPAT, 'UTF-8') : NULL;   
$encoding_get = mb_detect_encoding($_GET['term']);   
$encoding_data = mb_detect_encoding($inputData);   
$utf8converted = @iconv(strtolower($encoding_get), 'utf-8', $inputData);   
$checkconversion = mb_detect_encoding($utf8converted);

Sending lowcase normal characters (ea..), I get all as ASCII.
Sending lowcase accented characters (éèà..), I get all as UTF8.

So I'm lost as the server receives the proper char string, produces a json return (tested without ajax) but it looks like the client does not receive or interprate this properly.

score 0 · Answer 1 · edited May 23 '17 at 10:08

For those facing the same kind of ...%$@ issue, here is what I've done to solve my case :

Checking the char encoding at each node (eg client, apache server, mysql server), using mb_detect_encoding on the server side,
Finally pointed out the problem location node : in my case passing UTF8 chars to the mysql server i/o latin ISO-8859-1, so mysql server did not return the expected answers, which I could not detect or debug with direct url POSTing data to the server script. So I had log the input and output in a file, checking entry character encoding and mysql server output.
Changed the ajax request to POST i/o GET,
Solved by encoding $_POST data to ISO prior sending the mysql server request, using mb_convert_encoding, as well described here.

Ajax autocompletion lost in character encoding

1 Answers1