Why are non-ASCII characters displayed as weird symbols?

Question

I have two cases here:

My database contains a lot of information which I want to fetch to the page. Some of this information is name which contain non-ASCII characters like Uwe Rülke

- Old solution which works well:

I fetch the data from the database and populate the page directly from a VB while loop. In this case all the chars are displaying correctly Uwe Rülke.

- New solution which doesn't work properly:

The VB While loop doesn't throw the data directly to the page, rather in a JavaScript strings (to enhance performance by not calling the database each now and then). But when I used the information stored in the JavaScript variables, I got something like this: Uwe R�lke.

In both cases, the page's encoding is:

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">

Where did I go wrong?

This is the code used to fetch (from the database) and then save to JavaScript strings.

I'm using AJAX LOAD from a page called ISEquery to build a specific request and query it from the database. It is used to either fetch data as an Excel file or as plain HTML. At this point the characters are well represented.

Then the magic happens, and the characters get mis-represented. I checked it in the exctractFields function:

$("<div></div>").load("ISEquery.asp?isExcel=0&" + info, function(){
                            // Extracting the fields into an array
                            var rows = "";
                            var options = "";
                            $(this).children().each(function(index){
                                var fieldsArray = exctractFields($(this).html());
                                rows += createISELine(fieldsArray);
                                options += createISELine_ComboBox(fieldsArray);
                            });
                        });

AFAIK, Javascript uses UTF-8, so it's a misecoding. You need to output your page in UTF-8 — This company is turning evil., Jul 23 '14 at 11:24
@Kroltan, nope didn't work! I changed it between several recordings and same result. — user2517028, Jul 23 '14 at 11:30
Show us the code that you use to transfer the data to the browser. — Aaron Digulla, Jul 23 '14 at 11:32
@AaronDigulla the question is updated with the relevant code. — user2517028, Jul 23 '14 at 12:08
@Kroltan ISO-8859-1 work just fine with AJAX you can specify which encoding you would like to use. — Sebastien, Jul 23 '14 at 12:38
Client-side logic is too late for HTTP GET requests I suspect. Your server (ISEquery.asp) looks like it is already sending characters in one charset but telling the browser they belong to another. If you can sniff the HTTP response headers, then you'll see what character set your server has declared it is using. Many times people convert pages 'to AJAX' and forget to check the character encodings of the AJAX responses, since some servers automatically add character set HTTP response codes to responses that it can identify as containing HTML, CSS or JS but not others. — Lee Kowalkowski, Jul 23 '14 at 12:39
response body in the browser shows the non Ascii character correctly but it response.json() renders it as � . Any ideas? — partizanos, Apr 18 '17 at 20:15
What is "VB"? [Visual Basic for Applications](https://en.wikipedia.org/wiki/Visual_Basic_for_Applications) (VBA)? — Peter Mortensen, Aug 15 '21 at 12:07

score 6 · Answer 1 · edited Aug 15 '21 at 12:12

6

The � means that you used a character which can't be represented properly.

Somewhere between the server and the client, you need to encode the string data properly. I don't know how you transfer the data from the server to the client (generate JavaScript, Ajax, and GET requests). It's hard to say how to fix this.

But what you need to do: For every step, you must make sure that you know what the current encoding of the data is and what the recipient expects.

For example, if you generate inline JavaScript code in an HTML page, then the string value must be encoded with the same encoding as the page (iso-8859-1). If you use Ajax, then usually you have to use UTF-8.

edited Aug 15 '21 at 12:12

Peter Mortensen

30,738
21
105
131

answered Jul 23 '14 at 11:36

Aaron Digulla

321,842
108
597
820

Thank you, it is a useful answer, but didn't really resolve my problem yet. I update the question with a peace of code, it may be more informative now – user2517028 Jul 23 '14 at 12:09
1

ISO-8859-1 (Latin1) can be use without problems in AJAX. You do not need to use UTF-8. But it is recommended that for Web pages/apps you use UTF-8/UTF-16 since UTF contains more character than Latin and your users will most likely be using different languages such as French Spanish Russian Chinese Japanese. – Sebastien Jul 23 '14 at 12:29
Here is how you setup some encoding on web pages/apps: HTML: PHP:header('Content-Type: text/html; charset=UTF-8'); AJAX (noJQuery): xhr.setRequestHeader("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8"); – Sebastien Jul 23 '14 at 12:31
-EDIT- xhr in the previous comment is a XMLHttpRequest object or a ActiveXObject object – Sebastien Jul 23 '14 at 12:35
One last thing the encoding the code file is using does affect the rendering of the page. For exemple: if your code is in a file encoded in Latin1 or ANSI (both are very similar) and you would like to display UTF-8, you will need to switch the file encoding to UTF-8 without BOM. (What I do to switch the file encoding is: I open it in Notepad++ and change the encoding then save the file). – Sebastien Jul 23 '14 at 12:41

score 2 · Accepted Answer · edited Aug 15 '21 at 12:14

2

I followed the string from server to the page and found that it is gets misrepresented after the AJAX LOAD, so I found this answer which resolved my problem. Although I had to use the charset="iso-8859-1" for it to work rather than charset="UTF-8".

So the final answer is:

-Encoding in the HTML page:

<meta http-equiv="Content-Type" content="text/html"; charset="iso-8859-1">

-Encoding the Ajax data:

 $.ajaxSetup({
          'beforeSend' : function(xhr) {
           xhr.overrideMimeType('text/html; charset=iso-8859-1');
        },
    });

And now characters are displayed correctly.

_{(The lead was from Aaron Digulla's answer.)}

edited Aug 15 '21 at 12:14

Peter Mortensen

30,738
21
105
131

answered Jul 23 '14 at 12:57

user2517028

784
1
11
25

This suggests that you've configured your server to always return data with the encoding `iso-8859-1`. This will work as long as only people from (West-)Europe use it. You should consider using `UTF-8` everywhere, instead. – Aaron Digulla Jul 23 '14 at 13:45

score -2 · Answer 3 · edited Aug 15 '21 at 12:09

-2

The JavaScript default encoding for strings is UTF-16 (16 bits) while ISO 8859-1 is 8 bits.

What is the default JavaScript character encoding?

I think you can use encodeURI() to convert your special characters to ASCII characters and afterwards you can decode it with decodeURI():

JavaScript encodeURI() Function (W3Schools)

edited Aug 15 '21 at 12:09

Peter Mortensen

30,738
21
105
131

answered Jul 23 '14 at 11:28

GCallie

413
1
3
11

3

-1 The default JavaScript source file encoding is UTF-8. 16bit character values are used to build JS strings in memory. That has nothing to do with the source code encoding. – Aaron Digulla Jul 23 '14 at 11:31
@AaronDigulla I meant it for strings, my fault. I included the link where I refer to an question where they explain the encoding for strings. – GCallie Jul 23 '14 at 11:35
In that case, the answer doesn't help. The problem is that somewhere between his VB script and the client, he's not using the correct encoding. – Aaron Digulla Jul 23 '14 at 11:37

Why are non-ASCII characters displayed as weird symbols?

3 Answers3

Linked