Special/accented UTF-8 characters not recognised by PhantomJS

Question

I am currently having an issue with PhantomJS (version 2.1.1/Windows 7) not recognising UTF-8 characters. Prior to asking this question, I have found the following two articles useful to configuring the command prompt:

• PhantomJS arguments.js example UTF-8 not working
• Unicode characters in Windows command line - how?

As suggested by the second article, I used the command

chcp 65001

to change the code page to UTF-8. I then also set the command prompt's default font to Lucida console.

To test this had worked, I created the following UTF-8 text file

---------------------------------------------------------
San José
Cañon City
Przecław Lanckoroński
François Gérard Hollande
El Niño
vis-à-vis
---------------------------------------------------------

and then ran the following command to demonstrate whether characters were being recognised and correctly displayed by the command prompt:

type utf8Test.txt

After this worked, I turned my attention to PhantomJS. Following the instructions here i created the below settings json file to ensure that UTF-8 is the input and output character encoding (though this appears to be the default for according to the official documentation).

{
    "outputEncoding: "utf8",
    "scriptEncoding": "utf8"
} 

I then ran the following JavaScript through PhantomJS using the aforementioned json settings file in the same command prompt window:

console.log("---------------------------------------------------------");

console.log("San José");
console.log("Cañon City");
console.log("Przecław Lanckoroński");
console.log("François Gérard Hollande");
console.log("El Niño");
console.log("vis-à-vis");

console.log("---------------------------------------------------------");

page = require('webpage').create();

// Display the initial requested URL
page.onResourceRequested = function(requestData, request) { 
    if(requestData.id === 1){
        console.log(requestData.url);
    }
};

// Display any initial requested URL response error
page.onResourceError = function(resourceError) {
    if(resourceError.id === 1){
        console.log(resourceError.status + " : " + resourceError.statusText);
    }
};

page.open("https://en.wikipedia.org/wiki/San_José", function(status) {
    console.log("---------------------------------------------------------");
    phantom.exit();
});

The output from running this script is shown below:

From this I can see that PhantomJS is not able to understand UTF-8 special characters, and furthermore it passes the "unknown" character to websites when provided with a special or accented character as below:

URL passed to PhantomJS:   
https://en.wikipedia.org/wiki/San_José

URL passed to remote host: 
https://en.wikipedia.org/wiki/San_Jos%EF%BF%BD

----------------------------------------------

%EF%BF%BD
�

instead of:

%C3%A9
é

This causes websites to respond with '400 : Bad Request' errors, and in the case of Wikipedia specifically, requesting the URL https://en.wikipedia.org/wiki/San_Jos%EF%BF%BD results in an error message of:

Bad title - The requested page title contains an invalid UTF-8 sequence.

So, with all this being said, does anyone know how to remedy this? There are many websites these days that use UTF-8 special/accented characters in their page urls, and it would be great if PhantomJS could be used to access them.

I really appreciate any help or suggestions you can provide me with.

Answer 1

var url = 'https://en.wikipedia.org/wiki/San_José';

page.open(encodeURI(url), function(status) {
    console.log("---------------------------------------------------------");
    console.log(page.evaluate(function(){ return document.title }));
    phantom.exit();
});

Yes, it's garbling those symbols on Windows (on Linux it works beautifully) but at least you will be able to open pages and process them.