UTF8 character encoding using node Request module

Question

I have a simple app that uses a jquery ajax request to send form data to a node server, which in turn submits to a third party api using the Request module for node js.

The issue I'm having is that accented (and other similar) characters are not encoded correctly when they reach the third party server. For example é is recorded as Ã©

I am fairly sure this is to do with the settings for Request as I get the same results when I bypass the ajax call.

Here are the settings I am using:

html:

<meta http-equiv="Content-type" content="text/html; charset=utf-8" />

jquery ajax settings:

type        : 'POST',
url         : '/api',
data        : formData, // A json object
dataType    : 'json',
ContentType : 'text/html; charset=utf-8'

Request module settings in node (there is nothing happening to the form data between ajax post and being sent by request):

request.post({
    url: "https://testurl.com/api/",
    form: formData,
    headers: {'Content-Type': 'application/json; charset=utf-8'}
} ...

I have read various SO solutions but had no success, so any suggestions greatly appreciated.

Have you tried setting the "encoding" - option of request specifically to "utf-8"? Do you have a body-parser in your application stack? (so does the formData actually get parsed as JSON before you assign it to `request` or is it still a string there?). What happens when you `console.log(formData)` on your Node-Server? — David Losert, Mar 12 '15 at 08:59
Yes I've tried `encoding: 'utf-8'`, same result. Im using Body Parser: `app.use(bodyParser.urlencoded({ extended: false }));` and `app.use(bodyParser.json());` When i log formData just before it goes to request I get what looks like a javascript object `{ first_name: 'é' }` — gethyn1, Mar 12 '15 at 09:47
You said you get "the same results" when you bypass the ajax call? You mean sending the data to the third party directly? Then it works? — David Losert, Mar 12 '15 at 11:35
I mean when I use the Request post route on the node server without involving ajax, I still get the incorrect character. If I post directly to the third party I don't get the error (using their endpoint without node or ajax involved). This suggests to me that the issue is with Request. — gethyn1, Mar 12 '15 at 13:24
Try to set `encoding: 'utf8'` (without the Dash) - according to the thread here it cold be the solution: http://stackoverflow.com/questions/8332500/module-request-how-to-properly-retrieve-accented-characters-%EF%BF%BD-%EF%BF%BD-%EF%BF%BD — David Losert, Mar 12 '15 at 13:51
You might need to post some more code showing how you handle `formData` all the way through (although you say there is nothing happening to it). What do you see if you put the form data out to the console before you POST it from nodejs? Why are you setting `contentType` in your `jQuery` `ajax` ... why not leave default `application/x-www-form-urlencoded; charset=UTF-8`? — javabrett, May 27 '15 at 06:40

score 2 · Accepted Answer · answered Jun 01 '15 at 09:57

After researching character encoding I discovered that the issue is to do with multibyte encoding of various characters (a quick google will find some good SO posts on the subject).

I thought it was odd that Request didn't handle this automatically, so I played around with the Request syntax and managed to fix the issue. Here is my revised code that works:

request(
    {
        url: 'https://testurl.com/api/',
        method: 'POST',
        json: true,
        headers: {
            'content-type': 'application/json'
        },
        body: formData
    },
    function (error, response, body) {
        ...     
    }
);

UTF8 character encoding using node Request module

1 Answers1