Node.js mikeal/request module - Garbled non-utf8 website (Shift_JIS)

Question

I am trying to access a non utf-8 website using request module. Response is garbled for this request.

var request = require('request');
request('http://www.alc.co.jp/', function (error, response, body) {
  if (!error && response.statusCode == 200) {
    console.log(body) // Print the web page.
  }
});

Even after setting the encoding option to Shift_JIS I am seeing garbled Japanese text.

https://github.com/ashtuchkin/iconv-lite I think you can use this to decode SHIFT_JIS buffer into UTF-8 string. — Jerry, Mar 29 '15 at 08:13
found another for Node here. I'm going to try and make it into a Meteor package. https://github.com/polygonplanet/encoding.js — ChatGPT, Mar 29 '15 at 09:20

score 6 · Answer 1 · answered Apr 03 '15 at 08:59

You need to do the conversion yourself. The example code below uses node-iconv.

    var Iconv = require('iconv').Iconv;
    var request = require('request');
    request({
      uri: 'http://www.jalan.net/',
      encoding: null,
    }, function (error, response, body) {
      if (!error && response.statusCode == 200) {
        body = new Iconv('shift_jis', 'utf-8').convert(body).toString();
        console.log(body); // Print the web page.
      }
    });

The encoding: null parameter asks request not to convert the Buffer (a byte array) into String yet.
We pass this buffer to Iconv for converting into another Buffer of UTF-8 encoding.
Now this Buffer is good for being converted into a String.

(BTW, http://www.alc.co.jp has switched to UTF-8, so I substituted with another site.)

Thanks Alan! This should help some people. Cheers! – ChatGPT Apr 05 '15 at 11:47 — ChatGPT, Apr 05 '15 at 11:47

Node.js mikeal/request module - Garbled non-utf8 website (Shift_JIS)

1 Answers1