Convert windows1252 to utf-8 in NodeJS special characters

Question

I am building a web application in NodeJS version 12. I have data from an old MySQL database. There are several fields that contain characters that are not displaying properly due to an encoding issue with the old database. There are some similar questions already but none of them have solved my issue. After trying, I'm a little closer to a solution, but still need help on this.

Current value in database to convert:

Rikuchi SokuryoÌ„bu [cartographer], 陸地測é‡éƒ¨

Desired new database value:

Rikuchi Sokuryōbu [cartographer], 陸地測量部

The issue is the same as described in this similar question. However, the accepted answer does not solve my issue. I need to write a NodeJS to convert the data in the database into a readable string.

I also tried following the answer in this similar question. I understand that the value needs to be first converted to binary and then to the desired encoding. However, it does not return the desired result. I tried this with the iconv and iconv-lite packages.

ATTEMPT 1:

let buf = new Buffer(body, 'binary');
let conv = new iconv.Iconv('windows-1252', 'utf8');
let str = conv.convert(buf).toString();
console.log(`original: ${body} output: ${str.toString()}`);

// original: Rikuchi SokuryoÌ„bu [cartographer], 陸地測é‡éƒ¨
// output: Rikuchi SokuryoМbu [cartographer], й"ёеS°жё¬й!Џй’Ё

ATTEMPT 2: iconv-lite

let buf = new Buffer(body, 'binary');
const str = iconvlite.decode(buf, 'windows-1252');
console.log(`original: ${body} output: ${str.toString()}`);

// original: Rikuchi SokuryoÌ„bu [cartographer], 陸地測é‡éƒ¨
// output: Rikuchi SokuryoМbu [cartographer], й"ёеS°жё¬й!Џй’Ё

ATTEMPT 3: iconv-lite

// This one *almost* works however there are still some undefined characters

let buf = new Buffer(body, 'utf-8');
const win = iconvlite.encode(buf, 'windows-1252');
console.log(`original: ${body} output: ${win.toString()}`);

// original: Rikuchi SokuryoÌ„bu [cartographer], 陸地測é‡éƒ¨
// output: Rikuchi Sokuryōbu [cartographer], 陸地測�?部

UPDATE:

This website string-functions.com can encode and decode strings.

The entire problematic string is decoded correctly with the settings: "Encode with: Windows-1252" and "Decode with: utf-8"

It also works perfectly for larger examples of this problem. I just need to replicate exactly how this site is doing the conversion. My code in attempt #3 is very close, but there must be a step missing.

Answer 1

I solved this by using the windows-1252 module to encode the original text and then decoded it using the iconv-lite module.

const win = windows1252.encode(body);
const str = iconvlite.decode(win, 'utf-8');
return str.toString();

Answer 2

Great post, and thank you @pengz, that worked for me. I agree all the other solutions missed either the apostrophe and other characters. This is the only solution that worked for me. One thing to note however: I was able to accomplish the same, but all with iconvlite:

const win = iconvlite.encode(body, 'windows-1252');
const str = iconvlite.decode(win, 'utf-8');
return str.toString();

Answer 3

If you get in response something like questions mark (Unicode block) �, it means your notepad (IDE) shows it in utf8 format, but it gets its data in another format.

To fix it I used iconv:

const { Iconv } = require('iconv');
const convertedXmlResponse = Iconv('windows-1251', 'utf8')
    .convert(dataBuffer)
    .toString();