Parsing XML to JSON UTF-8

Question

I'm using xml2js, since I need to convert a XML feed to JSON. But when I receive the XML, it shows Æ, Ø & Å as expected. But after parsing it.

I receive: Ø as \ufffd or �.

I'm already setting the encoding as UTF-8, so I'm not sure what I'm doing wrong. Can anyone enlighten me :-)?

var fs = require('fs')
var https = require('https')
var xml2js = require('xml2js')
var parser = new xml2js.Parser()

router.get('/api/xml', (req, res) => {
  https.get('urlForRequest', function (response) {
    var response_data = '';     

    response.setEncoding('utf8');
    response.on('data', function (chunk) {
         response_data += chunk;             
    });
    response.on('end', function () {
      parser.parseString(response_data, function (err, result) {
        if (err) {
          console.log('Got error: ' + err.message);
        } else {
          res.json(result)
        }
      });
    });
    res.on('error', function (err) {
      console.log('Got error: ' + err.message);
    })
  })
})

UPDATE:

I tried to follow your steps. If I fetch the XML and store it locally in a .xml file everything works great. But if I fetch it from my source (Exactly same GET request) then it doesn't work.

Response for curl http://localhost:9090/products.xml -v > download.xml

Connected to localhost (::1) port 9090 (#0)
GET /products.xml HTTP/1.1
Host: localhost:9090
User-Agent: curl/7.54.0
Accept: */*

HTTP/1.1 200 OK
X-Powered-By: Express
Accept-Ranges: bytes
Cache-Control: public, max-age=0
Last-Modified: Thu, 07 Jun 2018 09:56:41 GMT
ETag: W/"9471b6-163d9ad4696"
Content-Type: text/xml; charset=UTF-8
Content-Length: 9728438
Date: Thu, 07 Jun 2018 10:00:09 GMT
Connection: keep-alive

Reponse for curl to the 'real' source (It's a https request, if it matters)

User-Agent: curl/7.54.0
Accept: */*

HTTP/1.1 200 OK
Date: Thu, 07 Jun 2018 10:10:29 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16
X-Powered-By: PHP/5.4.16
Vary: Accept-Encoding
Connection: close
Transfer-Encoding: chunked
Content-Type: text/xml; charset=iso-8859-1

Terry Lennox · Accepted Answer · 2018-06-07T11:07:28.067

I've set up some script files to replicate your results.. everything seems to work OK from what I can see.

I've created an express server to serve a static XML file with ÅØ characters in it. This is always a good place to start with these issues, isolate the problem.

server.js

const express = require("express");
const app = express();
const port = 3000;

app.use('/', express.static(__dirname));
app.listen(port);

console.log('Express started on port ' + port + '...');

index.js

const xml2js = require('xml2js')
const parser = new xml2js.Parser()
const http = require('http');

var url = 'http://localhost:3000/test.xml';

http.get(url, (response) => {

    var response_data = '';     
    // Try latin1 encoding.
    response.setEncoding('latin1');
    response.on('data', function (chunk) {
         response_data += chunk;             
    });
    response.on('end', function () {
      parser.parseString(response_data, function (err, result) {
        if (err) {
          console.log('Got error: ' + err.message);
        } else {
          console.log('Result JSON: ', JSON.stringify(result, null, 4));
        }
      });
    });

});

test.xml

<root>
    <testÅØ id="198787">
    </testÅØ>
</root>

All files are in the same directory. Start server.js then index.js, this should download the test xml file and display the parsed result. Using my setup I get the following output:

{
    "root": {
        "testÅØ": [
            {
                "$": {
                    "id": "198787"
                }
            }
        ]
    }
}

I'm wondering if there is an issue with the original XML file. I'd try using curl to download the data and see what the file looks like, e.g.

curl urlForRequest -v > download.xml

I'd check the headers coming back, I'm getting

curl http://localhost:3000/test.xml -v > download.xml

HTTP/1.1 200 OK
X-Powered-By: Express
Accept-Ranges: bytes
Cache-Control: public, max-age=0
Last-Modified: Thu, 07 Jun 2018 09:10:31 GMT
ETag: W/"34-163d982ff58"
Content-Type: text/xml; charset=UTF-8
Content-Length: 52
Date: Thu, 07 Jun 2018 09:52:46 GMT
Connection: keep-alive

For my setup.

I can see the encoding for the real file is iso-8859-1, I'm betting this is the issue! — Terry Lennox, Jun 07 '18 at 10:15
Maybe try this: response.setEncoding("latin1"); I'll update my answer. The encoding of the file is _not_ UTF-8 so we should try something else! — Terry Lennox, Jun 07 '18 at 11:06
Excellent! The server should really.be using utf8 but this is a good workaround! — Terry Lennox, Jun 07 '18 at 11:15

score 0 · Answer 2 · answered Feb 27 '23 at 11:21

0

What solve to me was using utf-8 on readAsText function

reader.readAsText(file, 'utf-8');

answered Feb 27 '23 at 11:21

Juan Colchete

51
4

Parsing XML to JSON UTF-8

2 Answers2