0

I'm very new to Node.js. My goal is to develop d3 charts in HTML handling Data objects(Json Array) that have 100K plus rows using a CSV spreadsheet. I want to load the CSV file and convert it to JSON Array object in the server side so this will not crash IE11 when loading the CSV file in the front end. The CSV is a big file. When I create a node js file that creates a server and load/convert the CSV into JSON Array using require('csvtojson') module, I want to send the JSON Array object to my html so i can grab that object and plug into D3 charts. I'm trying to avoid having the "Not enough storage..." jquery error when using ajax. Is this possible? I'm trying really hard to find an example but no luck. Only sites i found is how to display the json as a string into the html page. Is there a way to grab the JSON Array object in the HTML page? This is the code i use to create a server and to make all my html pages display on the browser.

Updated: I added the csv() and it converts the data into jsonObject. So far its just getting the one json object which this is not the case. I just want to push the 'jsonData' object so i can grab that in the HTML page and plug it to d3. Is this possible? Please, need help. I cant use d3.csv() in html because the file is too large and the IE11 browser will crash. Thats why I do this in the server. 

csv().fromFile(csvFilePath)
.on('json', (jsonObj) => {
    console.log(jsonObj);
    jsonData = jsonObj;
})
.on('done', (error) => {
    console.log('end');
}); http.createServer(function (req, res) {
    var q = url.parse(req.url, true);
    var filename = "." + q.pathname;

    fs.readFile(filename, function(err, html) {
        if(err) {
            res.writeHead(404, {'Content-Type': 'text/html'});
            return res.end('404 Not Found');
        }
        res.writeHead(200, {'Content-Type': 'text/html'});
        res.write(html);
        return res.end();
    });
}).listen(8080);

Is there a way to have the JSON Array object available when I load the HTML page and grab the object to plus into d3?

Also I tried this method but it didnt work which I open this question awhile back: Not enough storage error when using Ajax. Large response

Thank you. I tried explaining the best i can. Please need help..

timJIN520
  • 299
  • 1
  • 3
  • 15
  • `d3.json` can read json files from external sources into your d3 script. Or maybe I'm not understanding the question. – Ryan Morton Nov 08 '17 at 20:07
  • @RyanMorton no need for d3.json. I already have a function that handles and plugs the JSON Array data object into the d3 charts. The thing i'm trying to figure out is load the file from the server side, have the JSON array object ready in the server side, and on html page grab that object. Before i tried using d3.csv(...) in html but the IE11 browser can't handle amount of memory of that csv file and crashes. That's why I'm doing this on the server. Sorry, very new to Node.js and trying to find a solution without using DB – timJIN520 Nov 08 '17 at 20:18
  • So you just need to stream the csv? https://stackoverflow.com/questions/33129677/nodejs-promises-streams-processing-large-csv-files – Ryan Morton Nov 08 '17 at 21:44
  • @RyanMorton not sure what Stream does in node.js. I read about it and not sure how it works.. I have the jsonObject in the server side. I know how to parse the csv into the jsonObjects. Now i want to make that jsonObject available when i load the client side 'HTML' so i can grab that object. Is that possible? – timJIN520 Nov 08 '17 at 22:18
  • I'm more confused now, so, all you need is to pass the JSON from the server to the client? You should rewrite your question. Have you tried using: https://stackoverflow.com/questions/34149101/how-to-pass-an-object-to-client-javascript-nodejs-express – Ryan Morton Nov 08 '17 at 23:11
  • @RyanMorton Yes. However, my json array is going to have over 100K plus of rows. On the client side, if i do var getData = <%- JSON.stringify(JsonObject) %>, the view source will display all the items and i dont want to overfill the view source since its going to be 100K data. Is there a way to just return an object only? I hope that makes since. – timJIN520 Nov 09 '17 at 16:52
  • It does not make sense. – Ryan Morton Nov 09 '17 at 16:55

0 Answers0