Buffer.js Error in Node -- JSON too large

Question

I'm receiving this error

buffer.js:495
        throw new Error('"toString()" failed');

I'm trying to parse large amounts of data at once using this function:

exports.markers = function(resp, resp2) {

  this.markersOne = function() {
    var lat, lng = [];
    return resp.map(function(element) {
      lat = element.location_1.coordinates[1];
      lng = element.location_1.coordinates[0];
      return new google.maps.LatLng(lat, lng);
    })
  }

  this.markersTwo = function() {
    return resp2.map(function(element) {
      var lat, lng = [];
      if (element.location_1 != undefined && element.location_1.length > 0) {
        lat = element.location_1.slice(1, -12);
        lng = element.location_1.slice(10, -1);
        return new google.maps.LatLng(lat, lng);
      }
    })
  }

  var result = this.markersOne().concat( this.markersTwo() );
  console.log(result);
  return result;

  }

I have two databases downloaded as JSON files living in the directory.

one is 304 MB and the other is 80MB.

Seeing as how 304 MB is too large, what are other ways I can get around this?

It doesn't look like the functions you gave us is where the issue is. Where are you creating a buffer in your code? — matt, Feb 11 '17 at 21:06
The issue is importing the 304MB JSON file `//var database1 = require('../databases/LA_Crime1')` When I commented it out, it was fine. I know that this is the cause, I don't know how to fix it. — Quesofat, Feb 11 '17 at 21:08
Checkout this [stackoverflow](http://stackoverflow.com/questions/11874096/parse-large-json-file-in-nodejs) post. Seems like you have to read in the file with a read stream rather then just require the file. — matt, Feb 11 '17 at 21:10
Here is a great library to use: https://github.com/dominictarr/JSONStream — matt, Feb 11 '17 at 21:11
Thank you . If you post this as an answer I will accept it as a solution! — Quesofat, Feb 11 '17 at 21:12

score 1 · Accepted Answer · edited May 23 '17 at 12:24

1

Moved my comments to an answer:

Checkout this stackoverflow post. Seems like you have to read in the file with a read stream rather then just require the file.

You could try this library: http://www.github.com/dominictarr/JSONStream

edited May 23 '17 at 12:24

Community

1
1

answered Feb 11 '17 at 21:13

matt

1,680
1
13
16

Buffer.js Error in Node -- JSON too large

1 Answers1