0

I developed an application which can display all my data stored in a JSON file.

My program works perfectly with a small file (under 5MB). But as soon as I want to use a big file (50MB to 1GB), my browser does not want load the data.

I use D3.js library to read the JSON.

How can I read a big JSON file?

Should I use database like MySQL insted?

Luca Kiebel
  • 9,790
  • 7
  • 29
  • 44
Patrick Michel
  • 75
  • 8
  • 1 GB is a lot to load in for a browser – Luca Kiebel Jun 07 '18 at 12:37
  • Related: https://stackoverflow.com/a/42323461/5768908. Also, what you call a *small* file (5MB) is way over the limit I normally tell to my clients: D3 runs on the user's computer, and not everybody will wait several seconds (in a slow/regular connection) for a chart to show up. Finally, with that huge amount of data points, drawing the chart will be a challenge itself. – Gerardo Furtado Jun 07 '18 at 12:37
  • 1
    Is there a good reason not to use a database? Loading a big JSON file seems possible (https://stackoverflow.com/questions/1262376/is-there-a-limit-on-how-much-json-can-hold https://www.ibm.com/support/knowledgecenter/en/SS9H2Y_7.5.0/com.ibm.dp.doc/json_parserlimits.html) but making a request for such a large amount of data in one go will be very time consuming. – OliverRadini Jun 07 '18 at 12:37
  • Indeed, properly chunked, you can get the rendering to work efficiently, but downloading a 1gig file everytime is not workable. If you really don't want to use a database, make it a static site generated at the server. The optimal solutions stays putting the data in a database. – Shilly Jun 07 '18 at 12:44

1 Answers1

0

To answer the first question, you could serve a big file (50MB is big) from a web server. It would be very slow and inefficient to do that.

Which alludes to the second question; should you use a database like MySQL instead? You should probably use a database, and access and serve the parts of that data that you need as it is needed. You could use MySQL, or some other database for this, there are lots of options.

OliverRadini
  • 6,238
  • 1
  • 21
  • 46