7

I need to display some data from a database to the user. The data is in a json file and has a pretty huge size. The size of json file is roughly around 15MB. I created a service and used the promise api to make a successful request and load the data and show it to user by doing on ng-repeat on a div. Now as you understand the page will show the data only when the file is available and making a get request to fetch a 15MB file takes enormous amout of time. I see in some cases Firefox simple stops loading the file after some time. Now my question is what is the Angular way of doing such a task.

I am thinking of doing something like first showing just a few entries from json file and then on scrolling the rest of the page will be populated by the remaining data but I guess that won;t be possible because when the get request it made, it will first completely download the file and then display data?

Angular provides something called ng-cloak but that's just for flickering avoidance. Is there something like ng-cloak in angular that I can use? Any other Ideas or how to deal with such scenarios or what is the angular way of accomplishing this??

Mike
  • 3,348
  • 11
  • 45
  • 80
  • Any reason why you can't use pagination?? – harishr Apr 20 '14 at 12:34
  • pagination, comes into picture when I have loaded the data. – Mike Apr 20 '14 at 12:47
  • It should be before you load the data, I meant server-side pagination... You get only paged data from server and if needed get more data from server.. Don't load too much data at start.. it will help keep the page responsive.. – harishr Apr 20 '14 at 12:49
  • backend is not in question. All I have is front end and 20mb json file – Mike Apr 20 '14 at 14:04
  • 1
    you dont just have a frontend, angularjs is served from somewhere and a file is served from somewhere. That somewhere is(based on your comments) a web server. – Vlad Gurovich Apr 21 '14 at 21:47
  • Is there any reason you can't just use a [`$http`](https://docs.angularjs.org/api/ng/service/$http) GET request to the JSON file and set the `timeout` to a high number? – James Vickery Jul 31 '17 at 14:17

2 Answers2

7

Based on your requirements of dealing with essentially a huge JSON payload. FWIW you have two options:

  • Your server supports HTTP/JSON streaming:

    create an angular onreadystatechange handler and use a streaming JSON parser like oboe.js

  • Your server DOES NOT support HTTP/JSON streaming

    Do what everyone else is suggesting and provide a paginated access to that payload so that the browser can load it in chunks on demand.

Vlad Gurovich
  • 8,463
  • 2
  • 27
  • 28
  • I am loading json too as local file and it doesn't come from a cross domain server – Mike Apr 20 '14 at 17:41
  • how are you running your angular application? do you run it from a (local) web server like http://localhost:9000 or do you just open an html file using file URI scheme: file://c:/my_html_file.html ? – Vlad Gurovich Apr 20 '14 at 17:44
  • I run it on a localhost. what I meant to say that the json is in the same directory where my js files are. I am not pulling json from another domain. I am running it locally with all files under my control.. – Mike Apr 20 '14 at 18:22
  • 3
    which means that you are running it on a web server which is still subject to one of the two options i provided -- it either supports http streaming or it doesnt. whether you run the server that serves your angular app locally on your development machine or you run it on a remote VPS does not matter. – Vlad Gurovich Apr 20 '14 at 18:43
1

Make your backend deliver the file in chunks, so that your angular frontend can retrieve part of it at a time.

Something like:

/backend/getentries?skip=500&count=100

The have your frontend repeatedly call the backend while counting up skip and append the results you are getting back to whatever you already have in the frontend.

Anders Bornholm
  • 1,326
  • 7
  • 18