AngularJS and big JSON data

Question

For a custom AngularJS application that uses data from an API I've been creating I've come across the use of Angular oboe. Oboe is a bower package that helps streaming large JSON files to the view. So after some trial and error, I've managed to build a decent oboe GET method that obtains around 4000 JSON items in around 2 seconds. But my problem starts when adding more GET methods to the same view.

At first there weren't any problems but eventually, the loading time keeps getting bigger and bigger. So I've tried using the Oboe Cached: true configuration. Sadly it doesn't work at all. Every time I load the page all the data is loaded again instead of obtained from the browser Cache

In my example below, you're able to see the structure of one of my oboe function's that I've been trying to cache. A JSfiddle link is added down below as well.

The Function and the view

 function createProduct(id, name) {
        this.id = id;
        this.name = name;
    }

    $scope.products = [];

    oboe({
        url: 'config/get/getProducts.php',
        method: 'GET',
        cached: true
    }).path('products.product.*', function () {
        // we don't have the person's details yet but we know we
        // found someone in the json stream. We can eagerly put
        // their div to the page and then fill it with whatever
        // other data we find:
    }).start(function () {
        console.log("start");
    }).node('products.product.*', function (products) {
        // we just found out their name, lets add it
            // to their div:
            $scope.products.push({
                id: products.id,
                name: products.name.language
            });
            $scope.totalItems = $scope.products.length;
            return new createProduct(products.id, products.name);
    }).done(function () {
        console.log( $scope.products );
    });

    // Refresh data
    $scope.refreshData = function() {
        cartService.refreshData()
            .then(function(response) {
                $scope.cartItems = response.cartItems;
                $scope.totalCartItems = response;
                $scope.selectedCustomer = response;
            })
    };

<script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.2.23/angular.min.js"></script>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<div class="productimg col-lg-4 col-md-4" ng-repeat="product in products | limitTo : limit : (currentPage - 1) * limit track by product.id"
             ng-class="{lastItem: $last}" scroll-bottom="event">
            <div class="row">
                <div class="col-md-12" ng-bind="product.id"></div>
                <div class="col-md-12">
                    <a ng-bind="product.name" href="{{product.id}}.nl"></a>
                </div>
            </div>
        </div>
        <div class="col-lg-12 text-center margin-t-30">
            <ul uib-pagination
                total-items="totalItems"
                ng-model="currentPage"
                items-per-page="limit">
            </ul>
        </div>

In the JSfiddle You're able to see the code. I couldn't get the JSON to work on on JSfiddle but see it as the following line but then around 10000 "product" rows.

{"products":{"product":[{"id":"1240","id_manufacturer":"0","id_supplier":"0","id_category_default":"2","id_tax_rules_group":"8","quantity":"0","id_shop_default":"1","reference":{},"ean13":"0","price":"0.000000","active":"0","date_add":"2014-07-15 12:06:34","date_upd":"2018-04-21 12:22:37","name":{"language":"zie voorschot factuur 03"}},{"id":"1241","id_manufacturer":"0","id_supplier":"0","id_category_default":"2","id_tax_rules_group":"8","quantity":"0","id_shop_default":"1","reference":{},"ean13":"0","price":"0.000000","active":"0","date_add":"2014-07-15 12:06:41","date_upd":"2018-04-21 12:22:37","name":{"language":"zie voorschot factuur 04"}},{"id":"8908","id_manufacturer":"0","id_supplier":"15","id_category_default":"2","id_tax_rules_group":"8","quantity":"0","id_shop_default":"1","reference":"041002","ean13":"5712084210057","price":"45.454545","active":"1","date_add":"2015-11-12 18:03:47","date_upd":"2017-11-18 09:57:27","name":{"language":"Vaavud Sleipnir smartphone wind meter"}}}}

So the real struggle I'm facing is that getting the data from the network tab takes around ten seconds. (there is an API request at "getProducts.php"). Then parsing that to the view costs around 30 seconds. (way too long). Secondly, I would like to cache the getProducts request so that the products are directly obtained the next time the view is loaded. With a normal $http.get() and cache: true. It worked but then I'm still facing the slow binding, even with Oboe.

If there is any more information needed please let me know in the comments below.

As always, thanks in advance!

Sending all products in one go is more like a poor design in itself. You should be using pagination as well as passing only data needed. — Tarun Lalwani, May 18 '18 at 07:43
@TarunLalwani First of all it is an HTTP request over an REST API. So i do need to call the full data in one request. Secondly iknow that it isn't the best way but it is needed for searching through all the items. This is a one-page application wich means that no refreshes should be called. i tried using the pagination but i need to find a middle solution to say — Deathstorm, May 18 '18 at 07:48
How often do the products change? Because one small change would mean an invalidated cache and the approach would become useless — Tarun Lalwani, May 18 '18 at 08:21
@TarunLalwani Not that often, maybe twice a day, or more often if there are new products added — Deathstorm, May 18 '18 at 08:29
I cannot help with other things but you shouldn't directly push to scope variable every time. Create a regular array push in to it and once the stream is done, assign it to the scope array. In case of 10000 item it should really help with the binding. — Shyam Babu, May 18 '18 at 08:52
I would limit the api call to a 100 or so entries and with scrolling continuously load the next items (you should have some sort of id to reference in your sql statement). For search I would adapt the existing endpoint that without search it will look for items that match "" and for the search it will load only those who match the string. I hope that makes sense. — Domenik Reitzner, May 18 '18 at 09:00
yes i did. And i am using it already right now. Thanks for the tip! @ParshwaShah — Deathstorm, May 24 '18 at 07:41

score 4 · Accepted Answer · answered May 18 '18 at 07:33

We had a project like this which had too much API with heavy object data. Some of tips we used are these:

1-Arrays instead of Objects

With a bit searching and reading about these data structures, you'll find that the memory consumed by objects is more than the memory used by Arrays. So convert your Array of objects to array of arrays in server side.

2-Use one more level cache

Store data in a server side cache, something like MongoDb and Redis. Handle requests with these steps:

Check the browser cache. If data is available and up-to-date, use this data.
If data is not available in browser, send request to server. Check the server side cache. If data is available and up-to-date, Send this data to client and store it in cache and also use it.
If data is not available in server cache, Send request to API and get data, Store it in server cache, and then sent it back to client and store it in cache and also use it.

3-Send minimum data required by client each time

A client may not stay in a page and see all data shown. Prepare minimum data you need for each page and if client request, bring the others.

For your case, instead of 10,000 product, Just prepare 1000 of them for example and if client needs more, prepare more data, append to your client cache and show it.

If all 10,000 products are required by client, prepare them in several requests.

AngularJS and big JSON data

1 Answers1