how to work with a large array in javascript

Question

In my application I have a very big array (arround 60k records). Using a for loop I am doing some operations on it as shown below.

var allPoints = [];
for (var i = 0, cLength = this._clusterData.length; i < cLength; i+=1) {
    if (allPoints.indexOf(this._clusterData[i].attributes.PropertyAddress) == -1) {
        allPoints.push(this._clusterData[i].attributes.PropertyAddress);
        this._DistClusterData.push(this._clusterData[i])
    }
}

When I run this loop the browser hangs as it is very big & in Firefox is shows popup saying "A script on this page may be busy, or it may have stopped responding. You can stop the script now, or you can continue to see if the script will complete". What can I do so that browser do not hang?

What does the data consist of, what's your goal after running this loop? — StudioTime, Aug 02 '16 at 13:49
Can you decrease amount of data by adding some pagination or something like this. Browser hangs because it's trying to process all the data and all operations wait until it finishes. I suggest you to add pagination or do some data normalisation to make this faster. — Mykola Borysyuk, Aug 02 '16 at 13:49
It is a wrong approach, Your browser will be definitely crashed. You can try to split your task in 2 or 3 different tasks. It may be solve your problem — Sunil Kumar, Aug 02 '16 at 13:53

deceze · Answer 1 · 2016-08-02T13:57:18.680

You need to return control back to the browser in order to keep it responsive. That means you need to use setTimeout to end your current processing and schedule it for resumption sometime later. E.g.:

function processData(i) {
    var data = clusterData[i];
    ...

    if (i < clusterData.length) {
        setTimeout(processData, 0, i + 1);
    }
}

processData(0);

This would be the simplest thing to do from where you currently are.

Alternatively, if it fits what you want to do, Web Workers would be a great solution, since they actually shunt the work into a separate thread.

Having said this, what you're currently doing is extremely inefficient. You push values into an array, and consequently keep checking the ever longer array over and over for the values it contains. You should be using object keys for the purpose of de-duplication instead:

var allPoints = {};

// for (...) ...
if (!allPoints[address]) {  // you can even omit this entirely
    allPoints[address] = true;
}

// later:
allPoints = allPoints.keys();

score 0 · Answer 2 · answered Aug 02 '16 at 13:53

First of all, avoid the multiple this._clusterData[i] calls. Extract it to a variable like so:

var allPoints = [];
var current;
for (var i = 0, cLength = this._clusterData.length; i < cLength; i+=1) {
    current = this._clusterData[i];
    if (allPoints.indexOf(current.attributes.PropertyAddress) == -1) {
        allPoints.push(current.attributes.PropertyAddress);
        this._DistClusterData.push(current)
    }
}

This should boost your performance quite a bit :-)

trincot · Answer 3 · 2016-08-02T14:29:17.843

As others already pointed out, you can do this asynchronously, so the browser remains responsive.

It should be noted however that the indexOf operation you do can become very costly. It would be better if you would create a Map keyed by the PropertyAddress value. That will take care of the duplicates.

(function (clusterData, batchSize, done) {
    var myMap = new Map();
    var i = 0;
    (function nextBatch() {
        for (data of clusterData.slice(i, i+batchSize)) {
            myMap.set(data.attributes.PropertyAddress, data);
        }    
        i += batchSize;
        if (i < clusterData.length) {
            setTimeout(nextBatch, 0);
        } else {
            done(myMap);
        }
    })();
})(this._clusterData, 1000, function (result) {
    // All done
    this._DistClusterData = result;
    // continue here with other stuff you want to do with it.
}.bind(this));

score -1 · Answer 4 · edited May 23 '17 at 12:14

-1

Try considering adding to the array asynchronously with a list, for a set of 1000 records at a time, or for what provides the best performance. This should free up your application while a set of items is added to a list.

Here is some additional information: async and await while adding elements to List<T>

edited May 23 '17 at 12:14

Community

1
1

answered Aug 02 '16 at 13:53

JDavila

409
1
7
22

how to work with a large array in javascript

4 Answers4