How to split the process of looping through a huge array?

Question

I have an array with 1000 items that are complex objects.

I need to loop through the array to run some function, when the array is small like 100 products, the performance seems to be alright.

But when it reaches 400 or above products, the performance goes down significantly on the browser.

So I wanted to do something like first loop through top 100 products, then continue loop the next 100 products subsequently.

Not sure is that a good idea? Is there any better way to handle looping through a huge array?

Code Example

The actual case is to loop through selected 1000 of checkboxes.

I have a function to append li dynamically to a DOM.

populateEnrichedProductList = (id, name) => {
        let html =
                `<li class="list-group-item product-item-name" productName="${name}" productId="${id}" data-desc="" data-ingredient="" data-allergens="">` +
                `${name}` +
                `<span class="error"><i class="fas fa-exclamation-triangle"></i></span>` +
                '<div class="spinner">' +
                '<div class="spinner-border text-warning" role="status">' +
                '<span class="sr-only">Loading...</span>' +
                '</div >' +
                '</div>' +
                '</li >';

                $('#selectedProductList').removeClass('hidden');    
                $('#productListUL').addClass('hidden');
                $('#selectedProductList').append(html);
    };

Here I loop through the checked items, and run the function populateEnrichedProductList for each item of the array.

let selectedProducts = '#selectProductForm input:checked';
$(selectedProducts).each(function(){
            let pid   = $(this).attr('id'),
                pname = $(this).attr('name');

            populateEnrichedProductList(pid, pname);
        });

As mentioned, there were no issues when array is small, but performance dropped when array has more data.

It would be great if anyone can show an example of better handling it to improve the performance.

That's certainly one solution, however it's generally a better idea to filter the dataset when retrieving it from the server. Then you reduce the issue of weight of data on the client. Note that without seeing your actual code all we can give you are theoretical solutions, based on assumptions. If you want a workable solution we need to see the code and also an example of the data. — Rory McCrossan, Oct 09 '19 at 10:44
try this : https://stackoverflow.com/questions/34169702/looping-through-a-huge-array-in-javascript — Ankur Dubey, Oct 09 '19 at 10:44
If the length of the operation is not a huge issue, then you can space out the executions over the event queue via `setTimeout`. — VLAZ, Oct 09 '19 at 10:46
@RoryMcCrossan thanks for the suggestion. I have added some example code. Please take a look. — jenna_3108, Oct 09 '19 at 11:02
OK, first point of order - *do not generate HTML*. That will be very slow since it uses the HTML parsing for *each element*. Second *don't add the elements one by one*. You're probably triggering [reflow](https://stackoverflow.com/questions/27637184/what-is-dom-reflow) multiple times which is also an expensive operation. Generate the DOM elements programmatically and add them all at once if possible. You might be able to use a document fragment. This should speed up the operation significantly. — VLAZ, Oct 09 '19 at 11:37
As a separate note, if you have template literals, you don't need to concatenate them on each line. Template literals are already multiline. — VLAZ, Oct 09 '19 at 11:40
Thank you @VLAZ! I just refactored my code based on your comments and I can see the performance has improved very much! — jenna_3108, Oct 10 '19 at 08:44

score 1 · Answer 1 · answered Oct 09 '19 at 10:47

You can split your array for parts. Look this code. This is example with crypto to elements of array, but you can use this pattern with your logic.

const crypto = require('crypto')

const arr = new Array(200).fill('something')
function processChunk() {
  if (arr.length === 0) {
    // code executed after processing the entire array
  } else {
    console.log('processing chunk');
    // select 10 elements and remove them from the array
    const subarr = arr.splice(0, 10)
    for (const item of subarr) {
    // perform complex processing of each of the elements
      doHeavyStuff(item)
    }
    // put the function in the queue
    setImmediate(processChunk)
  }
}

processChunk()

function doHeavyStuff(item) {
  crypto.createHmac('sha256', 'secret').update(new Array(10000).fill(item).join('.')).digest('hex')
}

// This fragment is needed only to confirm that, processing a large array,
// We give the opportunity to execute another code.

let interval = setInterval(() => {
  console.log('tick!')
  if (arr.length === 0) clearInterval(interval)
}, 0)

How to split the process of looping through a huge array?

1 Answers1