How to process large array in javascript without blocking UI with chunk processor

Question

I want to process a large array without blocking the UI, but I also need the processed array to reprint some features in a map. I have used the first approach in the most voted answer of this question: Best way to iterate over an array without blocking the UI

The problem is that when the function finish the first chunk and therefore make use of setTimeout, the thread is released and the map only repaint the first chunk_size elements, in this case 100 but I want all 1000 elements to be multiplied by 10.

I tried to use promises but I am missing something because the same problem occurs. In this JSbin you can check what happens: https://jsbin.com/fatijosufu/edit?js,console An array of 1000 elements is created and the processLargeArray function should multiply each element by 10. When the array is printed after calling the function, only the 100 first elements have been processed, this is, only the first chunk of the array.

const array1 = []

for (let i = 0; i < 1000; i++){
  array1[i] = i;
}

function processLargeArray(array) {
    // set this to whatever number of items you can process at once
    var chunk = 100;
    var index = 0;
    function doChunk() {
        var cnt = chunk;
        while (cnt-- && index < array.length) {
            array[index] = array[index] * 10;
            ++index;
        }
        if (index < array.length) {
            // set Timeout for async iteration
            setTimeout(doChunk, 1);
        }
    }    
    doChunk();    
}

processLargeArray(array1);

console.log(array1);

The way I tried to solve the problem with promises, but that is not working: https://jsbin.com/fatijosufu/edit?js,console

const array1 = []

for (let i = 0; i < 1000; i++){
  array1[i] = i;
}

function processLargeArray(array) {
    return new Promise((resolve, reject) => {
      // set this to whatever number of items you can process at once
      var chunk = 100;
      var index = 0;
      function doChunk() {
          var cnt = chunk;
          while (cnt-- && index < array.length) {
              array[index] = array[index] * 10;
              ++index;
          } 
          if (index < array.length) {
              // set Timeout for async iteration
              setTimeout(doChunk, 1);
          }
      }    
      doChunk();    
      resolve(array);
    });
  
}

processLargeArray(array1).then((array) => {
  console.log(array);
});

*"In this JSbin you can check what happens"* can you describe what we are supposed to see exactly and what you would expect it to look like instead? — Constantin Groß, Jul 23 '21 at 07:09

georg · Answer 1 · 2021-07-23T08:23:33.960

A simpler method to keep the UI responsive would be to insert await someTimeout right in the processing loop, thus splitting it into several "micro tasks" and giving the UI a chance to update. Here's a working example:

BIG_ARRAY = Array(1000).fill(0)

function heavy(x) {
    let y = 0;
    for (let i = 0; i < 1e6; i++)
        y += Math.sin(y);
    return x + y;
}


function processAll() {
    for (let x of BIG_ARRAY)
        heavy(x)
}

function pause() {
    return new Promise(r => setTimeout(r, 0))
}

async function processChunks() {
    for (let [n, x] of BIG_ARRAY.entries()) {
        heavy(x)
        if (n % 10 === 0)
            await pause()
    }
}

<button onclick='processAll()'>all</button>
<button onclick='processChunks()'>chunks</button>

<button onclick='document.getElementById("test").innerHTML += "*"'>
test if UI is responsive
</button>

<p>Press 'all' or 'chunks' to start and then press 'test' periodically to test the UI.</p>

<p id="test"></p>

A cleaner, but a bit more complex alternative would be to use real threading, i.e. workers.

The non-blocking UI seems to work but how can I notice processChunks() has finished processing all the array? It returns a promise and when i write: `processChunks().then(() => console.log(BIG_ARRAY))` an array full of 0's is printed. — Javier Brenes, Jul 23 '21 at 09:17
This is because my example doesn't do anything with an array, try it with some useful code. — georg, Jul 23 '21 at 12:39
This works! Thanks a lot! I've been looking for this solution since Web Worker can only help in certain cases — ByteBuddy, Jul 12 '23 at 09:03

How to process large array in javascript without blocking UI with chunk processor

1 Answers1