Chaining a lot of (>40,000) native promises together eats up too much memory

Question

I have a project where I need to write a function to calculate several things sequentially and then write the results to a SQL DB. Unfortunately, I need to repeat this over 40,000 times. I use node.js and promises to accomplish this but the memory usage goes to almost 2GB a lot of times the program just dies after 10,000 or so calculations. I developed my own native promiseEach function which takes a array items sequantially and chains it with promises.

What Am I doing wrong here?:

function promiseEach(array,promiseFn){
    return new Promise(function(resolve,reject){
        try {
            var promiseArray = [];
            var json = { array: array, counter: 0 }
            for (var i = 0; i < array.length; i++) { 
                promiseArray.push(promiseFn) 
            }
            promiseArray.reduce(function(preFn,curFn,index,pArray){
                return  preFn
                .then( function(z){ return json.array[json.counter++] })
                .then(curFn)
            }, 
            Promise.resolve(json.array[json.counter]))
            .then(resolve,reject)

        }catch(err){
            console.log("promiseEach ERROR:");
            reject(err)
        }
    })
}

Answer 1

Well, you're overcomplicating the function for a start - from what I can tell, you're calling promiseFn for each of the content of array, in sequence

The following code is identical in function to your code

function promiseEach(array, promiseFn) {
    return array.reduce(function(prev, item) {
        return prev.then(function() {
            return promiseFn(item);
        });
    }, Promise.resolve());
}

No guarantee that this will fix the actual issue though

for completeness, as you are coding in nodejs, the same code as written in ES2015+

let promiseEach = (array, promiseFn) => 
    array.reduce((prev, item) => 
        prev.then(() => 
            promiseFn(item)
        )
    , Promise.resolve());

Answer 2

Two lines in the posted code

    .then( function(z){ return json.array[json.counter++] })
    then(curFn)

seem to indicate promseFn is to be called with a new parameter after the operation performed by a previous call to it completes, and that the fulfilled value of the previous call is not made use of in the promise chain.

A suggestion to avoid creating (39,999 or so) intermediate chained promises before returning from the each function, is to use intermediate promises returned by promiseFn to call a promise factory function when they become fulfilled, and return a final promise which is only fulfilled by the last intermediate promise.

The concept code which follows does not contain a call to reduce because no reduce operation is performed:

function promiseEach( array, promiseFn) {
    var resolve, reject;
    var counter = 0;
    var final = new Promise( function( r, j) { resolve = r; reject = j});
    function nextPromise() {
        promiseFn( array[counter++])
        .then( (counter < array.length ? nextPromise : resolve), reject);
    }
    if( array.length) {
        nextPromise();
    }
    else {
        reject( new Error("promiseEach called on empty array"));
    }
    return final;
}

Added Notes:

The promiseEach function above was tested in node/js using both a synchronous and asynchronous promiseFn with no concurrent updates to the data array.

• Synchronous Test

  function promiseSynch( z) {
      return new Promise( function(resolve,reject){resolve(z);});
  }
  

was able to process an array with 1 million (1000,000) entries) in about 10 seconds using a Win7 i86 notebook with 1GB memory and a browser, editor and task manager open at the same time. Memory usage was flat at around 80% of available memory.

• Asynchronous Test

  function promiseAsynch( z) {
      var resolve;
      function delayResolve() {
          resolve( z);
      }
      return new Promise( ( r, j)=>{
          resolve = r;
          setTimeout(delayResolve,1);
      });
  }
  

managed an array of 100,000 entries in a little over 20 minutes on the same notebook, again with flat memory usage of around 80%.

Conclusion

The testing suggests that Promises are not causing memory leaks simply because they are being used, and that they should be capable of handling passing lengthy array data sequentially into a promise returning function.

This implies there is some other causes of memory allocation failure, or mysterious processing outages, which needs to be found before the problem can be fully solved. As a suggestion you might start with a search for memory leak issues related to the DB library being used.