slowness in promise chain

Question

my codes transfers huge number of files from one cloud storage to another cloud storage (same region). The workflow is downloading the source file stream, then uploading the stream to the target storage.

If running them in a non-promise loop, the transferring is fast(say, 100M/s), but will hit the memory limit. Finally the server crashes.

if in a promise chain, i.e. run the next job after the last job completes, the crash problem is solved, but the transferring speed is very slow (say 10M/s).

My question: why the promise would affect the downloading & uploading speed? or anything I missed?

code snippet:

transferArray.forEach(function (eachTransfer) { 
      queue = queue.then(function(result){
        // put result somewhere
        return eachFileTransfer(jobId,userid,eachTransfer);
      });  
    }); 
    queue.then(function(){
      console.log('done');
    });

I am thinking to use PromisePool with concurrences, but not sure how much the speed would be improved, and the reasonable number of concurrences I should set. the ref post: Execute promises concurrently with a buffer pool size in Javascript

Are you running them concurrently or one after the other? One causes `10 * 100M/s * 1` vs `1 * 1000M/s * 10`. Pool size is going to be very situational. — zero298, Apr 10 '18 at 16:58
Without any code we will all be guessing. My guess is that the promise chain is running one at a time, while the for loop is running a bunch of transfers simultaneously — eventually trying to run too many. — Mark, Apr 10 '18 at 16:59
*"I am thinking to use PromisePool with concurrences"* Very likely the answer. Set the concurrency to the number of transfers you want running at the same time (from your question, perhaps 10). — T.J. Crowder, Apr 10 '18 at 17:02
thank you guys! my code snippet is posted. I think you are right the native speed is just like that, but in non-promise mode, all are running together, so i thought the native speed is fast.. i will try with PromisePool — Xiaodong Liang, Apr 10 '18 at 17:05
Have a look at https://stackoverflow.com/a/38778887/1048572 and https://stackoverflow.com/a/39197252/1048572 — Bergi, Apr 10 '18 at 17:33

score 0 · Answer 1 · answered Apr 10 '18 at 17:13

0

Cause i guess with your loop you run a lot of requests in parallel, wich will be faster (but will also exceed all kind of limits). Therefore just as you said, use multiple promise chains:

  const inParallel = 10;
  const promises = (new Array(inParallel)).fill(Promise.resolve());

  for(const [index, transfer] of transferArray.entries())
    promises[index % inParallel] = promises[index % inParallel].then(() =>  eachFileTransfer(jobId,userid,eachTransfer));

answered Apr 10 '18 at 17:13

Jonas Wilms

132,000
20
149
151

1

This assumes that all transfers take the same time. If not, you'll see a significant slowdown towards the end when only the "slower" chains will still run. – Bergi Apr 10 '18 at 17:34
yes, it seems still slowness, and only one transferring is running one time. but anyway, thank you Jonas W. probably my usage has problem. will check and also try with what Bergi recommended. – Xiaodong Liang Apr 11 '18 at 03:57
@xiadong just go with the PromisePool module, its exactly what bergi recommends – Jonas Wilms Apr 11 '18 at 11:52

score 0 · Accepted Answer · answered Apr 12 '18 at 11:23

finally, I made it working by PromisePool. The code snippet below is based on the other post: Execute promises concurrently with a buffer pool size in Javascript

Since my code is relatively complex than this post, thought it might be helpful for some others:

var PromisePool = require('es6-promise-pool'); 

//single promises
function p1(){
  return new Promise(function(resolve, reject) {
      console.log('p1');  
      setTimeout(resolve, 2000, 'foo');
 });
}
function p2(){
  return new Promise(function(resolve, reject) {
      console.log('p2');  
      setTimeout(resolve, 2000, 'foo');
  });
}
function p3(){
 return new Promise(function(resolve, reject) {
    console.log('p3');  
    setTimeout(resolve, 2000, 'foo');
 });
}


var tasks=[];
var loopIndex = 0;
[1,2,3,4,5,6,7,8].forEach(function(v,i){

   console.log(v,i);

   //build promise chain
   var x = (v) => new Promise(function(resolve, reject) {
      console.log(v);

      p1().then(function(r){
         return p2();
      })
      .then(function(r){
        return p3();
      })
      .then(function(r){
        //console.log('embedded chain done');
        resolve('embedded chain done');
       }).catch(function(e){
        reject('embedded chain failed');
     }); 
    //setTimeout(resolve, 2000, 'foo');
  })

  //build one promise task
  tasks.push({fun:x,param:i});

  if(++loopIndex == 8){
     //once the array is done
    const promiseProducer = () => {
        while(tasks.length) {
           console.log('processing ');
           const task = tasks.shift();
           return task.fun(task.param);  
        } 
        return null;
     }

      // concurrent Promises set to 3
      const pool = new PromisePool(promiseProducer, 3); 
     //start to promise all, yet with concurrent 3 tasks.
     const poolPromise = pool.start(); 
     poolPromise.then(() => { console.log('done!'); })
    }  

});

slowness in promise chain

2 Answers2