Synchronize multiple Promises while allowing multiple number of retries

Question

I am trying to build a downloader that automatically retries downloading. Basically, a task queue which retries tasks for a certain number of times. I first tried using Promise.all() but the "trick" to circumvent the fail-on-first-reject described here did not help (and is an anti-pattern as described further down in that thread)

So I got a version working which seems to somewhat do what I want. At least the results it prints are correct. But it still throws several uncaught exception test X errors/warnings and I don't know what to do about that.

The Code:

asd = async () => {

  // Function simulating tasks which might fail.
  function wait(ms, data) {
    return new Promise( (resolve, reject) => setTimeout(() => {
      if (Math.random() > 0.5){
        resolve(data);
      } else {
        reject(data);
      }
    }, ms) );
  }

  let tasks = [];
  const results = [];

  // start the tasks
  for ( let i = 0; i < 20; i++) {
    const prom = wait(100 * i, 'test ' + i);
    tasks.push([i, prom]);
  }

  // collect results and handle retries.
  for ( let tries = 0; tries < 10; tries++){
    failedTasks = [];
    for ( let i = 0; i < tasks.length; i++) {

      const task_idx = tasks[i][0];

      // Wait for the task and check whether they failed or not.
      // Any pointers on how to improve the readability of the next 6 lines appreciated.
      await tasks[i][1].then(result => {
        results.push([task_idx, result])
      }).catch(err => {
        const prom = wait(100 * task_idx, 'test ' + task_idx);
        failedTasks.push([task_idx, prom])
      });
    }

    // Retry the tasks which failed.
    if (failedTasks.length === 0){
      break;
    } else {
      tasks = failedTasks;
    }
    console.log('try ', tries);
  }

  console.log(results);
}

In the end, the results array contains (unless a task failed 10 times) all the results. But still uncaught exceptions fly around.

As not all rejected promises result in uncaught exceptions, my suspicion is, that starting the tasks first and applying then()/catch() later is causing some timing issues here.

Any improvements or better solutions to my problems are appreciated. E.g. my solution only allows retries "in waves". If anyone comes up with a better continuous solution, that would be much appreciated as well.

As you have assumed correctly, you have timing issue. The restarted task that you push to `failedTasks` could fail again before you exit the inner loop and do `tasks = failedTasks` and actually you don't have a `.catch` callback attached to that Promise until your reach it again in the inner loop, so its very likely that you get a unhandled rejection error. — t.niese, Aug 30 '18 at 13:02
"*and is an anti-pattern as described further down in that thread*" - no, not at all. See my comments on that answer. — Bergi, Aug 30 '18 at 13:25
@Lniese I seem to catch the exception by adding the `.catch()` later though. Which makes sense but is also confusing. But I guess the interpreter can not know whether I catch an error later and has to throw at some point. I was wondering whether there was an option to say "I will catch this exception later, don't worry" @Bergi thanks for the clarification. I actually ran into the issue you warned against. Well thanks to the accepted answer I have a more sane code structure now. — David Schumann, Aug 31 '18 at 06:49

t.niese · Accepted Answer · 2018-08-30T12:52:54.040

Using await and asnyc allows to solve that in a much clearer way.

You pass an array of tasks (functions that when executed start the given task) to the execute_tasks. This function will call for each of those tasks the execute_task, passing the task function to it, the execute_task will return a Promise containing the information if the task was successful or not.

The execute_task as a loop that loops until the async task was successful or the maximum number of retries reached.

Because each of the tasks has its own retry loop you can avoid those waves. Each task will queue itself for a new execution as it fails. Using await this way creates some kind of cooperative multitasking. And all errors are handled because the task is executed in a try catch block.

function wait(ms, data) {
  return new Promise((resolve, reject) => setTimeout(() => {
    if (Math.random() > 0.5) {
      resolve(data);
    } else {
      reject(new Error());
    }
  }, ms));
}


async function execute_task(task) {
  let result, lastError;
  let i = 0
  
  //loop until result was found or the retry count is larger then 10
  while (!result && i < 10) {
    try {
      result = await task()
    } catch (err) {
      lastError = err
      // maybe sleep/wait before retry
    }
    i++
  }

  if (result) {
    return { success: true, data: result }
  } else {
    return { success: false, err: lastError }
  }
}

async function execute_tasks(taskList) {
  var taskPromises = taskList.map(task => execute_task(task))
 
  // the result could be sorted into failed and not failed task before returning
  return await Promise.all(taskPromises)
}


var taskList = []
for (let i = 0; i < 10; i++) {
  taskList.push(() => {
    return wait(500, {
      foo: i
    })
  })
}

execute_tasks(taskList)
  .then(result => {
    console.dir(result)
  })

Synchronize multiple Promises while allowing multiple number of retries

1 Answers1