4

I am trying to build a wrapper over Notion JS SDK's iteratePaginatedAPI that handles errors as well. I feel particularly lost on how do I catch API errors in such a way that I can actually retry them (aka retry the iteration that failed). Here's my attempt:

async function* queryNotion(listFn, firstPageArgs) {
  try {
    for await (const result of iteratePaginatedAPI(listFn, firstPageArgs)) {
      yield* result
    }
  } catch (error) {
    if (error.code === APIErrorCode.RateLimited) {
      console.log('rate_limited');
      console.log(error);
      sleep(1);
      // How would I retry the last iteration?
    }
  }
}

Coming from the Ruby world, there is a retry in a rescue block. Any help would be appreciated!

linkyndy
  • 17,038
  • 20
  • 114
  • 194

2 Answers2

1

Very interesting problem. The issue is that the exception comes from the for await itself, not from its body, so you cannot catch it there. When the exception hits, loops are over.

Note that the iterator might be done after a rejection/exception, in which case there is nothing you can do except starting a new one.

That said, you can always call Iterator.next() yourself and process the result manually. The next() call of an async iterator will return an object like {value: Promise<any>, done: boolean}, and when running it in a loop, you can await the promise in a try..catch and only exit the loop when done becomes true:

async function* queryNotion(listFn, firstPageArgs) {
  const asyncGenerator = mockIteratePaginatedAPI(listFn, firstPageArgs)
  while (true) {
    const current = asyncGenerator.next()
    if (current.done) {
      break
    }
    try {
      yield* await current.value
    } catch (e) {
      console.log(`got exception: "${e}" - trying again`)
      continue
    }
  }
}

function* mockIteratePaginatedAPI(a, b) {
  for (let i = 0; i < 8; i++) {
    yield new Promise((resolve, reject) => setTimeout(() => [3, 5].includes(i) ? reject(`le error at ${i}`) : resolve([i]), 500))
  }
}

(async function() {
  for await (const n of queryNotion('foo', 'bar')) {
    console.log(n)
  }
})()

If we keep a reference to the generator, we can also put it back into a for async. This might be easier to read, however a for await ...of will call the iterator's return() when exiting the loop early, likely finishing it, in which case, this will not work:

async function* queryNotion(listFn, firstPageArgs) {
  const asyncGenerator = mockIteratePaginatedAPI(listFn, firstPageArgs)
  while (true) {
    try {
      for await (const result of asyncGenerator) {
        yield* result
      }
      break
    } catch (e) {
      console.log('got exception:', e, 'trying again')
    }
  }
}

function* mockIteratePaginatedAPI(a, b) {
  for (let i = 0; i < 8; i++) {
    yield new Promise((resolve, reject) => setTimeout(() => [3, 5].includes(i) ? reject(`le error at ${i}`) : resolve([i]), 500))
  }
}

(async function () {
  for await (const n of queryNotion('foo', 'bar')) {
    console.log(n)
  }
})()
Moritz Ringler
  • 9,772
  • 9
  • 21
  • 34
  • In your second snippet, would `asyncGenerator` retry the last iteration when the `while(true)` loop activates? As far as I understand, `next()` will always be called on it, and without a "rewind" or something the like, it will never retry the same iteration. – linkyndy Feb 15 '23 at 11:30
  • @linkyndy I think that is the same with both snippets, it solely depends on how the `iteratePaginatedAPI` is implemented. In the end, that one is doing the iterating. – Moritz Ringler Feb 15 '23 at 13:26
  • 1
    @linkyndy I think that is the same with both snippets, it solely depends on how the `iteratePaginatedAPI` is implemented. In the end, that one is doing the iterating and resolves or rejects the promises. If it keeps counting up even if a request failed, you might be able to build a new generator that starts at the failed position. – Moritz Ringler Feb 15 '23 at 13:36
  • Right, you are right. `iteratePaginatedAPI` is detailed here: https://github.com/makenotion/notion-sdk-js/blob/main/src/helpers.ts#L44. From what I get, `nextCursor` is lost in case of an error, so would it mean that when I retry inside `while(true)` I will start iterating from the start on the Notion results? – linkyndy Feb 15 '23 at 18:50
  • @linkyndy I think it is not just losing `nextCursor` but the whole iterator is dead when a request fails, as it doesn't yield anything anymore. So while the loop does another round, the generator would just say that it is empty and the function finishes. Looks like you have three options: 1) create your own `iteratePaginatedAPI` which handles failures 2) Somehow keep track of `nextCursor` and create a new iterator starting at the last `nextCursor` on fail 3) Have the `listFn()` you pass to the generator handle failures. My guess is that the last one is easiest. – Moritz Ringler Feb 15 '23 at 19:39
  • Now it all adds up. Thanks for your patience! – linkyndy Feb 15 '23 at 20:26
0

Simply add a continue statement inside your if

async function* queryNotion(listFn, firstPageArgs) {
  try {
    for await (const result of iteratePaginatedAPI(listFn, firstPageArgs)) {
      yield* result
    }
  } catch (error) {
    if (error.code === APIErrorCode.RateLimited) {
      console.log('rate_limited');
      console.log(error);
      await sleep(1);
      continue; // retry the last iteration
    }
  }
}
Michael Rogers
  • 190
  • 1
  • 11