Why are these promise rejections global?

Question

We have a fairly complex code base in NodeJS that runs a lot of Promises synchronously. Some of them come from Firebase (firebase-admin), some from other Google Cloud libraries, some are local MongoDB requests. This code works mostly fine, millions of promises being fulfilled over the course of 5-8 hours.

But sometimes we get promises rejected due to external reasons like network timeouts. For this reason, we have try-catch blocks around all of the Firebase or Google Cloud or MongoDB calls (the calls are awaited, so a rejected promise should be caught be the catch blocks). If a network timeout occurs, we just try it again after a while. This works great most of the time. Sometimes, the whole thing runs through without any real problems.

However, sometimes we still get unhandled promises being rejected, which then appear in the process.on('unhandledRejection', ...). The stack traces of these rejections look like this, for example:

Warn: Unhandled Rejection at: Promise [object Promise] reason: Error stack: Error: 
    at new ApiError ([repo-path]\node_modules\@google-cloud\common\build\src\util.js:59:15)
    at Util.parseHttpRespBody ([repo-path]\node_modules\@google-cloud\common\build\src\util.js:194:38)
    at Util.handleResp ([repo-path]\node_modules\@google-cloud\common\build\src\util.js:135:117)
    at [repo-path]\node_modules\@google-cloud\common\build\src\util.js:434:22
    at onResponse ([repo-path]\node_modules\retry-request\index.js:214:7)
    at [repo-path]\node_modules\teeny-request\src\index.ts:325:11
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

This is a stacktrace which is completely detached from my own code, so I have absolutely no idea where I could improve my code to make it more robust against errors (error message seems to be very helpful too).

Another example:

Warn: Unhandled Rejection at: Promise [object Promise] reason: MongoError: server instance pool was destroyed stack: MongoError: server instance pool was destroyed
    at basicWriteValidations ([repo-path]\node_modules\mongodb\lib\core\topologies\server.js:574:41)
    at Server.insert ([repo-path]\node_modules\mongodb\lib\core\topologies\server.js:688:16)
    at Server.insert ([repo-path]\node_modules\mongodb\lib\topologies\topology_base.js:301:25)
    at OrderedBulkOperation.finalOptionsHandler ([repo-path]\node_modules\mongodb\lib\bulk\common.js:1210:25)
    at executeCommands ([repo-path]\node_modules\mongodb\lib\bulk\common.js:527:17)
    at executeLegacyOperation ([repo-path]\node_modules\mongodb\lib\utils.js:390:24)
    at OrderedBulkOperation.execute ([repo-path]\node_modules\mongodb\lib\bulk\common.js:1146:12)
    at BulkWriteOperation.execute ([repo-path]\node_modules\mongodb\lib\operations\bulk_write.js:67:10)
    at InsertManyOperation.execute ([repo-path]\node_modules\mongodb\lib\operations\insert_many.js:41:24)
    at executeOperation ([repo-path]\node_modules\mongodb\lib\operations\execute_operation.js:77:17)

At least this error message says something.

All my Google Cloud or MongoDB calls have await and try-catch blocks around them (and the MongoDB reference is recreated in the catch block), so if the promise were rejected inside those calls, the error would be caught in the catch block.

A similar problem sometimes happens in the Firebase library. Some of the rejected promises (e.g. because of network errors) get caught by our try-catch blocks, but some don't, and I have no possibility to improve my code, because there is no stack trace in that case.

Now, regardless of the specific causes of these problems: I find it very frustrating that the errors just happen on a global scale (process.on('unhandledRejection', ...), instead of at a location in my code where I can handle them with a try-catch. This makes us lose so much time, because we have to restart the whole process when we get into such a state.

How can I improve my code such that these global exceptions do not happen again? Why are these errors global unhandled rejections when I have try-catch blocks around all the promises?

It might be the case that these are the problems of the MongoDB / Firebase clients: however, more than one library is affected by this behavior, so I'm not sure.

Answer 1

a stacktrace which is completely detached from my own code

Yes, but does the function you call have proper error handling for what IT does?
Below I show a simple example of why your outside code with try/catch can simply not prevent promise rejections

//if a function you don't control causes an error with the language itself, yikes

//and for rejections, the same(amount of YIKES) can happen if an asynchronous function you call doesn't send up its rejection properly
//the example below is if the function is returning a custom promise that faces a problem, then does `throw err` instead of `reject(err)`)

//however, there usually is some thiAPI.on('error',callback) but try/catch doesn't solve everything
async function someFireBaseThing(){
  //a promise is always returned from an async function(on error it does the equivalent of `Promise.reject(error)`)
  //yet if you return a promise, THAT would be the promise returned and catch will only catch a `Promise.reject(theError)`
  
  return await new Promise((r,j)=>{
    fetch('x').then(r).catch(e=>{throw e})
    //unhandled rejection occurs even though e gets thrown
    //ironically, this could be simply solved with `.catch(j)`
    //check inspect element console since stackoverflow console doesn't show the error
  })
}
async function yourCode(){
  try{console.log(await someFireBaseThing())}
  catch(e){console.warn("successful handle:",e)}
}
yourCode()

Upon reading your question once more, it looks like you can just set a time limit for a task and then manually throw to your waiting catch if it takes too long(because if the error stack doesn't include your code, the promise that gets shown to unhandledRejection would probably be unseen by your code in the first place)

function handler(promise,time){ //automatically rejects if it takes too long
  return new Promise(async(r,j)=>{
    setTimeout(()=>j('promise did not resolve in given time'),time)
    try{r(await promise)} catch(err){j(err)}
  })
}
async function yourCode(){
  while(true){ //will break when promise is successful(and returns)
    try{return await handler(someFireBaseThing(...someArguments),1e4)}
    catch(err){yourHandlingOn(err)}
  }
}

Answer 2

Elaborating on my comment, here's what I would bet is going on: You set up some sort base instance to interact with the API, then use that instance moving forward in your calls. That base instance is likely an event emitter that itself can emit an 'error' event, which is a fatal unhandled error with no 'error' listener setup.

I'll use postgres for an example since I'm unfamiliar with firebase or mongo.

// Pool is a pool of connections to the DB
const pool = new (require('pg')).Pool(...);

// Using pool we call an async function in a try catch
try {
  await pool.query('select foo from bar where id = $1', [92]);
}
catch(err) {
  // A SQL error like no table named bar would be caught here.
  // However a connection error would be emitted as an 'error'
  // event from pool itself, which would be unhandled
}

The solution in the example would be to start with

const pool = new (require('pg')).Pool(...);
pool.on('error', (err) => { /* do whatever with error */ })