How to prevent "Given transaction number 1 does not match any in-progress transactions" with Mongoose Transactions?

Question

I am using Mongoose to access to my database. I need to use transactions to make an atomic insert-update. 95% of the time my transaction works fine, but 5% of the time an error is showing :

"Given transaction number 1 does not match any in-progress transactions"

It's very difficult to reproduce this error, so I really want to understand where it is coming from to get rid of it. I could not find a very clear explanation about this type of behaviour.

I have tried to use async/await key words on various functions. I don't know if an operation is not done in time or too soon.

Here the code I am using:

export const createMany = async function (req, res, next) {
  if (!isIterable(req.body)) {
    res.status(400).send('Wrong format of body')
    return
  }
  if (req.body.length === 0) {
    res.status(400).send('The body is well formed (an array) but empty')
    return
  }

  const session = await mongoose.startSession()
  session.startTransaction()
  try {
    const packageBundle = await Package.create(req.body, { session })
    const options = []
    for (const key in packageBundle) {
      if (Object.prototype.hasOwnProperty.call(packageBundle, key)) {
        options.push({
          updateOne: {
            filter: { _id: packageBundle[key].id },
            update: {
              $set: {
                custom_id_string: 'CAB' + packageBundle[key].custom_id.toLocaleString('en-US', {
                  minimumIntegerDigits: 14,
                  useGrouping: false
                })
              },
              upsert: true
            }
          }
        })
      }
    }
    await Package.bulkWrite(
      options,
      { session }
    )
    for (const key in packageBundle) {
      if (Object.prototype.hasOwnProperty.call(packageBundle, key)) {
        packageBundle[key].custom_id_string = 'CAB' + packageBundle[key].custom_id.toLocaleString('en-US', {
          minimumIntegerDigits: 14,
          useGrouping: false
        })
      }
    }
    res.status(201).json(packageBundle)
    await session.commitTransaction()
  } catch (error) {
    res.status(500).end()
    await session.abortTransaction()
    throw error
  } finally {
    session.endSession()
  }
}

I expect my code to add in the database and to update the entry packages in atomic way, that there is no instable database status. This is working perfectly for the main part, but I need to be sure that this bug is not showing anymore.

Answer 1

You should use the session.withTransaction() helper function to perform the transaction, as pointed in mongoose documentation. This will take care of starting, committing and retrying the transaction in case it fails.

const session = await mongoose.startSession();
await session.withTransaction(async () => {
    // Your transaction methods
});

---

Explanation:

The multi-document transactions in MongoDB are relatively new and might be a bit unstable in some cases, such as described here. And certainly, it has also been reported in Mongoose here. Your error most probably is a TransientTransactionError due to a write-conflict happening when the transaction is committed.

However, this is a known and expected issue from MongoDB and these comments explain their reasoning behind why they decided it to be like this. Moreover, they claim that the user should be handling the cases of write conflicts and retrying the transaction if that happens.

Therefore, looking at your code, the Package.create(...) method seems to be the reason why the error gets triggered, since this method is executing a save() for every document in the array (from mongoose docs).

A quick solution might be using Package.insertMany(...) instead of create(), since the Model.insertMany() "only sends one operation to the server, rather than one for each document" (from mongoose docs).

However, MongoDB provides a helper function session.withTransaction() that will take care of starting and committing the transaction and retry it in case of any error, since release v3.2.1. Hence, this should be your preferred way to work with transactions in a safer way; which is, of course, available in Mongoose through the Node.js API.

Answer 2

The accepted answer is great. In my case, I was running multiple transactions serially within a session. I was still facing this issue every now and then. I wrote a small helper to resolve this.

File 1:

// do some work here
await session.withTransaction(() => {});

// ensure the earlier transaction is completed
await ensureTransactionCompletion(session);

// do some more work here
await session.withTransaction(() => {});

Utils File:

async ensureTransactionCompletion(session: ClientSession, maxRetryCount: number = 50) {
        // When we are trying to split our operations into multiple transactions
        // Sometimes we are getting an error that the earlier transaction is still in progress
        // To avoid that, we ensure the earlier transaction has finished
        let count = 0;
        while (session.inTransaction()) {
            if (count >= maxRetryCount) {
                break;
            }
            // Adding a delay so that the transaction get be committed
            await new Promise(r => setTimeout(r, 100));
            count++;
        }
    }