0

I am updating a list of transactions by saving the transaction into the database list, I do not want to have duplicate entries in the list so I use $addtoset

this is because the request can be fired multiple times and we want to make sure that any changes are idempotent to the database. the only catch now is that we want to only store the latest 20 transactions

this could be done with a $push $sort $slice but I need to make sure duplicate entries are not available. there was a feature request to mongo back in 2015 for this to be added to the $addtoset feature, but they declined this due to 'sets' not being in an order... which is what the $sort function would have been

I thought I could simply append an empty push update to the update object, but from what I understand, each update is potentially threaded and can lead to undesirable edits if the push/slice fires before the $addtoset

right now, the values are an aggregated string with the following formula timestamp:value but I can easily change the structure to an object {ts:timestamp, value:value}

Update: current code, not sure if it will work as intended as each operation maybe independent

          await historyDB
            .updateOne(
              { trxnId: txid },
              {
                $addToSet: {
                  history: {
                    ts: time,
                    bid: bid.value,
                    txid: trxn.txid,
                  }
                },
                $push: {
                  history: {
                    $each: [{ts:-1}],
                    $sort: { ts: 1 },
                    $slice: -10,
                  },
                },
              },
              { upsert: true },
            ).exec();
DIGI Byte
  • 4,225
  • 1
  • 12
  • 20
  • Please provide an example of a document structure before and after the update. Otherwise it is not obvious how your data look like. – user14967413 Apr 19 '22 at 07:32
  • I read it carefully, but wasn't sure if `history` is an array inside history collection. It seems unusal. You don't describe this in the text. The query treats `history` as an array, but it could have been by mistake, what could I know. – user14967413 Apr 19 '22 at 08:30
  • if you have used mongo, you would be familiar with the structure. the $addtoset is working as intended, we just need to limit the total number of entries so the oldest one is gone, using a read => update => set cycle is not valid as the request could be out of date and overwrite another source as we are dealing with sharding clients – DIGI Byte Apr 19 '22 at 09:33

1 Answers1

1

Your query doesn't work, as you are trying to update history multiple times, which is not allowed in simple update document and raises error Updating the path 'history' would create a conflict at 'history'.

You can however subsequently update history field multiple times with aggregation pipeline.

await historyDB.updateOne(
  { trxnId: txid},
  [{
    $set: {
      history: {
        $let: {
          vars: {
            historyObj: {
              ts: time,
              bid: bid.value,
              txid: trxn.txid,
            },
            historySafe: { $ifNull: ["$history", []] }
          },
          in: {
            $cond: {
              if: { $in: ["$$historyObj", "$$historySafe"] },
              then: "$history",
              else: { $concatArrays: [ "$$historySafe", ["$$historyObj"] ] }
            }
          }
        }
      }
    },
  },
  {
    $set: {
      history: {
        $function: {
          body: function(entries) {
            entries.sort((a, b) => a.ts - b.ts);
            return entries;
          },
          args: [{ $ifNull: ["$history", []] }],
          lang: "js"
        }
      }
    },
  },
  {
    $set: {
      history: {
        $slice: [ "$history", -10 ]
      }
    }
  }],
  { upsert: true },
).exec()

As of MongoDB 6.0, the second $set stage, which provides sorting, can be replaced with $sortArray operator (see here).

user14967413
  • 1,264
  • 1
  • 6
  • 12