Mongoose: Bulk upsert but only update records if they meet certain criteria

Question

I am designing an item inventory system for a website that I am building.

The user's inventory is loaded from a Web API. This information is then processed so that it is more suited to my web app. I am trying to combine all the item records into one MongoDB collection - so other user inventories will be cached in the same place. What I have to deal with is deleting old item records if they are missing from the user's inventory (i.e. they sold it to someone) and also upserting the new items. Please note I have looked through several Stack Overflow questions about bulk upserts but I was unable to find anything about conditional updates.

Each item has two unique identifiers (classId and instanceId) that allow me to look them up (I have to use both IDs to match it) which remain constant. Some information about the item, such as its name, can change and therefore I want to be able to update those records when I fetch new inventory information. I also want new items that my site hasn't seen before to be added to my database.

Once the data returned from the Web API has been processed, it is left in a large array of objects. This means I am able to use bulk writing, however, I am unaware of how to upsert with conditions with multiple records.

Here is part of my item schema:

const ItemSchema = new mongoose.Schema({
    ownerId: {
        type: String,
        required: true
    },
    classId: {
        type: String,
        required: true
    },
    instanceId: {
        type: String,
        required: true
    },
    name: {
        type: String,
        required: true
    }
    // rest of item attributes...
});

User inventories typically contain 600 or more items, with a max count of 2500.

What is the most efficient way of upserting this much data? Thank you

Update:

I have had trouble implementing the solution to the bulk insert problem. I made a few assumptions and I don't know if they were right. I interpreted _ as lodash, response.body as the JSON returned by the API and myListOfItems also as that same array of items.

import Item from "../models/item.model";
import _ from 'lodash';

async function storeInventory(items) {
    let bulkUpdate = Item.collection.initializeUnorderedBulkOp();

    _.forEach(items, (data) => {
        if (data !== null) {
            let newItem = new Item(data);
            bulkUpdate.find({
                classId: newItem.classId,
                instanceId: newItem.instanceId
            }).upsert().updateOne(newItem);
            items.push(newItem);
        }
    });

    await bulkUpdate.execute();
}

Whenever I run this code, it throws an error that complains about an _id field being changed, when the schema objects I created don't specify anything to do with schemas, and the few nested schema objects don't make a difference to the outcome when I change them to just objects.

I understand that if no _id is sent to MongoDB it auto generates one, but if it is updating a record it wouldn't do that anyway. I also tried setting _id to null on each item but to no avail.

Have I misunderstood anything about the accepted answer? Or is my problem elsewhere in my code?

Possible duplicate of [Bulk upsert in MongoDB using mongoose](https://stackoverflow.com/questions/25285232/bulk-upsert-in-mongodb-using-mongoose) — dnickless, Jan 17 '18 at 15:01
Does this answer your question? [Trying to do a bulk upsert with Mongoose. What's the cleanest way to do this?](https://stackoverflow.com/a/60330161/5318303) — Mir-Ismaili, Feb 21 '20 at 01:46

score 4 · Accepted Answer · edited Aug 24 '18 at 23:03

4

This is how I do it :

        let bulkUpdate = MyModel.collection.initializeUnorderedBulkOp();

        //myItems is your array of items
        _.forEach(myItems, (item) => {
            if (item !== null) {
                let newItem = new MyModel(item);
                bulkUpdate.find({ yyy: newItem.yyy }).upsert().updateOne(newItem);
            }
        });

        await bulkUpdate.execute();

I think the code is pretty readable and understandable. You can adjust it to make it work with your case :)

edited Aug 24 '18 at 23:03

letalumil

396
1
8
22

answered Jan 17 '18 at 14:52

HRK44

2,382
2
13
30

So from what I can gather, to search with the two unique IDs I can use `bulkUpdate.find({ classId: newItem.classId, instanceId: newItem.instanceId }).upsert().updateOne(newItem);`? Thank you for your very informative example. – Crazy Redd Jan 17 '18 at 17:41
I am having trouble implementing this solution. When I try and run this code I get `WriteError({"code":16837,"index":0,"errmsg":"The _id field cannot be changed`. I don't know if this has anything to do with nested schema, but something is trying to change _id field which is causing errors when I execute the update. – Crazy Redd Jan 18 '18 at 19:39
Did you set up the '_id' field in your 'newItem' object? If no, I can't help much more on that without having more code. – HRK44 Jan 19 '18 at 13:16
I have updated my answer to include my implementation of the answer you provided. I made some assumptions that I don't think are quite right. Thanks for your help. – Crazy Redd Jan 19 '18 at 14:45
Ok I think the error comes from the ```items.push(newItem);```. I think I mislead you by adding the variable myListOfItems, I just use it to store my items that come from some other source. I'm updating my answer to remove this. – HRK44 Jan 19 '18 at 15:21

Mongoose: Bulk upsert but only update records if they meet certain criteria

Update:

1 Answers1