Mongoose update or insert many documents

Question

I'm trying to use the latest version of mongoose to insert an array of objects, or update if a corresponding product id already exists. I can't for the life of me figure out the right method to use (bulkWrite, updateMany etc) and I can't can't seem to figure out the syntax without getting errors. For example, i'm trying

Product.update({}, products, { upsert : true, multi : true }, (err, docs) => console.log(docs))

this throws the error DeprecationWarning: collection.update is deprecated. Use updateOne, updateMany, or bulkWrite instead. MongoError: '$set' is empty. You must specify a field like so: {$set: {<field>: ...}

and using updateMany just gives me the $set error. I can't seem to find any examples of people using this method, just wondering if somebody can provide me with an example of how to properly use this.

To be clear, I generate an array of objects, then I want to insert them into my db, or update the entry if it already exists. The field I want to match on is called pid. Any tips or advice would be greatly appreciated!

EDIT:

My product schema

const productSchema = new mongoose.Schema({
  title : String,
  image : String,
  price_was : Number,
  price_current : {
    dollars : String,
    cents : String
  },
  price_save_percent : String,
  price_save_dollars : String,
  price_save_endtime : String,
  retailer : String
})

const Product = mongoose.model('Product', productSchema)

an example of the products array being passed

[
  {   
  title: 'SOME PRODUCT',
  image: '',
  price_was: '139.99',
  price_current: { dollars: '123', cents: '.49' },
  price_save_percent: '12%',
  price_save_dollars: '16.50',
  price_save_endtime: null,
  pid: 'VB78237321',
  url: '' 
  },
  { ... },
  { ... }
]

what does `products` contain? and also show your schema as well — Ashh, Feb 15 '19 at 17:19
I've updated my question with the product schema and an example of the `products` object i'm passing in — Shan Robertson, Feb 15 '19 at 17:25
what should be update criteria to insert or update the product documents? — Ashh, Feb 15 '19 at 17:28
the `pid` field is the only unique id for the product and its from the retailer so it'll be pretty static i assume. — Shan Robertson, Feb 15 '19 at 17:31

score 20 · Accepted Answer · edited Nov 01 '20 at 03:19

20

You basically need bulkWrite operation

The array you want to update with

const products = [
  {   
    title: 'SOME PRODUCT',
    image: '',
    price_was: '139.99',
    price_current: { dollars: '123', cents: '.49' },
    price_save_percent: '12%',
    price_save_dollars: '16.50',
    price_save_endtime: null,
    pid: 'VB78237321',
    url: ''
  }
]

The query for bulk update

Model.bulkWrite(
  products.map((product) => 
    ({
      updateOne: {
        filter: { retailer : product.pid },
        update: { $set: product },
        upsert: true
      }
    })
  )
)

edited Nov 01 '20 at 03:19

Ali Mousavi

855
7
15

answered Feb 15 '19 at 17:37

Ashh

44,693
14
105
132

ok that works exactly as i wanted! Thanks alot for the info man. So one question i do have is... thats literally looping over my array. Some of these updates could be 1000's of products. Is this THE way to do it? Is there no way to simply pass the array of objects to mongoose? – Shan Robertson Feb 15 '19 at 17:46
2

`bulkWrite` operation can handle up to 100000 operations simultaneously. – Ashh Feb 15 '19 at 17:48
Nice solution. I'm dealing with a similar situation and have the following question: say I use bulkWrite() to update multiple documents at once, but while waiting for the promise to be resolved (or rejected), the documents continue to update. What should I do in this case? Am I safe once bulkWrite() returns (not fulfilled, just returns the promise) or will the updates override the documents awaiting the bulk operation? – omer Jun 03 '19 at 17:14
@omer It would be better if you ask a new question with proper explanation. – Ashh Jun 03 '19 at 17:20
@Ashh I tried to use your code, it didn't work as i expected. for some reason it is inserting some documents - and i cant understand when the process ends. I have the same situation like ShanRobertson, But the problem is that i need to know when the insert is done (it takes really long time) For now im just looping through my products array and checking if each product exists. If so - update it, if not - insert new one. I was searching for a ways to improve it and it's the first time i see something that can actually work. could you help me with that? (and why is it so slow?!) – Àtishking May 10 '20 at 22:10
@Àtishking all the Mongoose functions return a promise, or call a callback (3rd parameter) if provided. So when all documents are written you can use `bulkWrite().then((res)=>{ do something })` or `bulkWrite(query, opts, (err, res)=>{ do something })`. – scipilot Apr 03 '21 at 06:47
Hi, using this code the duplicate entries are falling rather than updation why it is like that please reply. – BKM Oct 21 '22 at 09:06

Mongoose update or insert many documents

1 Answers1

Linked