2

I have been scouring the internet for a few hours to look at a solution where doing bulk upserts in Meteor.js's smart collections is efficient.

Scenario:

I am hitting an api to get updated info for 200 properties asynchronously every 12 hours. For each property I get an array of about 300 JSON objects on average. 70% of the objects might not have been updated. But for the rest of 30% objects I need to update them in the database. As there is no way to find those 30% objects without matching them with the documents in database, I decided to upsert all documents.

My options:

  1. Run a loop over the objects array and upsert each document in the database.
  2. Remove all documents from collection and bulk insert the new objects.

For option 1 running a loop and upserting 60K documents (which are going to increase with time), takes a-lot of time. But at the moment seems like the only plausible option.

For option 2 meteor.js does not allow bulk inserts in their smart collections. Even for that we will have to loop over the array of objects.

Is there another option where I can achieve this efficiently?

Jehanzeb.Malik
  • 3,332
  • 4
  • 25
  • 41

2 Answers2

1

MongoDb supports inserting an array of documents. So you can insert all your documents in one call from Meteors 'rawCollection'.

MyCollection.remove({}); // its empty

var theRaw = MyCollection.rawCollection();
var mongoInsertSync = Meteor.wrapAsync(theRaw.insert, theRaw);
var result = mongoInsertSync(theArrayOfDocs);

In production code you will wrap this in a try/catch to get hold of the error if your insert fails, result is only good if the inserting of the array of documents succeeds.

Pablo
  • 2,376
  • 5
  • 26
  • 34
0

The above solution with the rawCollection does insert, but it appears not to support the ordered:false directive to continue processing if one document fails, the raw collection exits on the first error, which is unfortunate.

"If false, perform an unordered insert, and if an error occurs with one of documents, continue processing the remaining documents in the array."

aloharich
  • 19
  • 5