2
  1. A large array of objects gets initialized (Photo.new).
  2. Some filtering is done to the Photo array, so 30% of them remains.
  3. The remaining array gets saved in one single call.

Questions

  1. What is the command to do step 3, which is essentially a batch save?
  2. Is there a limit to how many objects I can save at once?
mu is too short
  • 426,620
  • 70
  • 833
  • 800
meow
  • 27,476
  • 33
  • 116
  • 177

2 Answers2

3

I'm not familiar with the Ruby Mongoid driver, but it seems like you're looking for something like: 

photos.map &:save

If you are trying to do this in a less iterative fashion (i.e. in a single call) it appears that the Ruby Mongoid driver supports an insert method to save an array of hashes:

Photo.collection.insert(photos)

Source: Batch insert/update using Mongoid?

Community
  • 1
  • 1
coreyward
  • 77,547
  • 20
  • 137
  • 166
  • Well, technically it's not Mongoid, doing this you work with MongoDB driver bypassing it. Good answer anyway. I remember doing this with map/reduce inline query result and I was shocked when it created N documents instead of one as I wanted :) – Jake Jones Jun 09 '11 at 00:28
  • i know you can do that with an array of hashes, but can you do that with an array of initialized (not created) objects? Thanks for the answer! =) – meow Jun 09 '11 at 02:06
  • 1
    Like I said, any Ruby object is JSON hash actually. So it seems, that the answer is "yes". Btw, try to inspect your objects and you will see what they look like. – Jake Jones Jun 09 '11 at 02:20
  • Ruby objects are not JSON hashes. I have no idea where you're inferring this. Hashes and JSON map relatively cleanly to one another, but a Hash is just one type of object in Ruby, and it still isn't represented as JSON behind the scenes (consider you can use a regex or another object [including another hash] as a hash key). – coreyward Jun 09 '11 at 15:00
  • @mingyeow just call `array.map(&:as_document)`, see [this question](http://stackoverflow.com/questions/3772378/batch-insert-update-using-mongoid) – Frank C. Schuetz Jun 13 '13 at 15:59
1

Yes, there is a limit. Everything you're going to batch insert using the method coreyward just described should be within 16M (MongoDB's document size limit). Look here:

https://github.com/mongodb/mongo-ruby-driver/commit/4712a684689c11a31221c87354e5ae0864960226

So you should estimate your array's byte size and split it into several parts if it exceeds 16M.

Jake Jones
  • 1,170
  • 1
  • 11
  • 15
  • Hmm.... but i am inserting N objects. does it matter if the array i am using to contain N objects exceeds 16M? – meow Jun 09 '11 at 02:08
  • You can insert as many objects as you want until they don't exceed 16M total. This all is inserted in one big insert call, and it seems MongoDB doesn't accept arrays bigger than its document size limit. Your objects are JSON hashes anyway, so it doesn't really matter what are you going to insert from MongoDB's point. – Jake Jones Jun 09 '11 at 02:15