Groovy gmongo batch processing

Question

I'm currently trying to run a batch processing job in groovy with Gmongo driver, the collection is about 8 gigs my problem is that my script tries to load everything in-memory, ideally I'd like to be able to process this in batch similar to what Spring Boot Batch does but in groovy scripts

I've tried batchSize(), but this function still retrieves the entire collection into memory only to apply it to my logic in batch-process.

here's my example

momngoDb.collection.find().collect() it -> {
  //logic
}

score 0 · Answer 1 · answered Jan 21 '19 at 15:04

0

according to official doc:

https://docs.mongodb.com/manual/tutorial/iterate-a-cursor/#read-operations-cursors

def myCursor = db.collection.find()

while (myCursor.hasNext()) {
   print( myCursor.next() }
}

answered Jan 21 '19 at 15:04

daggett

26,404
3
40
56

score 0 · Accepted Answer · answered May 01 '19 at 16:05

After deliberation I found this solution to works best for the following reasons.

Unlike the Cursor it doesn't retrieve documents on a singular basis for processing (which can be terribly slow)
Unlike the Gmongo batch funstion, it also doesn't try to upload the the entire collection in memory only to cut it up in batches for process, this tends to be heavy on machine resources.

code below is efficient and light on resource depending on your batch size.

def skipSize = 0
def limitSize = Integer.valueOf(1000) batchSize (if your going to hard code the batch size then you dont need the int convertion)
def dbSize = Db.collectionName.count()

def dbRunCount = (dbSize / limitSize).round()

dbRunCount.times { it ->
    dstvoDsEpgDb.schedule.find()
            .skip(skipSize)
            .limit(limitSize)
            .collect { event ->
            //run your business logic processing
            }

    //calculate the next skipSize   
    skipSize += limitSize

}

Groovy gmongo batch processing

2 Answers2