Grails hibernate session in batches

Question

GORM works fine out of the box as long as there is no batch with more than 10.000 objects. Without optimisation you will face the outOfMemory problems.

The common solution is to flush() and clear() the session each n (e.g.n=500) objects:

Session session = sessionFactory.currentSession
Transaction tx = session.beginTransaction();
def propertyInstanceMap = org.codehaus.groovy.grails.plugins.DomainClassGrailsPlugin.PROPERTY_INSTANCE_MAP

Date yesterday = new Date() - 1

Criteria c = session.createCriteria(Foo.class)
c.add(Restrictions.lt('lastUpdated',yesterday))
ScrollableResults rawObjects = c.scroll(ScrollMode.FORWARD_ONLY)

int count=0;
while ( rawObjects.next() ) {
    def rawOject = rawObjects.get(0);

    fooService.doSomething()

    int batchSize = 500
    if ( ++count % batchSize == 0 ) {
        //flush a batch of updates and release memory:
        try{
            session.flush();
        }catch(Exception e){
            log.error(session)
            log.error(" error: " + e.message)
            throw e
        }
        session.clear();
        propertyInstanceMap.get().clear()
    }
}

session.flush()
session.clear()
tx.commit()

But there are some problems I can't solve:

If I use currentSession, then the controller fails because of session is empty
If I use sessionFactory.openSession(), then the currentSession is still used inside FooService. Of cause I can use the session.save(object) notation. But this means, that I have to modify fooService.doSomething() and duplicate code for single operation (common grails notation like fooObject.save() ) and batch operation (session.save(fooObject() ).. notation).
If I use Foo.withSession{session->} or Foo.withNewSession{session->}, then the objects of Foo Class are cleared by session.clear() as expected. All the other objects are not cleared(), what leads to memory leak.
Of cause I can use evict(object) to manualy clear the session. But it is nearly impossible to get all relevant objects, due to autofetching of assosiations.

So I have no idea how to solve my problems without making the FooService.doSomething() more complex. I'm looking for something like withSession{} for all domains. Or to save session at the begin (Session tmp = currentSession) and do something like sessionFactory.setCurrentSession(tmp). Both doesn't exists!

Any idea is wellcome!

This looks like work that should be done entirely in a service method. If within the service method, you use `currentSession`, does your controller still work? — doelleri, Dec 06 '12 at 18:10
I agree with @doelleri. Services are the wright place to do it. Also, remember that by default they are transactional, if you want to handle manually the status, use `Domain.withTransaction` and set the `static transactional = false` or let the service take care of the commit/rollback. — , Dec 06 '12 at 20:02
The code I posted there is already inside a service method. Yes I could use the transaction context of the service method, but it will not solve my problem. @doelleri - The answer to your question is: the controller brakes, so that the user can't do anything more in the application except to close the browser. (see problem 1) — Waldemar, Dec 07 '12 at 08:17
Take a look to [Spring Batch](http://static.springsource.org/spring-batch/). Is easy to integrate in grails and is very powerfull — Fabiano Taioli, Dec 07 '12 at 17:45
Thanks for suggestion! I would like to avoid Spring Batch due to its complexity. I have already quarz job integrated what will mean, that with Spring Batch and Quarz there will be two job scheduling approaches in one application. Is there any simpler solution? — Waldemar, Dec 10 '12 at 16:35

score 1 · Answer 1 · edited May 23 '17 at 12:17

1

I would recommend to use stateless session for this kind of batch processing. See this post: Using StatelessSession for Batch processing

edited May 23 '17 at 12:17

Community

1
1

answered Sep 30 '15 at 10:02

stenix

3,068
2
19
30

score 0 · Answer 2 · answered Feb 21 '17 at 23:15

A modified approach to what you are doing would be:

Loop over your entire collection (rawObjects) and save a list of all the ids for those objects.
Loop over the list of ids. At each iteration, look up just that single object, by its id.

Then use the same periodic clearing of the session cache like you are doing now.

By the way, someone else has suggested an approach similar to yours. But note that the code in this link is incorrect; the lines that clear the session should be inside the if statement, just like you have in your solution.

Grails hibernate session in batches

2 Answers2