Strategies for dealing with concurrency issues caused by stale Domain Objects (Grails/GORM/Hibernate)

Question

I like to refer to this issue as 'repeatable finder' problem because it it in some sense opposite to 'non repeatable read'. Because hibernate reuses objects attached to its session, results from a finder may include some old versions of objects which are now stale.

The problem is technically a Hibernate design gotcha, but since Hibernate session is implicit in Grails and Grails domain objects are long lived (HTTP request is long to me) I decided to ask this question in the context of Grails/GORM.

I would like to ask experts here if there are any commonly established strategies for dealing with this issue.

Consider this:

    class BankAccount {
      String name
      Float amount

      static constraints = {
        name unique: true
      }
    }

and 'componentA' code:

    BankAccount.findByName('a1')

'componentB code:

    def result = BankAccount.findAll()

Assume that componentA executes first, followed by some other logic, followed by componentB, result from component B is rendered by a view. Components A and B do not want to know much about each other.

This way componentB result contains old version of BankAccount 'a1'.

Many very embarrassing things can happen. If BankAccounts have been concurrently modified the presented list can, for example, contain 2 items with name 'a1' (uniqueness looks gone to the user!) or money transfers between accounts can show as partially applied transaction (if money was transferred from a2 to a1 then it will show deducted from a2 but not there yet for a1). These problems are embarrassing and can reduce user confidence in the application.

(ADDED 9/24/2014: Here is an eye opening example, this assert may fail:

  BankAccount.findAllByName('a1').every{ it.name == 'a1' }

Examples of how that happens can be found in any of the linked JIRA tickets or my blog. )

(ADDED 9/24/2014: NOTE: a seemingly sound advice to use database enforced unique keys in implementing equals() method is not concurrency safe. You may get 2 object with the same value of the 'business key' which are different.)

Possible solutions seem to be add a lot of discard() calls or a lot of withNewSession() calls and deal with LazyIntializationExeption and DuplicateKeyException, etc.
But if I do that why am I using hibernate/GORM? Calling refresh on each object returned from each query seems simply ridiculous.

My current thinking is that using short sessions/withNewSession in certain critical areas are the best approach but it does not solve the issue in all cases, just is some critical application areas.

Is this something Grails applications have to live with? Can you point me to any documentation/discussion about this issue?

EDITED 9/24/2014: Relevant Grails JIRA ticket: https://jira.grails.org/browse/GRAILS-11645, Hibernate JIRA: https://hibernate.atlassian.net/browse/HHH-9367 (has sadly been rejected), my blog has more detailed examples: http://rpeszek.blogspot.com/2014/08/i-dont-like-hibernategrails-part-2.html

ADDED 10/17/2014: I got several replies stating that this is any DB application/any ORM issue. This not correct.

It is true that this problem can be avoided by using long transactions (Hibernate session length/HTTP request length) + setting higher than normal DB isolation level of REPEATABLE READ. This solution is simply not acceptable (why do we have transnational services if for the application to work properly we need HTTP request long transactions!?)

DB applications and other ORMs will not exhibit this problem. They will not need long transactions to work and the problem is prevent with just READ COMMITTED.

It is now 2 months, since I posted this question here, and it has not received a meaningful answer. That is simply because this issue has no answer. It is something that Hibernate can fix, not something a Grails application can fix.ADDED 10/17/2014-END

Answer 1

Here is my own attempt to answer this question.

(ADDED 9/24/2014 There is simply no good solution to this problem. Sadly, HHH-9367 JIRA ticket has been rejected by Hibernate as 'not a bug'. The only solution suggested in that ticket was to use refresh (I assume that would require changing all queries to something that looks like:

BankAccount.findAllBy...(...).each{ it.refresh() }

Personally, I do not agree that this is a meaningful solution.)

As I have explained above, if Hibernate/GORM query returns a set of DomainObjects and some of these objects are already in hibernate session (populated by previous queries) the query will return these old objects, and these objects will not be automatically refreshed. This can cause some hard to spot concurrency issues. I call it Repeatable Finder problem.

This has nothing to do with 2nd level cache. This problem is caused by how hibernate works even without the 2nd level cache configured. (EDITED 9/24/2014: And, this is not any ORM, any DB application issue, the issue is specific to use of Hibernate).

Implications to your application:

(I can only explain the impacts I know of, I am not claiming that these are the only impacts).

Domain objects typically have a set of associated constraints/logical rules that need to hold across typically several records and are enforced by either the application or the database itself. I will borrow a term from FP and testing and will call these 'properties'.

Example properties: In the above BankAccount example, name uniqueness (enforced by DB) is a property (for example, you may use it in defining equals() method), if money is transferred between accounts, the total amount of money in these accounts needs to be a constant - this is a property.
If I modify my BankAccount class and add 'branch' association to it:

BankBranch branch

Then this is a property as well:

assert BankAccount.findAllByBranch(b).every{it.branch == b}.

(EDITED, This property should technically be enforced by DB and implementation of the finder method and developer may assume it is 'safe' and not breakable. In fact most 'where' criteria and 'joins' used by your app somewhere underneath hibernate define properties of similar nature.).

Repeatable finder problem can cause most of the properties to break under concurrent use (scary stuff!). For example I reiterate here a piece of code I wrote in the relevant JIRA ticket linked in the question:

... a1 has branch b1
BankAccount.findByName('a1')

... concurrently a1 is moved to branch b2
//fails because stale a1.branch == b1
assert BankAccount.findAllByBranch(b2).every{it.branch == b2} 

Your application probably uses explicit and implicit properties and may have logic to enforce them. For example, application may rely on names being unique and will exception or return wrong results if they are not unique (maybe name on its own is used to define equals()). This is explicit usage. Application may provide list views and it will be very embarrassing if list shows property violated (list for accounts under branch b2 shows some accounts with branch b1 - this is implicit usage). Any of such cases will be affected by "repeatable finder'.

If Grails code (not DB constraint) is used to enforce a property, then in addition to 'repeatable finder' more obvious concurrency concerns need to be addressed. (I am not discussing these here.)

Finding problems:

(This pertains to broken properties only. I do not know if repeatable finder causes other issues.)

So, I think the first step is to identify all properties in the app (EDITED: there will be many of them, potentially too many to examine - so, focusing on domain objects that are likely to change concurrently maybe the key here.), the second step is to identify where and how the application (implicitly or explicitly) uses these properties and how are they enforced. Code for each of these needs to be examined to verify that repeatable finder is not the issue.

It maybe a good idea, to simply enable SQL tracing (as well as tracing of where each HTTP request starts and ends) and examine log traces from identified areas of concern for any table name in the 'from' part of SQL. If such table shows up more than once per request this may be a good indication of a problem. Good Functional Test coverage could help in generating such log files.

This is obviously a not trivial process and there are no bullet proof solutions here.

Fixing problems:

Using discard() on objects from previous queries or running the query which relies on certain application property/properties in a separate hibernate session should solve the problem. Using new session approach should be more bullet proof. I do not recommend using refresh() here. (Note, hibernate provides no public API to query for objects attached to its session.)
Using new session will expose the application to some new problems like LazyInitalizationException or DupicateKeyException. These are trivial by comparison.

SIDE NOTE: I personally consider framework design decision which causes code to break when additional query is added: a terrible design flaw.

It is interesting to compare Hibernate against Active Record (which I know much less about). Hibernate took the ORM purist approach of trying to make RDBMS into OO, Active Record took the 'share nothing' approach of staying closer to DB and having DB deal with more complex concurrency issues.
Sure, in Active Record node.children.first().parent != parent but is that such a bad thing?
I admit to not understanding the reasons behind hibernate decision to not refresh objects in its cache when the new query is executed. Have they been concerned about side effects? Can Hibernate and Grails be lobbied to change that? Because that seems to be the BEST long term solution. (Edited 9/24/2014: my efforts to have Hibernate resolve the issue have failed.)

ADDED (2014/08/12): It could also helpful to rethink design of your Grails app and use GORM/Hibernate only as a very thin persistence layer. Designing such layer with a tight control over what queries are issued during each request should minimize this problem. This is obviously not what Grails framework advocates, (EDITED 9/24/2014 and it will only reduce not eliminate the problem.)

After lot of thinking about it, I seems to me that this maybe a major logical hole in Grails/Hibernate technology stack. There is really no good solution if you care about concurrency, you should be concerned.

Answer 2

Repeatable reads are a way of preventing lost updates in a database transaction. Most applications employ a read-modify-write data access pattern breaking database transaction boundaries and pushing transactions to the application-layer.

Hibernate employs a transactional write-behind policy, so entity state transitions are delayed as much as possible, to reduce database locking associated to DML statements.

In an application-level transaction, the first level cache acts as a application-level repeatable read mechanism. But while database locking ensures repeatable read consistency when using physical transactions, for application-level transactions you need an application-level locking mechanism. That's why you should always use optimistic locking in the first place.

Optimistic locking allows others to modify your previously loaded data while preventing you from updating stale data.

It's not the equals that's broken. The database constraints should always enforce unique business keys anyway.

For operations concerning account updates you should either use a single database transaction that ensures safety through lock acquisitions (SELECT FOR UPDATE) or use optimistic locking, so when others update your data you will get a stale entity exception.

I could replicate your use case. The entity is reused from the 1st-level cache. For SQL queries you have the freedom of loading concurrent changes. As long as you load entities for updating them later you should be fine, because the optimistic locking mechanism will prevent you from saving stale data.

If you use HQL/JPQL just for viewing then you might want to use projections instead.

Answer 3

A good article from Marc Palmer about these problems. I found it Very interesting. At the end of the article he gives somes "solutions" that could fit the needs of some of you.

The false optimism of GORM and Hibernate (archive)

Answer 4

As far as I can understand the question, the problem boils down to insufficient database transaction isolation.

I would also suggest that this problem can exist in anyapplicatoin, with any framework for database access.

Within a database transaction you have to live on the assumption that you are the only accessor to the database and have a consistent view of the database within that transaction.

Upon commit you might find that state has changed in a way incompatible with changes you made, and your transaction will roll back.

If you only did read-only access, then you have to still live on the assumption of consistency within the scope of your transaction, and the database shielding you from other concurrent modifications.

Hibernate's 2nd level cache spans transactions, so it should be cleared when there are concurrent modifications and in any case, a database can be modified by other applications, thus the 2nd level cache should be used with care.

But you said already the 2nd level cache is not your problem. Well, I agree. Your problem sounds like an insufficient degree of transaction isolation in your database. Can this be addressed?