3

In a shared expenses app that shows payments dues and shared expenses details for each group. As a financial application, so many operations are transactional, which requires strong consistency to ensure data integrity. We used Entity Groups and ancestor queries which seems to have solved the issue of strong consistency, this caused the entity group to be large in size. As the shared 'group' is now the parent of members, expenses, payments, dues..etc. Until now we don't see a problem, but we are worried as this scales, expenses and/or payments can scale to the order of 10~100K entities.

After a bit of research we found these suggestions:

  1. Turning back to eventual consistency, and writing to memcache to overcome eventual consistency. Is this the best solution? We couldn't find enough docs on how to update an entity in memcache, if you know any please share! Also how can we ensure writes are transactional in that case?
  2. Keep things as is, as there is no entity group size. But we are worried of what would happen later, as the docs mentioned keeping entity groups no larger than a single user's worth of data
  3. Switch to RDBMS, might be too late for that now

We don't have a problem with the 1 write/sec rule right now as usually in those groups there isn't much going on per second for this to cause a problem, but we don't want to run into an issue later. It would be cumbersome to migrate or change the data models, specially that changing or removing ancestors, requires rewriting the entities, lots of delete and write requests which also costs!

Appreciate your input

Community
  • 1
  • 1
Khaled
  • 907
  • 1
  • 8
  • 18
  • You're right to worry: large entity groups will lead to contention: https://cloud.google.com/appengine/articles/scaling/contention#keep-entity-groups-small – Dan Cornilescu Feb 05 '18 at 04:16
  • Thanks @DanCornilescu, right now we are using batch operations on most of our writes to avoid contention as much as we can. We are still looking for the best setup though to mitigate any issues – Khaled Feb 05 '18 at 13:02
  • Entity groups and ancestor queries are not a good solution here. Instead, use cross-group transactions to ensure strong consistency. – new name Feb 05 '18 at 14:24
  • Thanks @JeffO'Neill, we use cross-group transactions in some situations where we need to make sure all write operations are committed or none. Afaik, using transactions alone doesn't ensure strong consistency, it just makes sure that all write operations happen altogether or not at all. Strong consistency seem to be achieved only through using Ancestor queries or lookup by key method https://cloud.google.com/datastore/docs/articles/balancing-strong-and-eventual-consistency-with-google-cloud-datastore/#strong-consistency-on-reading-entity-values-and-indexes – Khaled Feb 05 '18 at 15:20
  • 1
    @Khaled, you can get strong consistency by passing the entity keys to your cross-group transaction and then retrieving the entities using the keys. – new name Feb 05 '18 at 21:07

0 Answers0