2

I'm familiar with Datastore's single-write-per-second limit (ok, 5, maybe) for entity groups. How does that square with transactions?

The docs seem to indicate that I can do multiple modifications inside a transaction- for example adding several descendant entities:

A single transaction can modify multiple entities in a single group, or add new entities to the group by making the new entity's parent an existing entity in the group.

https://cloud.google.com/appengine/docs/java/datastore/transactions

Dan McGrath
  • 41,220
  • 11
  • 99
  • 130
nflacco
  • 4,972
  • 8
  • 45
  • 78
  • It's technically 1 write transaction per second per Entity Group, where a transaction can have up to 500 entities for a single Entity Group. This means you can, at maximum, write 500 entities per second into a single Entity Group. Also note, you can peak higher at once per second, although if you sustain it you increase your risk of hitting contention as well as the eventual consistency of the system. – Dan McGrath Aug 06 '16 at 04:15
  • @DanMcGrath: Hm, this should be spelled out in the documentation... – Dan Cornilescu Aug 06 '16 at 04:37
  • @DanCornilescu, we are actively working to improve it. For example, on our new limits documentation you'll see it called out more clearly: https://cloud.google.com/datastore/docs/concepts/limits – Dan McGrath Aug 06 '16 at 04:42
  • @DanMcGrath: Excellent, Thx! – Dan Cornilescu Aug 06 '16 at 05:51
  • Thanks, I chatted with the Datastore team and was also able to confirm this. – nflacco Aug 13 '16 at 15:37

1 Answers1

2

Yes, you can do multiple write operations per entity group inside the same transaction, but with care:

  • pay atention to not have writes causing conflicts inside the same transaction, otherwise the transaction will eventually fail (even after retries)
  • try to keep the amount of writes inside each transaction low or somehow ensure ample intervals between transactions such that the aggregate write ops rate (or rather its short-term average) remains (well) below the 1 write/s limit - to leave room for momentary peaks and occasional retries on failures which chew on that limit as well. Otherwise you'll get concurrency/contention exceptions. See Objectify - many writes to same entity in short period of time with and without transaction

And, of course, you can write to up to 25 entity groups (at this time) inside cross-group transactions (each getting its own ~1 write/s limit, for an aggregate of up to ~25 writes/s).

Striking the right balance between eventual consistency and write throuput is not trivial. This might be of interest: What would be the purpose of putting all datastore entities in a single group?

Update credit to DanMcGrath's comment:

It's technically 1 write transaction per second per Entity Group, where a transaction can have up to 500 entities for a single Entity Group. This means you can, at maximum, write 500 entities per second into a single Entity Group. Also note, you can peak higher at once per second, although if you sustain it you increase your risk of hitting contention as well as the eventual consistency of the system. – Dan McGrath 1 hour ago

Community
  • 1
  • 1
Dan Cornilescu
  • 39,470
  • 12
  • 57
  • 97
  • Confirmation from google docs: https://cloud.google.com/datastore/docs/concepts/structuring_for_strong_consistency – nflacco Jul 12 '16 at 16:37