Datastore writes to same entitygroup from Dataflow

Question

I am writing to Datastore from Dataflow, pcollection has multiple entities of same entity group to be written to datastore. Does writing them to datastore would be slow because of same entity writes/updates limitations? Can I somehow do transactions from dataflow? And if yes, would writing same entitygroup in a single transaction be faster?

Answer 1

Yes, you need to respect the max average 1 write/sec per entity group. It is not enforced at the API level, it's your app's responsibility to do it, if you exceed it (for extended periods, their length also depending on the write peak rate) you'll just get errors.

I'm not familiar with dataflow, so I can't tell if you can do transactions, but even if you can, they won't help in this matter: you still need to respect the max write rate per entity group. Also, transactions are typically slower due to the extra work they do and the retries done when collisions are encountered. If you do not need them (to prevent collisions and/or for consistency) it'd be faster without them.

But keep in mind that a write can contain up to 500 entities, see update on this Q&A touching specifically on the transaction case: Datastore: Multiple writes against an entity group inside a transaction exceeds write limit?.