0

What I'm doing is creating a transaction where

1) An entity A has a counter updated to +1
2) A new entity B is written to the datastore.

It looks like this:

WrappedBoolean result = ofy().transact(new Work<WrappedBoolean>() {
    @Override
    public WrappedBoolean run() {

        // Increment count of EntityA.numEntityBs by 1 and save it
        entityA.numEntityBs = entityA.numEntityBs +1;
        ofy().save().entity(entityA).now();

        // Create a new EntityB and save it
        EntityB entityB = new EntityB();
        ofy().save().entity(entityB).now();

        // Return that everything is ok
        return new WrappedBoolean("True");

    }
});

What I am doing is keeping a count of how many EntityB's entityA has. The two operations need to be in a transaction so either both saves happen or neither happen.

However, it is possible that many users will be executing the api method that contains the above transaction. I fear that I may run into problems of too many people trying to update entityA. This is because if multiple transactions try to update the same entity, the first one to commit wins but all the others fail.

This leads me to two questions:

1) Is the transaction I wrote a bad idea and destined to cause writes not being made if a lot of calls are made to the API method? Is there a better way to achieve what I am trying to do?

2) What if there are a lot of updates being made to an entity not in a transaction (such as updating a counter the entity has) - will you eventually run into a scaling problem if a lot of updates are being made in a short period of time? How does datastore handle this?

Sorry for the long winded question but I hope someone could shed some light on how this system works for me with the above questions. Thanks.

Edit: When I mean a lot of updates being made to an entity over a short period of time, consider something like Instagram, where you want to keep track of how many "likes" a picture has. Some users have millions of followers and when they post a new picture, they can get something like 10-50 likes a second.

Dan McGrath
  • 41,220
  • 11
  • 99
  • 130
Micro
  • 10,303
  • 14
  • 82
  • 120

1 Answers1

1

The datastore allows about 1 write/second per entity group. What might not appear obvious is that standalone entities (i.e. entities with no parent and no children) still belong to one entity group - their own. Repeated writes to the same standalone entity is thus subject to the same rate limit.

Exceeding the write limit will eventually cause write ops to fail with something like TransactionFailedError(Concurrency exception.)

Repeated writes to the same entity done outside transactions can overwrite each-other. Transactions can help with this - conflicting writes would be automatically retried a few times. Your approach looks OK from this prospective. But it only works if the average write rate remains below the limit.

You probably want to read Avoiding datastore contention. You need to shard your counter to be able to count events at more than 1/second rates.

Dan Cornilescu
  • 39,470
  • 12
  • 57
  • 97
  • If I increment a counter in an Entity inside a transaction, couldn't that transaction be executed several times by the app (due to retries) and increment the counter more than needed? I read somewhere that transactions need to be idempotent and something like `myEntity.count = myEntity.count + 1;` inside a transaction could be executed more than once. Or am I wrong (assuming we are not going past the write limit)? – Micro Jul 07 '16 at 17:14
  • 1
    Any datastore changes performed in a transaction that eventually fails are being rolled back. But non-datastore ops (for example memcache ops, messages being sent, etc) done inside the failed transaction can't be rolled back - these are what the idempotent note refers to. – Dan Cornilescu Jul 07 '16 at 18:18