Acquisition of turn based concurrency lock for actor '{actorName}' timed out after {time}

Question

I have service which creates Actors of some type by some name:

var storer = this.serviceClient.Create<IStorer>(new ActorId(agencyToProcess.Name));

and then I call Actor's method.

await storer.StoreStatusesAsync().ConfigureAwait(false);

On this call I receive error :

System.AggregateException: One or more errors occurred. ---> Microsoft.ServiceFabric.Actors.ActorConcurrencyLockTimeoutException: Acquisition of turn based concurrency lock for actor 'actorName' timed out after 00:01:13.6480000. at Microsoft.ServiceFabric.Actors.Runtime.ActorConcurrencyLock.d__17.MoveNext()

I can't understand what this erorr means and how to fix it.

This problem doesn't happen everytime. (20 times out of 100).

[ServiceDescription("Storer", ServicePrefix = "MyApp")]
public interface IStorer : IActor
{
    Task StoreStatusesAsync();
}

This services are created in Observer. There are code full code which creates actors in observer.

public async void OnNext(AgencyModel agencyToProcess)
{
    try
    {
        var storer = this.serviceClient.Create<IStorer>(new ActorId(agencyToProcess.Name));

        await storer.StoreStatusesAsync().ConfigureAwait(false);
    }
    catch (Exception exception)
    {
        this.Logger.Error(exception);
    }
}

Diego Mendes · Accepted Answer · 2018-07-10T15:33:07.663

This happens because your service is trying to call an Actor that is currently locked processing another call.

By the looks of you code, you are triggering actors based on events, if two events targeting the same actor get called consecutively, one of them will wait for the previous to finish, and if the previous takes too long to complete, a timeout will throw the 'ActorConcurrencyLockTimeoutException'.

It does not happen often because each call might take a few seconds or less to process, but when you have many calls enqueued, the latest will wait for all previous to process in their respective order, and this will timeout soon or later.

To reduce these exceptions you could increase the timeout threshold, The default is 60 seconds, but in my opinion this is not a good idea, as it might enqueue too many requests and possibly not be able to process all of them, while holding resources and connections. These requests may also get lost when services are re-balanced.

The best solution is find an approach to throttle these requests.

Acquisition of turn based concurrency lock for actor '{actorName}' timed out after {time}

1 Answers1

Linked