Service Fabric reliable queue long operation

Question

I'm trying to understand some best practices for service fabric.

If I have a queue that is added to by a web service or some other mechanism and a back end task to process that queue what is the best approach to handle long running operations in the background.

Use TryPeekAsync in one transaction, process and then if successful use TryDequeueAsync to finally dequeue.
Use TryDequeueAsync to remove an item, put it into a dictionary and then remove from the dictionary when complete. On startup of the service, check the dictionary for anything pending before the queue.

Both ways feel slightly wrong, but I can't work out if there is a better way.

anderso · Accepted Answer · 2016-03-02T10:56:40.683

One option is to process the queue in RunAsync, something along the lines of this:

protected override async Task RunAsync(CancellationToken cancellationToken)
{
    var store = await StateManager.GetOrAddAsync<IReliableQueue<T>>("MyStore").ConfigureAwait(false);
    while (!cancellationToken.IsCancellationRequested)
    {
        using (var tx = StateManager.CreateTransaction())
        {
            var itemFromQueue = await store.TryDequeueAsync(tx).ConfigureAwait(false);
            if (!itemFromQueue.HasValue)
            {
                await Task.Delay(TimeSpan.FromSeconds(1), cancellationToken).ConfigureAwait(false);
                continue;
            }

            // Process item here
            // Remmber to clone the dequeued item if it is a custom type and you are going to mutate it.
            // If success, await tx.CommitAsync();
            // If failure to process, either let it run out of the Using transaction scope, or call tx.Abort();
        }
    }
}

Regarding the comment about cloning the dequeued item if you are to mutate it, look under the "Recommendations" part here: https://azure.microsoft.com/en-us/documentation/articles/service-fabric-reliable-services-reliable-collections/

One limitation with Reliable Collections (both Queue and Dictionary), is that you only have parallelism of 1 per partition. So for high activity queues it might not be the best solution. This might be the issue you're running into.

What we've been doing is to use ReliableQueues for situations where the write amount is very low. For higher throughput queues, where we need durability and scale, we're using ServiceBus Topics. That also gives us the advantage that if a service was Stateful only due to to having the ReliableQueue, it can now be made stateless. Though this adds a dependency to a 3rd party service (in this case ServiceBus), and that might not be an option for you.

Another option would be to create a durable pub/sub implementation to act as the queue. I've done tests before with using actors for this, and it seemed to be a viable option, without spending too much time on it, since we didn't have any issues depending on ServiceBus. Here is another SO about that Pub/sub pattern in Azure Service Fabric

The problem I have with this is that it keeps the transaction open for some time which doesn't seem correct. Also if the operation takes more than 4 seconds, the transaction fails anyway. — Nick Randell, Mar 02 '16 at 05:49
How long is "some time" ? Also, how often do you add stuff to the queue? — anderso, Mar 02 '16 at 08:53
Imagine some time could be 1 minute. As for adding, at peak times could be many times a second. I know if you take this to the extreme you could just back up work. What I'm trying to find out is there a best practice for reliably processing without causing deadlock in the system. — Nick Randell, Mar 02 '16 at 10:08
superb - that fits in with some of my thinking as well - I like this answer — Nick Randell, Mar 02 '16 at 15:07
@anderso - curious as to why you consistently use `.ConfigureAwait(false)` when awaiting async work? — Peter Lillevold, Feb 10 '17 at 08:57
@PeterLillevold See https://stackoverflow.com/questions/13489065/best-practice-to-call-configureawait-for-all-server-side-code on why to call `.ConfigureAwait(false)` on awaited async method invocations — Stephanvs, Jul 12 '17 at 08:11
@Stephanvs indeed. And with a lot of good discussion on why one _shouldn't_ have to call `.ConfigureAwait(false)`. Hence my question, why is the OP consistently using it in this case? Is it because there is some performance gain to be had, or is it because the OP have learned that "this is how you should do it, don't ask why" .. — Peter Lillevold, Jul 12 '17 at 11:16

score 3 · Answer 2 · answered Aug 07 '16 at 23:29

3

If very slow use 2 queues.. One a fast one where you store the work without interruptions and a slow one to process it. RunAsync is used to move messages from the fast to the slow.

answered Aug 07 '16 at 23:29

user1496062

1,309
1
7
22

Service Fabric reliable queue long operation

2 Answers2