Handle exceptions with TPL Dataflow blocks

Question

I have a simple tpl data flow which basically does some tasks. I noticed when there is an exception in any of the datablocks, it wasn't getting caught in the initial parent block caller. I have added some manual code to check for exception but doesn't seem the right approach.

if (readBlock.Completion.Exception != null
    || saveBlockJoinedProcess.Completion.Exception != null
    || processBlock1.Completion.Exception != null
    || processBlock2.Completion.Exception != null)
{
    throw readBlock.Completion.Exception;
}

I had a look online to see what's a suggested approach but didn't see anything obvious. So I created some sample code below and was hoping to get some guidance on a better solution:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using System.Threading.Tasks.Dataflow;

namespace TPLDataflow
{
    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                //ProcessB();
                ProcessA();
            }
            catch (Exception e)
            {
                Console.WriteLine("Exception in Process!");
                throw new Exception($"exception:{e}");
            }
            Console.WriteLine("Processing complete!");
            Console.ReadLine();
        }

        private static void ProcessB()
        {
            Task.WhenAll(Task.Run(() => DoSomething(1, "ProcessB"))).Wait();
        }

        private static void ProcessA()
        {
            var random = new Random();
            var readBlock = new TransformBlock<int, int>(x =>
            {
                try { return DoSomething(x, "readBlock"); }
                catch (Exception e) { throw e; }
            }); //1

            var braodcastBlock = new BroadcastBlock<int>(i => i); // ⬅ Here

            var processBlock1 = new TransformBlock<int, int>(x =>
                DoSomethingAsync(5, "processBlock1")); //2
            var processBlock2 = new TransformBlock<int, int>(x =>
                DoSomethingAsync(2, "processBlock2")); //3

            //var saveBlock =
            //    new ActionBlock<int>(
            //    x => Save(x)); //4

            var saveBlockJoinedProcess =
                new ActionBlock<Tuple<int, int>>(
                x => SaveJoined(x.Item1, x.Item2)); //4

            var saveBlockJoin = new JoinBlock<int, int>();

            readBlock.LinkTo(braodcastBlock, new DataflowLinkOptions
                { PropagateCompletion = true });

            braodcastBlock.LinkTo(processBlock1,
                new DataflowLinkOptions { PropagateCompletion = true }); //5

            braodcastBlock.LinkTo(processBlock2,
                new DataflowLinkOptions { PropagateCompletion = true }); //6


            processBlock1.LinkTo(
                saveBlockJoin.Target1); //7

            processBlock2.LinkTo(
                saveBlockJoin.Target2); //8

            saveBlockJoin.LinkTo(saveBlockJoinedProcess,
                new DataflowLinkOptions { PropagateCompletion = true });

            readBlock.Post(1); //10
                               //readBlock.Post(2); //10

            Task.WhenAll(processBlock1.Completion,processBlock2.Completion)
                .ContinueWith(_ => saveBlockJoin.Complete());

            readBlock.Complete(); //12
            saveBlockJoinedProcess.Completion.Wait(); //13
            if (readBlock.Completion.Exception != null
                || saveBlockJoinedProcess.Completion.Exception != null
                || processBlock1.Completion.Exception != null
                || processBlock2.Completion.Exception != null)
            {
                throw readBlock.Completion.Exception;
            }
        }
        private static int DoSomething(int i, string method)
        {
            Console.WriteLine($"Do Something, callng method : { method}");
            throw new Exception("Fake Exception!");
            return i;
        }
        private static async Task<int> DoSomethingAsync(int i, string method)
        {
            Console.WriteLine($"Do SomethingAsync");
            throw new Exception("Fake Exception!");
            await Task.Delay(new TimeSpan(0, 0, i));
            Console.WriteLine($"Do Something : {i}, callng method : { method}");
            return i;
        }
        private static void Save(int x)
        {

            Console.WriteLine("Save!");
        }
        private static void SaveJoined(int x, int y)
        {
            Thread.Sleep(new TimeSpan(0, 0, 10));
            Console.WriteLine("Save Joined!");
        }
    }
}

if you are open to alternatives, I would suggest going the Rx way with Observables - they much easier to wrap your head around — Krzysztof Skowronek, Jul 09 '19 at 12:23
@KrzysztofSkowronek I like Rx too but went with TPL DF as a way to try it out for this particular task :) — Rubans, Sep 06 '19 at 11:03

score 3 · Answer 1 · answered Jul 10 '19 at 04:16

I had a look online to see what's a suggested approach but didn't see anything obvious.

If you have a pipeline (more or less), then the common approach is to use PropagateCompletion to shut down the pipe. If you have more complex topologies, then you would need to complete blocks by hand.

In your case, you have an attempted propagation here:

Task.WhenAll(
    processBlock1.Completion,
    processBlock2.Completion)
    .ContinueWith(_ => saveBlockJoin.Complete());

But this code will not propagate exceptions. When both processBlock1.Completion and processBlock2.Completion complete, saveBlockJoin is completed successfully.

A better solution would be to use await instead of ContinueWith:

async Task PropagateToSaveBlockJoin()
{
    try
    {
        await Task.WhenAll(processBlock1.Completion, processBlock2.Completion);
        saveBlockJoin.Complete();
    }
    catch (Exception ex)
    {
        ((IDataflowBlock)saveBlockJoin).Fault(ex);
    }
}
_ = PropagateToSaveBlockJoin();

Using await encourages you to handle exceptions, which you can do by passing them to Fault to propagate the exception.

score 1 · Answer 2 · answered Aug 13 '19 at 11:00

Propagating errors backward in the pipeline is not supported in the TPL Dataflow out of the box, which is especially annoying when the blocks have a bounded capacity. In this case an error in a block downstream may cause the blocks in front of it to block indefinitely. The only solution I know is to use the cancellation feature, and cancel all blocks in case anyone fails. Here is how it can be done. First create a CancellationTokenSource:

var cts = new CancellationTokenSource();

Then create the blocks one by one, embedding the same CancellationToken in the options of all of them:

var options = new ExecutionDataflowBlockOptions()
    { BoundedCapacity = 10, CancellationToken = cts.Token };

var block1 = new TransformBlock<double, double>(Math.Sqrt, options);
var block2 = new ActionBlock<double>(Console.WriteLine, options);

Then link the blocks together, including the PropagateCompletion setting:

block1.LinkTo(block2, new DataflowLinkOptions { PropagateCompletion = true });

Finally use an extension method to trigger the cancellation of the CancellationTokenSource in case of an exception:

block1.OnFaultedCancel(cts);
block2.OnFaultedCancel(cts);

The OnFaultedCancel extension method is shown below:

public static class DataflowExtensions
{
    public static void OnFaultedCancel(this IDataflowBlock dataflowBlock,
        CancellationTokenSource cts)
    {
        dataflowBlock.Completion.ContinueWith(_ => cts.Cancel(), default,
            TaskContinuationOptions.OnlyOnFaulted |
            TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
    }
}

score 0 · Answer 3 · answered Jul 09 '19 at 11:32

at the first look, if have only some minor points (not looking at your architecture). it seems to me that you have mixed some newer and some older constructs. and there are some code parts which are unnecessary.

for example:

private static void ProcessB()
{
    Task.WhenAll(Task.Run(() => DoSomething(1, "ProcessB"))).Wait();
}

using the Wait()-method, if any exceptions happen, they will be wrapped in a System.AggregateException. in my opinion, this is better:

private static async Task ProcessBAsync()
{
    await Task.Run(() => DoSomething(1, "ProcessB"));
}

using async-await, if an exception occurs, the await statement rethrows the first exception which is wrapped in the System.AggregateException. This allows you to try-catch for concrete exception types and handle only cases you really can handle.

another thing is this part of your code:

private static void ProcessA()
        {
            var random = new Random();
            var readBlock = new TransformBlock<int, int>(
                    x => 
                    { 
                    try { return DoSomething(x, "readBlock"); } 
                    catch (Exception e) 
                    { 
                    throw e; 
                    } 
                    },
                    new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 1 }); //1

Why catch an exception only to rethrow it? in this case, the try-catch is redundant.

And this here:

private static void SaveJoined(int x, int y)
{
    Thread.Sleep(new TimeSpan(0, 0, 10));
    Console.WriteLine("Save Joined!");
}

It is much better to use await Task.Delay(....). Using Task.Delay(...), your application will not freeze.

Handle exceptions with TPL Dataflow blocks

3 Answers3

Linked