Dataflow TransformManyBlock throttling

Question

How can I throttle a TransformManyBlock in a Dataflow mesh?

I specified a BoundedCapacity but it looks like it only afects the input queue.
So my block keeps processing input and output queue keeps growing.

The following blocks also have a BoundedCapacity specified, and then all my items stack up in the output queue of the TransformManyBlock eating all my RAM.

Answer 1

Yes, you are right, and there is no built-in opportunity to limit the output queue. It's done in such way as there will be much more overhead to check, are the output queue full, or not.

One thing you should check is what method you are using for adding your messages. If it is Post, it will block the thread before the message will be posted. But if you're using SendAsync, you should await it, in other case you will flood your RAM with await-state machines and messages which are in a middle of posting.

However, at least two things you can do:

1. Create a custom block with such property. This could be challenging and I do not recommend this.
2. Introduce the BufferBlock in your chain for such purpose - this is default way to add some throttling in dataflow. In this case messages wouldn't proceed futher on your dataflow if there is no place. This solution will work if you link your blocks between each other.