Maximum spout capacity

Question

I'm using Heron for performing streaming analytics on IoT data. Currently in the architecture there is only one spout with parallelism factor 1.

I'm trying to benchmark the stats on the amount of data Heron can hold in the queue which it internally uses at spout.

I'm playing around with the method setMaxSpoutPending() by passing value to it. I want to know if there is any limit on the number which we pass to this method?

Can we tweak the parameter method by increasing system configuration or providing more resource to the topology?

Are you sure this should be tagged as a Storm/Spark question? It seems specifically about Heron. — Stig Rohde Døssing, Dec 30 '17 at 18:30
@StigRohdeDøssing But I couldn't find Heron tag while adding the question :( — tourist, Dec 31 '17 at 19:25
Sorry, I think you're right and there doesn't seem to be a Heron tag. I was just a little confused about why this was tagged as a Storm question :) — Stig Rohde Døssing, Jan 04 '18 at 15:57

Sanjeev Kulkarni · Accepted Answer · 2017-12-21T00:17:37.420

1

So if you have one spout and one bolt, then max spout pending is the best way to control the number of pending tuples. Max Spout pending can be increased indefinitely. However increasing it beyond a certain amount increases the probability of timeout errors happening and in the worst case there could be no forward progress. Also higher msp typically require more heap required for spout and other components of the topology.

edited Dec 21 '17 at 00:17

answered Dec 18 '17 at 21:32

Sanjeev Kulkarni

26
3

Your response is more about seeking clarification than giving an actual answer to the problem. It would be better done as a comment to the question. – Nick Dec 18 '17 at 21:40
@Sanjeev I'm receiving encoded data(temperature) in spout and passing it on to a bolt in the topology to compute the moving average on the decoded data(temperature). I'm doing some benchmarking on the data the spout can handle in flight. So that we can tweak the resource allocation to the topology and see if something changes – tourist Dec 19 '17 at 22:30

S. Hines · Answer 2 · 2018-01-04T16:28:35.500

MSP is used to control the topology ingestion rate; it tells Storm the maximum number of tuples that may be unacknowledged at any given time. If the MSP is lower than the parallelism of the topology, it can be a bottle neck. On the other hand, increasing MSP beyond the topology parallelism level can lead to the topology being 'flooded' and unable to keep up with the inbound tuples. In such a situation the 'message timeout' of the topology will be exceeded and Storm will attempt to replay them while still feeding new tuples. Storm will stop feeding new inbound tuples only when the MSP limit is reached.

So yes, you can tweak it but keep an eye out for increasing timed out tuples indicating that your topology is overwhelmed.

BTW, if you're processing IoT events you may be able to increase parallelism by grouping the spout tuples by the device id (tuple stream per device) using field grouping.

Maximum spout capacity

2 Answers2