How does back pressure property work in Spark Streaming?

Question

I have a CustomReceiver which receives a single event(String).The received single event is used during spark application's run time to read data from nosql and to apply transformations.When the processing time for each batch was observed to be greater than batch interval I set this property.

spark.streaming.backpressure.enabled=true

After which I expected the CustomReceiver to not trigger and receive the event when a batch is processing longer than batch window, which didn't happen and still a backlog of batches were being added. Am I missing something here?

Have you read https://vanwilgenburg.wordpress.com/2015/10/06/spark-streaming-backpressure/? — Justin Pihony, Jan 25 '17 at 01:04
Does that mean the existing PIDEstimator wouldn't help in the above problem? And there isn't way I can plugin my own estimator to stop receiving new single event when a batch is still processing? — darkknight444, Jan 25 '17 at 01:48
Yes and that event is used later in application run time to fetch data from nosql. — darkknight444, Jan 25 '17 at 07:00

score 4 · Accepted Answer · answered Jun 08 '18 at 14:17

4

Try to check this and this articles.

answered Jun 08 '18 at 14:17

Eugene Lopatkin

2,351
1
22
34

How does back pressure property work in Spark Streaming?

1 Answers1