Stack Overflow
Stack Overflow

Questions tagged [backpressure]

Backpressure refers to the build-up of data at an I/O switch when buffers are full and not able to receive additional data. No additional data packets are transferred until the bottleneck of data has been eliminated or the buffer has been emptied.

Backpressure refers to the build-up of data at an I/O switch when buffers are full and not able to receive additional data. No additional data packets are transferred until the bottleneck of data has been eliminated or the buffer has been emptied.

Source - Techopedia

164 questions
58
votes
1 answer

Backpressure mechanism in Spring Web-Flux

I'm a starter in Spring Web-Flux. I wrote a controller as follows: @RestController public class FirstController { @GetMapping("/first") public Mono getAllTweets() { return Mono.just("I am First Mono") } } I know…
Sam
  • 6,770
  • 7
  • 50
  • 91
30
votes
5 answers

Using BlockingCollection: OperationCanceledException, is there a better way?

I'm making use of the (frankly great) BlockingCollection type for a heavily multithreaded, high-performance app. There's a lot of throughput through the collection and on the micro-level it's highly performant. However, for each 'batch' it will…
Kieren Johnstone
  • 41,277
  • 16
  • 94
  • 144
28
votes
3 answers

What does the term "backpressure" mean in Rxjava?

I am a beginner of RxJava and I am curious about the meaning of "backpressure". Does it mean that the producer puts pressure behind the consumer's back? Or does it mean that consumers are putting pressure on producers? (Pressure in opposite…
ohlab
  • 467
  • 5
  • 7
18
votes
3 answers

What is a correct way to pause piped readable stream from writeable one in nodejs?

I am writing a module, which is a writeable stream. I want to implement pipe interface for my users. If some error happens, i need to pause readable stream and emit error event. Then, user will decide - if he is ok with error, he should be able to…
avasin
  • 9,186
  • 18
  • 80
  • 127
14
votes
2 answers

Back pressure in Kafka

I have a situation in Kafka where the producer publishes the messages at a very higher rate than the consumer consumption rate. I have to implement the back pressure implementation in kafka for further consumption and processing. Please let me know…
sirigiri sai kumar
  • 141
  • 1
  • 1
  • 3
13
votes
1 answer

Node.JS Unbounded Concurrency / Stream backpressure over TCP

As I understand it, one of the consequences of Node's evented IO model is the inability to tell a Node process that is (for example) receiving data over a TCP socket, to block, once you've hooked up your receiving event handlers (or otherwise…
MikeL
  • 483
  • 2
  • 6
  • 12
11
votes
2 answers

How Spark Structured Streaming handles backpressure?

I'm analyzing the backpressure feature on Spark Structured Streaming. Does anyone know the details? Is it possible to tune process incoming records by code? Thanks
Aniello Guarino
  • 197
  • 2
  • 10
9
votes
1 answer

Where is the memory leak when mapcat breaks backpressure in core.async?

I wrote some core.async code in Clojure and when I ran it it consumed all available memory and failed with an error. It appears that using mapcat in a core.async pipeline breaks back pressure. (Which is unfortunate for reasons beyond the scope of…
Rachel K. Westmacott
  • 2,038
  • 20
  • 15
8
votes
5 answers

RxJs: lossy form of zip operator

Consider using the zip operator to zip together two infinite Observables, one of which emits items twice as frequently as the other. The current implementation is loss-less, i.e. if I keep these Observables emitting for an hour and then I switch…
JeB
  • 11,653
  • 10
  • 58
  • 87
7
votes
1 answer

Difference between BackpressureStrategy.BUFFER and onBackpressureBuffer operator in rxjava 2

I am new to the world of reactive programming and I am trying to create a simple backpressure aware message processing using rxjava 2. Following is the workflow I am trying to achieve: Flowable of a continues string stream. Perform a time consuming…
Rahul khandelwal
  • 331
  • 3
  • 14
7
votes
1 answer

How does back pressure property work in Spark Streaming?

I have a CustomReceiver which receives a single event(String).The received single event is used during spark application's run time to read data from nosql and to apply transformations.When the processing time for each batch was observed to be…
darkknight444
  • 546
  • 8
  • 21
7
votes
1 answer

What is back pressure on stream processing?

I started learning NodeJS, and streams seems to be something that people use a lot. In most of the documentation that I had read there are mentions of the "back-pressure problem" that occurs when you are processing big sized files but I haven't…
David Florez
  • 1,460
  • 2
  • 13
  • 26
6
votes
3 answers

Android : Live Data skipping values

I am using Live Data to publish states from View Model to Fragments, this might result in states getting published frequently. But the Mutable Live Data is skipping the initial values and taking the latest value available. There is an article which…
ashwin mahajan
  • 1,654
  • 5
  • 27
  • 50
6
votes
1 answer

RxJava2 Observable backpressure

recently I realized that I don't understand how RxJava2 backpressure works. I made small test and I expect that it should fail with MissingBackpressureException exception: @Test public void testBackpressureWillFail() { …
Rostyslav Roshak
  • 3,859
  • 2
  • 35
  • 56
6
votes
1 answer

Spark Streaming Kafka backpressure

We have a Spark Streaming application, it reads data from a Kafka queue in receiver and does some transformation and output to HDFS. The batch interval is 1min, we have already tuned the backpressure and spark.streaming.receiver.maxRate parameters,…
YichaoCai
  • 71
  • 1
  • 4
1
2 3
…
10 11