Questions tagged [producer-consumer]

The Producer-Consumer Problem (also known as the bounded-buffer problem) is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer used as a queue.

The Producer-Consumer Problem (also known as the bounded-buffer problem) is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer used as a queue. The producer's job is to generate a piece of data, put it into the buffer and start again. At the same time the consumer is consuming the data (i.e., removing it from the buffer) one piece at a time. The problem is to make sure that the producer won't try to add data into the buffer if it's full and that the consumer won't try to remove data from an empty buffer.

The solution for the producer is to either go to sleep or discard data if the buffer is full. The next time the consumer removes an item from the buffer, it notifies the producer who starts to fill the buffer again. In the same way, the consumer can go to sleep if it finds the buffer to be empty. The next time the producer puts data into the buffer, it wakes up the sleeping consumer. The solution can be reached by means of inter-process communication, typically using semaphores. An inadequate solution could result in a deadlock where both processes are waiting to be awakened. The problem can also be generalized to have multiple producers and consumers.

References

1564 questions
171
votes
13 answers

Is Zookeeper a must for Kafka?

In Kafka, I would like to use only a single broker, single topic and a single partition having one producer and multiple consumers (each consumer getting its own copy of data from the broker). Given this, I do not want the overhead of using…
Paaji
  • 2,139
  • 4
  • 14
  • 11
72
votes
6 answers

Blocking queue and multi-threaded consumer, how to know when to stop

I have a single thread producer which creates some task objects which are then added into an ArrayBlockingQueue (which is of fixed size). I also start a multi-threaded consumer. This is build as a fixed thread pool…
Shivan Dragon
  • 15,004
  • 9
  • 62
  • 103
71
votes
5 answers

How to solve Warning: React does not recognize the X prop on a DOM element

I'm using a thing called react-firebase-js to handle firebase auth, but my understanding of react and of the provider-consumer idea is limited. I started with a built a very big JSX thing all at the top level, and that works without warnings. But…
68
votes
8 answers

Producer/Consumer threads using a Queue

I'd like to create some sort of Producer/Consumer threading app. But I'm not sure what the best way to implement a queue between the two. So I've some up with two ideas (both of which could be entirely wrong). I would like to know which would be…
Gareth
  • 2,180
  • 5
  • 19
  • 24
46
votes
4 answers

How to iterate Queue.Queue items in Python?

Does anyone know a pythonic way of iterating over the elements of a Queue.Queue without removing them from the Queue. I have a producer/consumer-type program where items to be processed are passed by using a Queue.Queue, and I want to be able to…
pgilmon
  • 848
  • 1
  • 7
  • 10
43
votes
7 answers

Job queue as SQL table with multiple consumers (PostgreSQL)

I have a typical producer-consumer problem: Multiple producer applications write job requests to a job-table on a PostgreSQL database. The job requests have a state field that starts contains QUEUED on creation. There are multiple consumer…
code_talker
  • 531
  • 1
  • 5
  • 5
42
votes
2 answers

How does consumer rebalancing work in Kafka?

When a new consumer/brorker is added or goes down, Kafka triggers a rebalance operation. Is Kafka Rebalancing a blocking operation. Are Kafka consumers blocked while a rebalancing operation is in progress?
java_geek
  • 17,585
  • 30
  • 91
  • 113
32
votes
3 answers

Try Dequeue in ConcurrentQueue

The TryDequeue in ConcurrentQueue will return false if no items in queue. If the queue is empty I need that my queue will wait until new item to be added in queue and it dequeue that new one, and the process will continues like that. Should I use…
C-va
  • 2,910
  • 4
  • 27
  • 42
32
votes
7 answers

C# producer/consumer

i've recently come across a producer/consumer pattern c# implementation. it's very simple and (for me at least) very elegant. it seems to have been devised around 2006, so i was wondering if this implementation is - safe - still applicable Code is…
lboregard
  • 329
  • 1
  • 4
  • 3
30
votes
3 answers

Single producer, single consumer data structure with double buffer in C++

I have an application at $work where I have to move between two real-time threads that are scheduled at different frequencies. (The actual scheduling is beyond my control.) The application is hard real-time-ish (one of the threads has to drive a…
25
votes
3 answers

Java how to avoid using Thread.sleep() in a loop

From my main I am starting two threads called producer and consumer. Both contains while(true) loop. Producer loop is UDP Server hence it does not require sleep. My problem is in the Consumer loop. Consumer loop remove the objects from the linked…
Jro
  • 466
  • 2
  • 6
  • 14
21
votes
5 answers

Java BlockingQueue with batching?

I am interested in a data structure identical to the Java BlockingQueue, with the exception that it must be able to batch objects in the queue. In other words, I would like the producer to be able to put objects into the queue, but have the consumer…
Nico Huysamen
  • 10,217
  • 9
  • 62
  • 88
21
votes
5 answers

Is adding tasks to BlockingQueue of ThreadPoolExecutor advisable?

The JavaDoc for ThreadPoolExecutor is unclear on whether it is acceptable to add tasks directly to the BlockingQueue backing the executor. The docs say calling executor.getQueue() is "intended primarily for debugging and monitoring". I'm…
user23987
21
votes
5 answers

RabbitMQ: fast producer and slow consumer

I have an application that uses RabbitMQ as the message queue to send/receive message between two components: sender and receiver. The sender sends message in a very fast way. The receiver receives the message and then does some very time-consuming…
tonga
  • 11,749
  • 25
  • 75
  • 96
19
votes
4 answers

can a kafka consumer filter messages before polling all of them from a topic?

It was said that consumers can only read the whole topic. No luck doing evaluations on brokers to filter messages. It implies that we have to consume/receive all messages from a topic and filter them on the client side. That's too much. I was…
mikej1688
  • 193
  • 1
  • 2
  • 10
1
2 3
99 100