0

So I have multiple producers who generate some tasks

and there is consumer (or multiple) who executes these tasks (f.e. count number of lines in a file)

enter image description here

My problem is that consumers should be treated evenly, meaning if one producer generate 10 tasks and other only 2 - consumer should first do 1 task from producer1, then task from producer2, then producer1 and then the rest

Basically for each producer system must guaranty that created tasks will not wait for large chunk of tasks from other producers

Can you help me with algorithm or ready to use broker/queue software that can achieve this goal ?

Djangonaut
  • 5,511
  • 7
  • 40
  • 53
  • Why not simply one queue (fifo) per producer, i.e. an array of queues. Each queue is examined successively. If a queue is empty, next queue is treated – Damien Jan 24 '21 at 08:47
  • @Damien thanks, yeah that's my thoughts... but is there ready to use solution from RabbitMQ/Kafka/etc. (my other problem is that nubmer of producers are random - one moment it can be one, on other can be 100) – Djangonaut Jan 24 '21 at 11:53
  • I wrote a solution to this over here: https://stackoverflow.com/questions/39914148/what-are-some-queuing-mechanisms-for-implementing-round-robin-queues/40065438#40065438 – Matt Timmermans Jan 24 '21 at 14:36
  • @PydevUA if you're using the Spring framework you can try https://github.com/sonus21/rqueue – sonus21 Jan 24 '21 at 16:57

0 Answers0