3

I would like to have this constraint on a queue in RabbitMQ:

Next message in the queue can't be dequeued before previous message (the one being processed) is acked.

Through this I will achieve ordered processing of events and parallel processing across multiple queues. How do I/can I configure RabbitMQ for this?

Edit (clarification): There will be many consumers all trying to get work from all the queues and since they can't get work from a queue that has an event being processed that isn't acked - ordered processing is maintained.

ytoledano
  • 3,003
  • 2
  • 24
  • 39

1 Answers1

4

Next message in the queue can't be dequeued before previous message (the one being processed) is acked.

you can do this through the consumer prefetch limit for a single consumer.

Through this I will achieve ordered processing of events and parallel processing across multiple queues.

unfortunately, this won't have the effect that you want.

you can set for an individual consumer. that consumer will wait for a message to be acknowledged before getting the next one.

However, this applies to the individual consumer, not the queue.

if you have 2 consumers, each of them will process a message in parallel. if you have 10 consumers, 10 messages will be processed in parallel.

the only way to process every message in order, is to have a single consumer with a prefetch of 1.

Derick Bailey
  • 72,004
  • 22
  • 206
  • 219
  • I will add that having queue declared as `exclusive` may do the job in some cases. In this case queue can be limited to connection it was created in, so consumers number can be easily controlled. In addition to it Dead Lettering and Alternate Exchanges may be used to catch missed messages to somewhere. – pinepain Sep 07 '15 at 12:22
  • Thank you for your answer. I have added a clarification to the question. There will be many workers all trying to dequeue from all queues. This is to achieve parallelism and availability. So having having the constraint being on the consumer isn't sufficient, it needs to be on the queue. – ytoledano Sep 07 '15 at 12:41
  • 1
    the answer remains the same. it can't be done with multiple consumers. even if you have an exclusive queue, having multiple consumers within that connection is still multiple consumers. you will either have parallel processing, or you will have ordered processing. this is not a limitation of rabbitmq, but of distributed systems / messaging in general. no system supports this because it is not possible. – Derick Bailey Sep 07 '15 at 13:58
  • there are several other SO questions and articles around the web about this: http://stackoverflow.com/questions/21363302/rabbitmq-message-order-of-delivery and http://camel.apache.org/parallel-processing-and-ordering.html as examples - as that second article suggests, you may be able to use something like a resequencer... but that will prevent parallel processing – Derick Bailey Sep 07 '15 at 14:02