0

I have two Services Producer and Consumer (both spring-boot) running on different servers. The idea is to produce aggregated data from multiple sources and publishing messages. The consumer then listens to them and does some processing.

I am using RabbitMQ between these services. The problem is sending messages at 20K/s in dynamic bursts (30-second burst + 3-4 seconds idle). The consumer is working around 10-15K/s. Due to this, there is a congestion and messages are getting piled up in RabbitMQ.

I cannot horizontally scale the system due to some constraints on Consumer. So I want to slow down the publishing rate depending on the consumption rate dynamically. This is critical as there is a requirement to add another producer in near future . I am using a common Redis and MongoDB and can use them for implementing this. I've heard of Back pressure, but never built anything using that. Can that help? Please suggest an approach. Thank you.

PS : I have read this and solutions present there do not work for me.

Pardha.Saradhi
  • 468
  • 1
  • 10
  • 27

1 Answers1

0

Two things i can think of

  1. Circuit Breaking
  2. Batch Size Based Publish

With Circuit Breaking, you can signal the producer to stop publishing. Build a monitoring micro-service, to monitor the consumer queue size on a periodic basis. Have a higher/lower watermark for the number of messages that can be the piled up in the rabbit mq queue. Once it hits a higher watermark signal the producers to stop and do the opposite on lower water mark. Since you are using rabbit mq, use a fan-out exchange to publish this event of circuit breaking. Have all you producers listen the messages on fan-out exchange and do the needful.

Publishers can use batching so that they publish at constant rate rather than dumping all the data into rabbit. Have this batch size configurable based on the number of producers your have.

Srikanth
  • 1,015
  • 12
  • 16
  • One problem is publishers and consumers get added and removed dynamically. So I need some "ServiceRegistry" kind of thing to monitor which (Adding another service) I can't do. – Pardha.Saradhi Jan 05 '18 at 10:37
  • The way i solved this problem was to use a routing key with topic exchange. Publishers publish message to routing key and consumers bind queues to the routing key. This way publishers and consumers are detached. You can add any number of consumers, they will all get message as long as they listen on the routing key, same applies to the producers too. This way i did not need an additional service registry - when rabbit is around. – Srikanth Jan 06 '18 at 07:44