1

We are an ActiveMQ setup with all default settings. Its broker url is like failover:(tcp://host1,tcp://host2,tcp://host3)?randomize=false

We have 2 message Queues where producer publishes persistent messages.

Messages generally have a steady flow but when markets open then there is burst of messages for almost 2 hours. This setup works fine for 2-3 days and then consumer slows down in processing messages and we see queue build ups.

Consumer performs following steps:

  1. Transform message.
  2. Process it
  3. Add to cache and DB.

Prefetch size is 1000.

Following are some of the observations:

  1. Consumer processing doesn't seem to be the issue else it would have always slowed down.
  2. Looks like ActiveMQ broker slows down in processing but i dont have evidence
  3. We have observed that out of 3 hosts only host 1 and 2 process majority of messages.

Any ideas on how to explain this slowness?

EDIT:

Analysed this further and found that Kahadb store [Persistent messages are stored in Kahadb in broker] reaches 100% storage when consumer slows down. I can understand that producer is blocked when storage reaches 100% but why would consumer slow down? Is it because Kahadb may slow down during this time and extracting messages equivalent to prefetch size takes more?

Lokesh
  • 7,810
  • 6
  • 48
  • 78
  • What is observed to be slow? What's the CPU load on each host? What happens after rebooting consumer / brokers? Do you have the possibility to profile? Can you reproduce the issue on other machines? – mm759 Sep 18 '16 at 15:35
  • Message consumption is observed to be slow. During slowness, CPU load doesn't shoot up. If we reboot consumer, slowness goes away [we haven't rebooted broker]. Issue is reproducible on QA as well as Prod machines. – Lokesh Sep 18 '16 at 15:38
  • Sounds like a problem with the consumer. Why do you suspect the broker? Isn't it possible that the consumer slows down over time? Maybe locking issues? Or OutOfMemoryError? Any not persistent state that could have influence? Is it possible that the cache slows down for example? – mm759 Sep 18 '16 at 15:46
  • Consumer has fixed set of steps to do, which remain constant throughout. It seems reconnection to broker solves the issue. We have checked and sufficient memory is available – Lokesh Sep 18 '16 at 15:48

0 Answers0