3

I have a Camel route that needs to split a large file (600k lines of id's) into 600k individual messages and then push them onto an Activemq queue. How do I optimize the route from the Camel side to increase throughput? I'm currently achieving ~150 messages/second throughput to AMQ. Here's what the route looks like currently. Any suggestions are appreciated!

from("file://directory")
  .split().jsonpath("$.ids").streaming().parallelProcessing()
  .log(LoggingLevel.INFO, "Split: ${body}")
  .to("activemq:queue:myqueue");
David
  • 579
  • 1
  • 5
  • 20
  • 2
    Since you do not need XA transactions, make sure you are using `PooledConnectionFactory`. – Bedla Jul 10 '18 at 21:50

1 Answers1

1

First things first, as pointed out by @Bedla, Pool your connection (i.e. wrap your connection factory inside a org.apache.activemq.pool.PooledConnectionFactory).! It will most likely give you a throughput boost in the range of x10 to x100 depending on network conditions, message size etc. More for smaller messages.

Then, when hunting throughput, dumping each of the 600k lines into your log file will do no one no good. Remove it or at least put it to trace/debug level.

If your broker is located elsewhere, like other part of the world, or a place with bad network latency in general, consider using async dispatch on the ConnectionFactory setting. It will not wait for the roundtrip of a confirmation of each message sent.

Then finally, if none of the above give decent result (I think just a pool should do) turn off message persistence. The broker disk might be a bottle neck for low spec/old servers. There are even tweaks to enhance certain OS/storage combos for performance.

Petter Nordlander
  • 22,053
  • 5
  • 50
  • 84
  • Thanks Petter, I'll give this a shot and see if it help. The broker and camel app are within the same cluster so that shouldn't be an issue. Unfortunately turning off persistence is not an option. – David Jul 10 '18 at 23:08
  • 1
    Anyway, I made a quick try and switching to PooledConnectionFactory gave somewhere around 200x throughput for small messages on a local machine broker. I made a more extensive answer for a similar question six years ago here: https://stackoverflow.com/questions/11840630/high-performance-jms-messaging/11842794#11842794 – Petter Nordlander Jul 11 '18 at 15:25