5

I have an third-party application that puts some messages to JMS Queue. Also I have an application that reads the messages from this queue. Depending on the type of the message I save this message to DB or send it to the third-party service. Also we shouldn't exceed some fixed calls per second limit not to overload the third-party.

Currently, two solutions came to my mind for this use case.

The first one is to ask the third party to send some custom headers, so that JMS consumer will be able to filter messages using JMS Selectors. So, in this case we'll be able to create two consumers, the first one will be able to read messages and save them to DB, and the second one will use some throttling/polling mechanism to send messages to third party at particular load. But this approach won't work for me cause it will take ages for a third party to add those custom headers. Something like this in Camel:

from("jms:queue?selector=firstSelector")
    .bean(dbSaver);

from("jms:queue?selector=secondSelector")
    .throttle(10)
    .bean(httpClient);

The second one is to create another two JMS Queues and a processor that will split the messages between these queues. Then goes the same logic as in first solution. However, this means that 2 extra JMS queues should be added. In Camel:

from("jms:parentQueue")
    .choice()
        .when(body().contains(...))
            .to("jms:fistChildQueue")
        .otherwise()
            .to("jms:secondChildQueue")
    .end()

from("jms:fistChildQueue")
    .bean(dbSaver);

from("jms:secondChildQueue")
    .throttle(10)
    .bean(httpClient);

Also, I've been thinking of using two in-memory queues instead of JMS Queues. However, in this case if there'll be a lot of messages in JMS Queue we can easily get into troubles with memory.

Could anyone suggest an architectural design for this use case? It would be great to see it in Camel Route style.

StasKolodyuk
  • 4,256
  • 2
  • 32
  • 41
  • Having two additional queues is not a big deal, JMS brokers are made to handle thousands of queues. That being said, Darius's option #1 does make more sense if you don't want to do any batching of inserts. – Miloš Milivojević Aug 12 '16 at 10:21

1 Answers1

2

1. Do you really need a queue for the flow to the DB? You could have bean(dbSaver) in the first route, or abstract it into a "direct" route instead of a jms-consuming route. This way, you have two queues instead of three.

2. A second approach: If you have control over the DB, you could write the second type of message to a different table. Then, an sql-consumer can poll for records, and delete them as it consumes them and passes them onward to the http service. However, the table is acting like a "roll your own Q". Probably more work for little payback, so maybe a second queue is better.

3. Finally, I wonder if you could reuse the same Queue. I see an option that will allow you to write back to the same queue. You could add a header and write back certain message. This could look confusing, and a bug could create an infinite loop.

If you already use JPA, this could be made a bit easier by using the camel-jpa component. As a consumer, it acts reads and deletes records (default behavior). I don't think the SQL/JDBC components have anything like that out of the box.

Darius X.
  • 2,886
  • 4
  • 24
  • 51
  • thanks for reply. The first option seems like the most suitable for now. Also I've thought about the third option but it can lead to really confusing cumbersome code but in case Camel provides this kind of functionality out of the box that'd be great. May be you can help me with this? – StasKolodyuk Aug 11 '16 at 08:17
  • If you use JPA, it would make #3 easier. I've edited the answer, – Darius X. Aug 11 '16 at 12:03
  • I'd go with #1, it's the most sensible and scalable one – Miloš Milivojević Aug 12 '16 at 10:16