Messaging, Queues and ESB's - I know where I want to be but not how to get there

Question

To cut a long story short, I am working on a project where we are rewriting a large web application for all the usual reasons. The main aim of the rewrite is to separate this large single application running on single server into many smaller decoupled applications, which can be run on many servers.

Ok here's what I would like:

I would like HTTP to be the main transport mechanism. When one application for example the CMS has been updated it will contact the broker via http and say "I've changed", then the broker will send back a 200 OK to say "thanks I got the message".

The broker will then look on its list of other applications who wanted to hear about CMS changes and pass the message to the url that the application left when it told the broker it wanted to hear about the message.

The other applications will return 200 OK when they receive the message, if not the broker keeps the message and queues it up for the next time someone tries to contact that application.

The problem is I don't even know where to start or what I need to make it happen. I've been looking at XMPP, ActiveMQ, RabbitMQ, Mule ESB etc. and can see I could spend the next year going around in circles with this stuff.

Could anyone offer any advice from personal experience as I would quite like to avoid learning lessons the hard way.

Answer 1

I've worked with JMS messaging in various software systems since around 2003. I've got a web app where the clients are effectively JMS topic subscribers. By the mere act of publishing a message into a topic, the message gets server-pushed dissemenated to all the subscribing web clients.

The web client is Flex-based. Our middle-tier stack consist of:

• Java 6
• Tomcat 6
• BlazeDS
• Spring-Framework
• ActiveMQ (JMS message broker)

BlazeDS has ability to be configured as a bridge to JMS. It's a Tomcat servlet that responds to Flex client remoting calls but can also do message push to the clients when new messages appear in the JMS topic that it is configured to.

BlazeDS implements the Comet Pattern for doing server-side message push:

Asynchronous HTTP and Comet architectures An introduction to asynchronous, non-blocking HTTP programming

Farata Systems has announced that they have modified BlazeDS to work with the Jetty continuations approach to implementing the Comet Pattern. This enables scaling to thousands of Comet connections against a single physical server.

Farata Systems Achieves Performance Breakthrough with Adobe BlazeDS

We are waiting for Adobe to implement support of Servlet 3.0 in BlazeDS themselves as basically we're fairly wedded to using Tomcat and Spring in combo.

The key to the technique of doing massively scalable Comet pattern is to utilize Java NIO HTTP listeners in conjunction to a thread pool (such as the Executor class in Java 5 Concurrency library). The Servlet 3.0 is an async event-driven model for servlets that can be tied together with such a HTTP listener. Thousands (numbers like 10,000 to 20,000) concurrent Comet connections can then be sustained against a single physical server.

Though in our case we are using Adobe Flex technology to turn web clients into event-driven messaging subscribers, the same could be done for any generic AJAX web app. In AJAX circles the technique of doing server-side message push is often referred to as Reverse AJAX. You may have caught that Comet is a play on words, as in the counterpart to Ajax (both household cleaners). The nice thing for us, though, is we just wire together our pieces and away we go. Generic AJAX web coders will have a lot more programming work to do. (Even a generic web app could play with BlazeDS, though - it just wouldn't have any use for the AMF marshaling that BlazeDS is capable of.)

Finally, Adobe and SpringSource are cooperating on establishing a smoother, out-of-the-box integration of BlazeDS in conjunction to the Spring-Framework:

Adobe Collaborates with SpringSource for Enhanced Integration Between Flash and SpringSource Platforms

Answer 2

First of all, don't worry about ESBs. The situation you've described lies well within the bounds of straightforward message-oriented middleware. You only "need" an ESB if you're doing things like mediations, content-based routing, protocol transformations; things where the middleware does stuff to the message, on top of routing it to the right place.

If you have a diverse set of destination applications that need to speak to each other - and it sounds like you do - you're right that messaging over a language agnostic protocol (like XMPP, STOMP or HTTP) is a neat solution. It basically means you don't have to write and run loads of Java daemons to translate messages into your favourite flavour of JMS.

STOMP is increasingly supported by message brokers, especially by the open-source ones, and there's a number of different client libraries. It is a lightweight protocol, specifically designed for messaging so you get a much richer feature set out of the box than you would with HTTP.

For me, XMPP is a bit of a weak option as it's not so well supported on the server side, although it is fun to be able to IM your broker :)

If you are set on HTTP, OpenMQ is very good, and I've personally used its Universal Message Service - basically a webapp wrapper around JMS destinations. It provides a REST-ful interface, with a similar set of verbs as STOMP provides.

Answer 3

As someone has already said, what your describing is basically the Publisher/Subscribe Model. This is very easily achieved using either an ESB or a message queue. I have had some experience with RabbitMQ. Its very good. Nothing gets lost and it deals with the publish subscribe model very well. I have in the past gone down the route on small scale systems of developing my own Message broker with a bespoke protocol over http. I wouldn't advise this, reason being is that as you start to develop it you keep thinking of ways of how to extend it.

RabbitMQ is developed in Erlang but it has java,net,python etc clients that can hook into it very easily. I have used the .net and python clients, it works well. I chose it for Erlangs reputation for creating solid systems that can cope with multiple things going on at the same time, very well. I would call them threads but I think that its smarter than just threads, I think I remember mutterings of the Actor Model and mailboxes, which I recall were pretty neat.

I was in a similar position as yourself but with very bad experiences of other messaging systems (Biztalk et al.) that were too propriety that tied you into a solution. If you can keep the messages separate from the transport and delivery mechanisms, then you can develop your system to your hearts content. I used JSON in the end as the packet sizes are small. You could use anything you like, some opt for SOAP messages, but I feel that these are way too heavy for most stuff, although it does allow you to nicely give XSD schemas to outsiders so that they can/could develop systems that interop with your system in the future.

http://www.rabbitmq.com/tutorials/tutorial-three-java.html, this is a link to the tutorial on the Publish/Subscribe model and how you would achieve it using a message queue system. Its for rabbitMQ, but to be honest it will work with ESB and any other Messaging queue system out there.

Answer 4

ESB (Enterprise Serial Bus) - Consider this when your application have much interaction with two or more external/separate applications where each of these won't communicate in a similar data format. Ex: Some systems may accept objects, XML, JSON, SMTP, TCP/IP, HTTP, HTTPS etc.

ESB has many features like: Routing,Addressing,Messaging styles,Transport protocols,Service messaging model.

Consider queue system if the producer - consumer applications follows the same type of data format.

Web services (SOAP / REST) is best if one application need the other application to complete the work flow. Use Queues if the application need asynchronous data transfer.

Answer 5

As it was already said earlier, having an ESB for you current case seems to me like to smash a fly with a hammer.

The ESB software itself will be time consuming and will require maintenance. If you go to open source solution, it might be more time consuming than using a licensed solution (IBM, ORACLE, ...).

Of course an ESB would do the job, and it would be really easy to develop a solution, but setting up an ESB would be way more difficult than doing the solution itself.

If your problem is limited to the case described, I would highly suggest you to build a simple architecture over OpenMQ (or similar), and using it through JMS

Answer 6

You're really talking about publish and subscribe with assured delivery. Most MOM software should easily support your use case.