solace - guaranteed ordering with horizontal scalability

Question

So on the non-exclusive queue we have ordered messages, (e.g. id 123 update event will be after id 123 create event). All good when we have one consumer only, but horizontal scalability is desired. By spawning another consumers, I understand that the broker will do round-robin amongst it, so chances are update events can be processed before its create event.

Is there any existing pattern on solace on how to solve this? I come to know that some brokers like kafka and activemq supports it that by specifying the id during publishing it will ensure that all messages with that id will line up only on a one consumer; thus, respecting the order of events.

Answer 1

(an excerpt from my blog that I haven't posted yet)

It is sometimes required that published events or message data containing a particular attribute(s) – e.g. customer ID, order ID, product ID, etc. – are always processed in the original order that they were produced in. This requirement means all related messages are delivered to the same consumer (or possibly group of consumers). That is, messages contain a key such that subsequent messages with the same key are always delivered to the same consumer and processed in order. This ensures that changes or updates regarding the particular attribute are always processed sequentially.

The term "consumer groups" was popularized by Apache Kafka, a log shipping application. In Kafka, consumer groups are groups of consumers that form a "logical consumer" to read from a single topic. A consumer in a Kafka consumer group connects to one or more partitions inside a Kafka topic, and reads sequential log records from a partition file. When records ("messages") are appended to a partition in a Kafka topic, the partition is chosen by a key attribute defined by the publisher.

This provides a form of "sticky load-balancing" such that records with the same key end up in the same partition and are therefore processed by the same consumer.

The same (and better) functionality can be achieved in Solace using a hierarchical topic structure, and Solace’s advanced topic filter capabilities.

Using Solace Topics for Partitioning

When defining your topic hierarchy or taxonomy, designate one level of the topic hierarchy as the partition key. E.g. for an order entry system, your topic structure could be:

estore/order/[ORDER_ID_PARTITION_KEY]/more/specific/rest/of/topic

The key is typically a hash of an important attribute of the published data, as discussed in the leading paragraph. In our example, the keyed attribute would be the Order ID, a large integer. Let us assume for simplicity that the partition key is: Order ID modulo 8: an integer between 0..7, giving 8 possible values, for up to 8 possible partitions.

To configure sticky load-balancing in Solace, configure at least as many queues as the number of consumers in a "consumer group"... let's say two! However, to allow easy future scaling, consider configuring more queues and having consumers bind to multiple queues:

Note the use of Solace multi-level wildcard > at the end of the subscriptions.

In this eStore example, if the customer gateway/API was enhanced to allow different types of order events (e.g. new, amend, cancel), it would be desirable for events related to the same Order ID to go to the same back-end processor. This would ensure a new order isn’t received by one processor, and the cancel got routed to another. Whenever an "order" type event is generated by a publisher, the 3rd level of the topic is used to key the message to a particular partition by taking the modulo 8 (or whatever) of the Order ID.

It is possible to have a very large number of partitions using this approach with essentially no change to the architectural pattern, using topics to define the partition (since topics and subscriptions are "cheap" in Solace). This allows future flexibility by being able to add more consumers into the consumer group by rebalancing the key subscriptions across a new number of queues. Simply start with a fairly large hash key space, or modulo... that way the publisher never has to worry about changing its partition key algorithm!

Note that this approach also allows you to key on two different attributes, using two different levels of the topic hierarchy. One group of queues could look at one level, another group of queues could shard on a different level.

Hope that helps!

Answer 2

The ideas of ordered messaging and horizontal scaling are essentially at odds with each other. In order to achieve order each message must be fully consumed and acknowledged before the next message is consumed. That results in serial message processing (i.e. no concurrency). However, the idea of horizontal scalability is to increase message throughput by adding consumers which can process messages concurrently. As you can see, if you want ordered message processing you can't consume messages concurrently which defeats the entire purpose of horizontal scaling.

ActiveMQ (and JMS in general) supports the idea of "message grouping" which you mentioned to in your question (where all messages in the same group share the same grouping property ID value). The broker selects a single consumer to receive all the messages in a particular group so that the messages are processed serially (i.e. in order). Since only specific groups of messages are ordered this can allow the concurrent consumption of messages from multiple groups. However, the essential first-in-first-out semantics of queue still hold so if you have lots of messages in each group or a small number of groups your overall consumer concurrency will still be fairly low and certainly much lower than if you didn't need order at all. Although I'm speaking here specifically about ActiveMQ there's no real way around this for any message broker so the same would apply to Solace (assuming it even supports message grouping in the first place).

Answer 3

Essentially, what you are looking for is sticky load balancing: Messages with a certain key get assigned to the same consumer while overall messages with different keys can get load balanced across different consumers. Take a look at this blog by my colleague Mat Hobbis, which describes how this can be done with Solace PubSub+: https://solace.com/blog/sticky-load-balancing-in-solace-pubsub-event-broker/