2

Is there a way to delay the consumption of a message in an Enterprise Service Bus until either X similar messages are ready or Y time elapsed?

My goal is to serialize and aggregate several messages in the consumer so I don't hit my storage for a message, but for a batch instead.

I'm currently playing with Rhino ESB. I would like to know my options with this ESB or any other you know about.

Thanks!

Raciel R.
  • 2,136
  • 20
  • 27

1 Answers1

0

In NServiceBus you can send a batch of messages in a single call. This will process as if each message was send individually. The other option is to use a Saga, or long-running workflow that is built-in. You can uniquely identify and correlate message state and leverage a timeout mechanism that keeps track of time.

Adam Fyles
  • 6,030
  • 1
  • 23
  • 29
  • @Adams, I can't implement the first option. Imagine I have several (tens to hundreds) of devices consuming a REST endpoint sending information that I would drop in the ESB. So, at no point in time I will have a batch of messages in the sender, which I know I can send with NSB or Rhinos ESB if that were the case. Ideally, I was thinking on something that allows me aggregate the messages in the consumer. Like "just consume messages if there are 100 in the queue". I have to check the Saga alternative. I'm new to ESB, would you mind elaborating on how would you do it? – Raciel R. Jan 31 '12 at 23:02
  • Before I answer can you elaborate on why you want to throttle your storage? If you are using messaging, the processing is now "offline" and shouldn't matter if it takes a few seconds or minutes even. – Adam Fyles Feb 01 '12 at 13:40
  • The destination table is huge, blocking it for writes is expensive and the number of messages to be written really high. This table is also heavily queried. I know I can work with the lock types I use to write, but the _ideal_ scenario in my head is to write batches instead of single writes. I was reading this article http://msdn.microsoft.com/en-us/magazine/ff796225.aspx by Ayende, and I think I can write a message module that delay the messages to be sent, and force the messages to be consumed once either a certain number are queued or after certain elapsed time. – Raciel R. Feb 01 '12 at 14:18
  • 1
    I'm a little wary of choking down the writes because in the long term with more data your timeout will just increase. Other options include sharding the huge table to make writes easier or completely segregate reads from writes. In this pattern after a write is completed, the read store(could be a cache) would require a message to be updated as well. In either case you would be set up for further scale out. – Adam Fyles Feb 02 '12 at 13:44
  • I just implemented a small service that is batching the messages every X minutes using MSMQ as the transport. I agree with you about the sharding or rw seggregation; but we are not there yet. Thanks a ton for your feedback. I'm marking your answer as accepted however I still believe _IMHO_ that ESB is appropiated for cases when the publisher and the subscriber exchange one message at a time. I will keep reading. – Raciel R. Feb 02 '12 at 14:41