7

I have set up Azure Event Hub. There are 2 publishers:

  1. Publisher 1 (with Send Policy)

  2. Publisher 2 (with Send Policy)

Publisher 1 will send Event 1 and Publisher 2 will send Event 2. Both Event 1 & Event 2 are different format.

Question 1: This would mean we have different messages in EH - what are the tradeoff with this approach? Should I instead create 2 EH (one for Publisher 1 and another for Publisher 2)? What is the best practice and design philosophy?

If I go with above approach, I would have to set up a Consumer with Listen Policy to look for these events and parse/transform these events and de-serialize these.

Question 2: Would I need 2 Consumers (Consumer 1 and Consumer 2) to read messages meant for them ? (Consumer 1 will read only Event 1 and Consumer 2 will read only Event 2)?

khar
  • 201
  • 4
  • 12

1 Answers1

10

Scenario 1: One Event Hub for multiple event types

In this scenario you have multiple options when it comes to sending and processing the messages:

  1. Just send messages to the Event Hub. Write a consuming process that reads the messages and based on the type handle them different.
  2. Just send messages to the Event Hub. Create different consumer groups, one for each message type. Have 2 processes that both read their own consumer group and skip the messages that they are not qualified to handle. (Each consumer group essentially read the same data, but they can be at different positions in the data stream). See Azure event hubs and multiple consumer groups
  3. Send the messages to a designated partition. For example, have messages of type A send to partition 1 and messages of type B to partition 2. This can however affect the scalability of the event hub. Create dedicated processes per (set of) partition(s). Each process will then only process messages of the same type. See https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-programming-guide#partition-key and https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-programming-guide#create-a-partition-sender

Scenario 2: An Event Hub per event types

This one is easy, just create 2 event hubs and 2 consuming processes. But you'll have to manage 2 event hubs and given the abilities of an Event Hub it might be overkill.

My advice

It depends on the amount of data, but based on my experience I would go to send all messages the one event hub and have one process reading the message and perform an action based on the message type using some c# code.

Peter Bons
  • 26,826
  • 4
  • 50
  • 74
  • Thanks @peter-bons - this is super useful - you read my mind :) . If we go with Scenario #1 and Option 2 ( _Create different consumer groups for 1 EH_), it looks i will have to **SKIP** many messages not meant for a consumer group. Are there any **performance trade offs** here? Also, EH is based on **Throughput Units (TU)**, how will this be impacted if we go with this option (or any option) **Would TU be impacted **? – khar Sep 22 '17 at 21:09
  • Correct, you need to skip messages in a consumer group. About the TU, how many messages and what size per message are we talking about? We have hundreds of events coming in per second but still use just 1 TU so I wouldn't worry about that much. Now, I am not an expert on TU / billing in the EH but to me it seems you are not getting more data in nor out when you decide to go for the chosen option. – Peter Bons Sep 23 '17 at 09:38
  • Thanks Peter. I am expecting around 50 events per sec and so not sure which approach will work out best without impacting performance and TU – khar Sep 25 '17 at 04:47
  • 1
    With that amount you really, really do not need to worry about performance in terms of TU. 1 TU, the minimum, is.more than enough. Remember, every TU gives 1mb ingress and 2 mb egress per second. Eventhub is designed to handle up to a million events per second. – Peter Bons Sep 25 '17 at 04:51