13

When creating a new Azure IOT Hub you are asked how many device-to-cloud partitions you need. You can select between 2-32 partitions for standard tiers.

I understand that the SKU and number of units determine the maximum daily quota of messages that you can send to IOT Hub. And that it is recommended to shard your devices into multiple IOT hubs to smooth traffic bursts. However, device-to-cloud partitions need clarification.

1>> What is the purpose of those device-to-cloud partitions under a single IOT hub?

2>> How are we supposed to take advantage of those IOT Hub device-to-cloud partitions? 

Thanks.

GHariz
  • 336
  • 2
  • 11
  • 1
    Explanation of partition from stackoverflow: "Partitions are used internally to allow scaling of the IoT (Event Hub) and allow for scaling out the consumer app (the one that reads the events out of the Hub). I have read this explanation and the true purpose of those partitions is still not clear. If they are used internally, then why should we set their size? How can the consumer app (receiving the messages) use those partitions? An example would be helpful. – GHariz Apr 07 '17 at 07:05

1 Answers1

9

1>> What is the purpose of those device-to-cloud partitions under a single IOT hub?

Partition property is setting for Event Hub-compatible messaging endpoint(messages/events) built in Azure IoT Hub. From here we can see "partitions" is a concept belongs to Event Hub.

Event Hubs is designed to allow a single partition reader per consumer group. A single partition within a consumer group cannot have more than 5 concurrent readers connected at any time. More partitions enables you to have more concurrent readers processing your data, improving your aggregate throughput.

Ref: Built-in endpoint: messages/events and How many partitions do I need?

2>> How are we supposed to take advantage of those IOT Hub device-to-cloud partitions?

Event Hubs has two primary models for event consumption: direct receivers and higher-level abstractions, such as EventProcessorHost. Direct receivers are responsible for their own coordination of access to partitions within a consumer group.

Ref:Event consumers.

More information about the partitioning model of Azure Event Hubs are here.

SimonSimCity
  • 6,415
  • 3
  • 39
  • 52
Rita Han
  • 9,574
  • 1
  • 11
  • 24
  • Hi @Rita I have a question. If I created an Iot Hub with 2 partitions, can I have 3 consumer groups reading the 2 partitions? – Andres Camacho Jul 09 '18 at 15:13
  • @AndresCamacho Yes, you can use 3 consumer groups to read 2 partitions and any consumer group can read all 2 partitions. There are no direct relationship between consumer group and partition. – Rita Han Jul 10 '18 at 06:34
  • I don't understand when you answered to GHariz "Event Hubs is designed to allow a single partition reader per consumer group.", what I undestood is that if I have two partitions in the Iot Hub, each partitions only can be read by one consumer group, so if I have three consumer group the third consumer group can't read any partition. – Andres Camacho Jul 10 '18 at 16:43
  • @AndresCamacho In another word, you can't concurrently read from two instance using one consumer group. For example, if you use explorer to monitor D2C message in consumer group "group1" while when you read D2C message from another program in same consumer group "group1", you will get an error. To solve it you need create a new consumer group for second program. – Rita Han Jul 11 '18 at 10:14
  • I still don't understand, what do you mean with "two instance"? In fact, I currently have three apps reading data of an Iot Hub and the three apps are in the same consumer group but if I exceed five applications, now I will get an error; so in your example of one explorer monitoring D2C, there would be no an error if another program begins to read data of the Iot Hub because only are two readers. If you said that there is no a relation between consumer group and the partitions, what is the function of the partitions if I could have "10" consumers groups reading a single partition? – Andres Camacho Jul 11 '18 at 13:54
  • 1
    This the latest on the subject of partitions: There can be at most 5 concurrent readers on a partition per consumer group; however it is recommended that there is only one active receiver on a partition per consumer group. Within a single partition, each reader receives all of the messages. If you have multiple readers on the same partition, then you process duplicate messages. You need to handle this in your code, which may not be trivial. However, it's a valid approach in some scenarios. Source: https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-features – GHariz Oct 23 '18 at 10:22