1

We have made an architecture change and now each streaming message is duplicated 5 time on average. Like 5 more messages is a lot and is impacting performance.

  • Before we were directly connecting to the capture blob storage of a third party event hub (this event hub had 32 partitions).
  • Now we have an azure function connected to a third party event hub. This azure function is pushing the message in our event hub . And we are using the capture from our event hub . Our event hub has only 3 partitions (we followed the microsoft recommendation for partition number)

I am aware that the topic of duplicates and event hub duplicates has been discussed extensively (see links below). And i am still puzzled by the number of duplicates i am getting. Is it expected that in , each message could be duplicated 5 time in average?

Our Throughput Unit is 1 , auto inflate 3. Partition number is 3.

Function code is below :

using Microsoft.Azure.EventHubs;
using Microsoft.Azure.WebJobs;
using SendGrid.Helpers.Mail;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;

namespace INGESTION
{
    public static class InvoiceMasterData
    {
        [FunctionName("InvoiceMasterData")]
        public static async Task Run([EventHubTrigger("InvoiceMasterData", Connection = "SAP_InvoiceMasterData")] EventData[] events,
                                     [EventHub("InvoiceMasterData", Connection = "Azure_InvoiceMasterData")] IAsyncCollector<EventData> outputEvents,
            [SendGrid(ApiKey = "AzureSendGridKey")] IAsyncCollector<SendGridMessage> messageCollector, ILogger log)
        {
            var genericFunctionStopper = new GenericFunctionStopper();

            await genericFunctionStopper.Loaddata(outputEvents, "InvoiceMasterData", messageCollector, log, events);
        }
    }
}

Sharing the event hub metrics below : enter image description here

And the function metrics : enter image description here

We also observe a 2nd unexpected behaviour, that i would like to share (maybe i should ask in another question).

Before, with the old architecture, we never had the same EnqueuedTimeUtc for each invoice primary key. Now with the new architecture, using the function, it happens all the time. This is a problem because we were using the EnqueudTimeUtc to deduplicate. Is it because somehow we are treating the messages in batch? Is it because our partition number is less ?

Any suggestion , observation, expertise would be much appreciated !

https://learn.microsoft.com/en-us/azure/azure-functions/functions-reliable-event-processing
Azure functions with Event Hub trigger writes duplicate messages
Azure Functions Event Hub trigger bindings
https://github.com/Azure/azure-event-hubs-dotnet/issues/358

OrganicMustard
  • 1,158
  • 1
  • 15
  • 36
  • What is the througput in terms of messages per sec? Because if you are routing all the messages of an event hub with 32 partitions to another one having just three with a very high message rate you are bound to get into performance issues, – Peter Bons Dec 02 '21 at 08:27
  • 1
    Can you share any insights in the amount of scaling operations that take place on the function?Could be it isn't stable and instances are fighting over the partition ownership causing checkpoints being not as effective as they should be. – Peter Bons Dec 02 '21 at 08:29
  • Thanks Peter, i shared some metrics about througput , hope it partially answer your question. Will try to find more information – OrganicMustard Dec 02 '21 at 09:36
  • 1
    To Peter's point - if you're pushing 32 partitions into 3, then there's a high degree of likelihood that you'll be throttled unless you're using an Event Hubs dedicated instance. When the throttle is triggered, you're very likely to see timeouts for other requests clustered around that time. Those timeouts create an ambiguous outcome, where the producer will retry to ensure no data loss but there's a decent chance the event already exists on the broker. – Jesse Squire Dec 02 '21 at 15:35
  • thanks for your answer. I don't see any throttled messages in our event hub metrics only successful request. Do you mean throttled from the event hub third party's side ? – OrganicMustard Dec 02 '21 at 23:08
  • 2
    I'm referring to Event Hubs itself. I'm not sure what messages/metrics you're referencing, but the SDK will automatically retry after respecting the throttle back-off, so the application wouldn't be seeing errors. Likewise for timeouts. Capturing SDK logs would expose them. – Jesse Squire Dec 03 '21 at 14:46

0 Answers0