1

I would appreciate some help picking out the best suited Azure services for my scenario - I am just beginning with Azure services and my knowledge is pretty limited.

I have data from multiple sources, and of different shapes, coming into an Event Hub. I need to subscribe to the events from the Event Hub and, based on their format, process them and ultimately save them into an SQL Database. All components - events consumers, the SQL Database - need to be hosted in the cloud.

How would I implement this in an "Azure Orientated Architecture"?

In an off cloud application, I would have competing consumers subscribing to the Event Hub. They would be some console applications or Windows services, and each would be processing the events asynchronously (this is further simplified by the event processing being idempotent).

Ideally, the Azure equivalent of the above consumers would scale up and down automatically, so I would like to not have to use VMs that host console applications (where I would need to keep an eye on the VM's resources myself). Scaling and deployment wise they would have to behave like App Services, however I'm under the impression that those are just for web applications. I've also briefly looked at Web Jobs, but those seem to be polling data at various intervals, whereas I need a proper event subscriber that the Event Hub pushes data into.

Any help will be greatly appreciated! Thank you.

Later Edit:

  1. I've looked into Web Jobs and they do allow continuous processing, hence looks like they can be used as automatically scaling subscribers.
  2. Ideally I would like to write the code for the subscribers in F#. C# is the other option if that is not available.
user2916547
  • 1,963
  • 5
  • 17
  • 21

2 Answers2

2

You can see my post regarding IoT Hub. Its basically the same for Event Hub. (each of the examples in the post can be used on Event Hubs). https://stackoverflow.com/a/38682324/6659347

In addition, For Event Hub you can also use Azure Function which has an Event Hub trigger - a function that will run whenever an event hub receive a new event. And it will also answer your requirement of scaling.

Make sure that if you are working with multiple consumers make use of the Event Hub Consumer Groups so each consumer can read the stream independently.

Community
  • 1
  • 1
shachar
  • 589
  • 3
  • 12
  • Thank you. I have looked at the options and the Stream Analytics imply that I have to do the processing of the events in the SA query language, isn't it? I would like to version control the code for the processing and make it more involved than what the QL allows. – user2916547 Aug 08 '16 at 10:56
  • The worker role (the 2nd option) - don't know much about it, but read it is being replaced by Web Jobs, so would rather use the continuous WebJob. The Storm option does look too expensive. The Functions sound great, however F# is not supported yet. Unless something more suitable exists, will go for these with C# code, or the continuous WebJobs. – user2916547 Aug 08 '16 at 10:59
  • @user2916547 - Regarding SA, yes you will need to use the QL for processing the data. If you don't actually need analytics on the stream itself, you can use the QL only to save the data into a DB and make your own queries. Azure Function is a good choice because they already have the built in listener for event hub in C#. – shachar Aug 08 '16 at 11:07
  • Unfortunately Stream Analytics QL is too restrictive for what I need. – user2916547 Aug 08 '16 at 11:50
  • Thank you for answering; eventually I will go for the WebJob approach, so I have marked Peter Bons's answer as the solution, but your answer was nevertheless instructional. – user2916547 Aug 09 '16 at 12:15
1

I'd say use a WebJob in combination with an EventProcessor. I wrote some demo code that can easily be transferred to a WebJob: https://github.com/DeHeerSoftware/SemanticLogging.EventHub/tree/master/SemanticLogging.EventHub.Processor

See https://azure.microsoft.com/en-us/documentation/articles/event-hubs-csharp-ephcs-getstarted/#receive-messages-with-eventprocessorhost for official documentation.

I've created a WebJob myself using this approach. Works like a charm.

Peter Bons
  • 26,826
  • 4
  • 50
  • 74