6

We want to migrate our dedicated servers to Azure platform for scaling easy and investigated a lot of Azure services for our needs. So one of the Azure service that we want to use is Azure Stream Analytics (ASA).

We've added some Azure Platforms according to our needs for performing some tests (it is not important what they for, for now). Here is the structure:

SimpleApp (Sending Request, Not In Azure) -> Event Hub 1 (EH1) -> ASA -> Event Hub 2 (EH2) -> Function App (FA)

  • SimpleApp sends a simple HTTP GET request to classic dedicated server which is named TESTSERVER. It tooks max 100-150ms and it represents our start time. After that it sends the message to EH1.
  • ASA's query is simple like this: SELECT * INTO [Output] FROM [Input]
  • Function App sends a simple HTTP GET request to TESTSERVER for identifying finish time.

We've shocked when we see the results from our TESTSERVER logs. It tooks 4000-5000ms!

Then we started to investigate the issue. Checked values like EventEnqueuedUtcTime and EventProcessedUtcTime to identify which block causes this slowness. But these time values are totally irrelevant. For example; EventEnqueuedUtcTime should be less than EventProcessedUtcTime but not! So this shows us time servers may be different even in different Azure blocks and we cannot use them to measure. Am I wrong?

Anyway, after this we suspected that maybe the last Azure Function App may cause this slowness. We thought that maybe Function App's Event Hub Trigger does not work well. So we designed a new test environment:

SimpleApp (Sending Request, Not In Azure) -> Event Hub 1 (EH1) -> Function App (FA1) -> Event Hub 2 (EH2) -> Function App 2 (FA2)

Second shock... It tooks only ~400ms totally!

Then, we've performed a lot of tests with different architecture which contains ASA but all of them are too slow for us.

Have you experienced any performance issues with ASA? Could you please share your experience and your flows' total time consumption?

Best regards.

msapcili
  • 63
  • 1
  • 6
  • 1
    what do you try to achieve with SA ? I can't figure out why you have a SA in the middle of your flow. – Thomas Jun 25 '16 at 04:23
  • Hello @Thomas, this flow is only a simple part of the architecture. I've simplified it to focus the main issue. ASA is not in the middle of the flow in real scenario but we want to use it for stream processing and it should be fast enough for providing **real-time** analytics. We also need time-windowing to measure users' visit counts and notify them if campaign boundaries are reached. – msapcili Jun 25 '16 at 13:09
  • Ok but you dont want to do a SELECT * inside you flow. You forward all your data from EventHub to SA so that there is no slow down in your flow. In parallel you can query SA and you will not have any impact on th time to process messages – Thomas Jun 25 '16 at 21:35
  • @Thomas, it directly does not impact my main flow but it is another important flow for notifications. It should also be fast enough, it's time consumption should not be greater than 1, 1.5 sec. Because I should notify the specified user according to result of streaming data processing. Thanks for your concern about the flow but the real problem is ASA is not fast unit as described "Real-time stream processing". A real time processor cannot be like this, it supports time windowing by second but I cannot get result from it in a sec. – msapcili Jun 26 '16 at 16:01

1 Answers1

5

There is a latency when merging all the event in chronological order from the Event Hub.

ASA will visit all partitions from EH, get the data and organize the events into chronological order. This means that data must arrive at all partitions in the EH. I think this will also explain the strange behavior you are seeing with the EventProcessedUtcTime, it might be that because the events are ordered, the logical processing time is before the actual arrival time. Although I'm unsure about this because I do not know the inner workings of ASA.

This latency will increase with the number of partitions used, especially when the dataflow is slow.

You can sidestep the merger by partitioning on the field partitionid from EH. Make sure you are sending the data to the correct partition in EH as well.

More information can be found here at the Stream Analytics blog.

Waaghals
  • 2,029
  • 16
  • 30
  • thank you so much for your contribution. According to the article you shared: **"My experiment shows with a select * query, if I send 1 event per second to Event Hub, the end to end delay is around 6 seconds."** This is exactly what I've experienced. I'll do some tests and let you know. Regards. – msapcili Jun 27 '16 at 02:37
  • Hello @Waaghals. Thank you so much again. We've also talked with our Microsoft representative and agreed that our performance issues are related with the architecture that you've pointed. But this slowness is still unacceptable for a real time processor so we will try to reach ASA team to get more tips. Best regards, – msapcili Jul 04 '16 at 15:42
  • @msapcili Did you ever get it down even more than 6s? – John Aug 07 '19 at 09:27
  • @John I'd changed the architecture and did not use ASA after this issue. – msapcili Aug 25 '19 at 20:31