Azure Stream Analytics input corrupts strings containing timezone info

Question

I am using Azure Event Hub for collection of timebased events. Connected Azure Stream Analytics (ASA) to it.

This results in losing the timezone info at ASA level.

What I ascertained is the following: I have sent data in JSON format containing a string with a timestamp compatible with ISO 8601. e.g.: "event_timestamp": "2016-09-02T19:51:38.657+02:00"

I checked by means of ServiceBus Explorer (thanks to the guys who wrote this tool) that this string arrived exactly as-is in Event Hub.

In the Stream Analytics I added the event hub as a input. When I use the option SAMPLE DATA in the Azure portal this result in data containing: "event_timestamp":"2016-09-02T17:51:38.6570000"

Why is Stream Analytics removing timezone info???

According to ISO 8601 not specifying a timezone in a timestamp means that de timestamp is converted to localtime. Does that mean the timezone where the Azure resource is running? How can I use geo-replication in that case?

This means that after consuming the data and presenting it in a dashboard all times are related to the time of the server where the stream analytic runs?

Do I need to add the timezone information seperately in the JSON payload and reconstruct it afterwards?

My conclusion is that actually ASA removes/destruct information from my data stream. Imagine this ASA query: SELECT * INTO [myoutput] FROM [myinput]

This would change the content (*) of my data. All strings that appear to be a datetime with timezone info will be converted.

In my opinion this is very unwanted behaviour.

I am very interested in the opinions of others in this forum.

Answer 1

Everything in Azure runs in UTC Timezone, unless otherwise supported and explicitly configured (there are not many services which support setting timezone).

If you look at your quoted samples closely you will notice that the timestamp is converted to UTC in the ASA, that's why the TimeZone info is missing:

Sent to event hub: "event_timestamp": "2016-09-02T19:51:38.657+02:00"

Received in ASA: "event_timestamp":"2016-09-02T21:51:38.6570000"

Note that your event is sent in 19:51:38.657 +2:00 and ASA reads 21:51:38.6570000 which is absolutely the same.

UPDATE

I am not expert on ISO standard, but here are some exerpts from ASA Docu: Azure Stream Analytics data types

datetime Defines a date that is combined with a time of day with fractional seconds that is based on a 24-hour clock and relative to UTC (time zone offset 0).

convertions:

datetime string converted to datetime following ISO 8601 standard

It is documented that date time is in UTC. Hence no need to explicitly specify it. Whether this comforts with the ISO I cannot tell, first because WikiPedia is not ISO Document, second because I am not ISO expert.