3

We're evaluating Azure Time Series Insights (Preview) for ingesting historical IoT telemetry data. We use device timestamp instead of event enqueued timestamp as Timestamp property (Non-default behavior)

We've hit a snag around ingesting historical data (>2 year) old. TSI silently drops these messages. We want to understand what are the best practices for ingesting old data.

Similarly on the querying side, are the results returned based on TSI view (last 400 days) or can we query all existing parquet files (ie all the data that was ingested via Event hub howsoever old it may be) ?

We've tried ingesting new data and it appears on TSI dashboard. However we're not able to ingest old data since the timestamp is older. These messages are dropped.

We expect to ingest historical sensor data preserving device timestamps. We can pivot on event enqueued timestamp but this makes querying a problem. Looking for suggestions for resolving both the ingestion and querying part of historical data via Time Series Insights.

AshokPeddakotla
  • 1,010
  • 7
  • 18
SuperManu
  • 77
  • 8
  • Can you elaborate more on how do you detect that the messages are silently dropped? Do you see any error message? Note: The Time Series Insights Preview uses Azure Blob storage with the Parquet file type. When GA TSI will make data available within 60 seconds of hitting an event source.During the preview, expect a longer period before the data is made available.Therefore please make sure you wait enough time before trying to look at historical data. – asergaz Jun 24 '19 at 11:55
  • Actually the data wasn't appearing since the UI has the limit of past 1 year. We were able to get the same via GetEvents API – SuperManu Aug 08 '19 at 09:11

0 Answers0