0

We have a table in Azure Table Storage that is storing a LOT of data in it (IoT stuff). We are attempting a simple migration away from Azure Tables Storage to our own data services.

I'm hoping to get a rough idea of how much data we are migrating exactly. EG: 2,000,000 records for IoT device #1234.

The problem I am facing is in getting a count of all the records that are present in the table with some constrains (EG: Count all records pertaining to one IoT device #1234 etc etc).

I did some fair amount of research to find posts that say that this count feature is not implemented in the ATS. These posts however, were circa 2010 to 2014.

I'm assumed (hoped) that this feature has been implemented now since it's now 2017 and I'm trying to find docs to it.

I'm using python to interact with out ATS.

Could someone please post the link to the docs here that show how I can get the count of records using python (or even HTTP / rest etc)?

Or if someone knows for sure that this feature is still unavailable, that would help me move on as well and figure another way to go about things!

Thanks in advance!

Rahul Raghunath
  • 175
  • 1
  • 4
  • 15

2 Answers2

1

Or if someone knows for sure that this feature is still unavailable, that would help me move on as well and figure another way to go about things!

This feature is still not available or in other words as of today there's no API which will give you a count of total number of rows in a table. You would have to write your own code to do so.

Could someone please post the link to the docs here that show how I can get the count of records using python (or even HTTP / rest etc)?

For this you would need to list all entities in a table. Since you're only interested in the count, you can reduce the size response data by making use of Query Projection and fetching just one or two attributes of the entities (may be PartitionKey and RowKey). Please see my answer here for more details: Count rows within partition in Azure table storage.

Community
  • 1
  • 1
Gaurav Mantri
  • 128,066
  • 12
  • 206
  • 241
  • Thnks for the answer! I actually read your answer before posting here. I might as well do the migration and keep track of the reads instead of attempting this count first. It's very disappointing that Microsofth hasn't yet built this simple feature – Rahul Raghunath May 10 '17 at 06:53
  • `It's very disappointing that Microsofth hasn't yet built this simple feature` - While I agree that it is disappointing yet I don't think its a simple feature considering the amount of data you can store in a table. It is going to be a massive engineering challenge to provide this feature. – Gaurav Mantri May 10 '17 at 06:58
1

Returning number of entities in the table storage is for sure not available in Azure Table Storage SDK and service. You could make a table scan query to return all entities from your table but if you have millions of these entities the query will probably time out. it is also going to have pretty big perf impact on your table. Alternatively you could try making segmented queries in a loop until you reach the end of the table.

Dogu Arslan
  • 3,292
  • 24
  • 43
  • Thanks for clarifying! It's disappointing that such a simple feature isn't yet implemented. I saw posts dating back to 2010. It's been seven years, yet nothing! Anyway, thanks for the clarification. You alternative idea works fine, but I might as well do the migration and log the reads and writes done. Thanks again :) – Rahul Raghunath May 10 '17 at 06:50