2

I would like to check how many entries are in a DynamoDB table that matches a query without retrieving the actual entries, using boto3.

I want to run a machine learning job on data from DynamoDB table. The data I'm training on is a data that answers a query, not the entire table. I want to run the job only if I have enough data to train on. Therefore, I want to check if I want to check that I have enough entries that match the query. It is worth mentioning that the DynamoDB table I'm querying is really big, therefore actual retrieving is no option unless I actually want to run the job.

I know that I can use boto3.dynamodb.describe_table() to get how many entries there are in the entire table, but as I mentioned earlier, I want to know only how many entries match a query.

Any ideas?

Sagi Shadur
  • 226
  • 1
  • 4
  • 14
  • [boto3.query()](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Client.query)? – Lamanus Sep 12 '19 at 11:51

1 Answers1

8

This was asked and answered in the past, see How to get item count from DynamoDB?

Basically, you need to use the "Select" parameter to tell DynamoDB to only count the query's results, instead of retrieving them.

As usual in DynamoDB, this is truncated by paging: if the result set (not the count - the actual full results) is larger than 1 MB, then only the first 1 MB is retrieved, and the items in it counted, and you get back this partial count. If you're only interested in checking whether you have "enough" results - this may even be better for you - because you don't want to pay for reading a gigabyte of data just to check if the data is there. You can even ask for a smaller page, to read less - depending on what you consider enough data.

Just remember that you'll pay Amazon not by the amount of data returned (just one integer, the count) but by the amount of data read from disk. Using such counts excessively may lead to surprising large costs.

Nadav Har'El
  • 11,785
  • 1
  • 24
  • 45