82

I've been going through AWS DynamoDB docs and, for the life of me, cannot figure out what's the core difference between batchGetItem() and Query(). Both retrieve items based on primary keys from tables and indexes. The only difference is in the size of the items retrieved but that doesn't seem like a ground breaking difference. Both also support conditional updates.

In what cases should I use batchGetItem over Query and vice-versa?

suv
  • 1,373
  • 1
  • 10
  • 18

5 Answers5

93

There’s an important distinction that is missing from the other answers:

  • Query requires a partition key
  • BatchGetItems requires a primary key

Query is only useful if the items you want to get happen to share a partition (hash) key, and you must provide this value. Furthermore, you have to provide the exact value; you can’t do any partial matching against the partition key. From there you can specify an additional (and potentially partial/conditional) value for the sort key to reduce the amount of data read, and further reduce the output with a FilterExpression. This is great, but it has the big limitation that you can’t get data that lives outside a single partition.

BatchGetItems is the flip side of this. You can get data across many partitions (and even across multiple tables), but you have to know the full and exact primary key: that is, both the partition (hash) key and any sort (range). It’s literally like calling GetItem multiple times in a single operation. You don’t have the partial-searching and filtering options of Query, but you’re not limited to a single partition either.

Craig Walker
  • 49,871
  • 54
  • 152
  • 212
  • 8
    Excellent answer. Was wracking my brain trying to figure out why BatchGetItem isn't working when I provided my hash key, forgetting that it expects the hash + range key. I ended up just firing N concurrent requests to run N queries whenever I have N hash ids without any range keys. Throttled, of course. It's better than nothing. Would be nice to have support for a BatchQuery that takes in a list of hash/partition keys and returns all the relevant results. – HandleThatError Sep 20 '19 at 22:50
  • 7
    This answer was the clearest to me, I feel it should be the accepted one. – geo909 Jun 04 '20 at 06:16
  • Since primary key is unique so how does BatchGetItems return multiple records? – Yug Singh Feb 15 '21 at 11:42
  • 1
    @YugSingh because you're passing it multiple primary keys in an array. – Craig Walker Feb 17 '21 at 15:57
  • If I pass entire primary key in query, will it be any different than batch_get_item? I'm asking this because what if I wan't to apply an extra filter expression which we can't do in batch_get_item but we can do in query. So, will it be any different in terms of performance? – Rahul Chouhan Jun 01 '23 at 02:46
53

As per the official documentation: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithTables.html#CapacityUnitCalculations

For BatchGetItem, each item in the batch is read separately, so DynamoDB first rounds up the size of each item to the next 4 KB and then calculates the total size. The result is not necessarily the same as the total size of all the items. For example, if BatchGetItem reads a 1.5 KB item and a 6.5 KB item, DynamoDB will calculate the size as 12 KB (4 KB + 8 KB), not 8 KB (1.5 KB + 6.5 KB).

For Query, all items returned are treated as a single read operation. As a result, DynamoDB computes the total size of all items and then rounds up to the next 4 KB boundary. For example, suppose your query returns 10 items whose combined size is 40.8 KB. DynamoDB rounds the item size for the operation to 44 KB. If a query returns 1500 items of 64 bytes each, the cumulative size is 96 KB.

You should use BatchGetItem if you need to retrieve many items with little HTTP overhead when compared to GetItem.

A BatchGetItem costs the same as calling GetItem for each individual item. However, it can be faster since you are making fewer network requests.

Isaiah Turner
  • 2,574
  • 1
  • 22
  • 35
stephane
  • 807
  • 7
  • 9
  • 23
    While this answer nicely explains how pricing works for DynamoDB, it's sort of misleading in the sense that it seems to give the impression that Query can potentially be used to replace BatchGet. My understanding is that it can't. See [this answer](http://stackoverflow.com/questions/31633141/batchgetitem-or-query-dynamodb-query-by-range) – mindreader Aug 22 '16 at 19:14
46

In a nutshell: BatchGetItem works on tables and uses the hash key to identify the items you want to retrieve. You can get up to 16MB or 100 items in a response

Query works on tables, local secondary indexes and global secondary indexes. You can get at most 1MB of data in a response. The biggest difference is that query support filter expressions, which means that you can request data and DDB will filter it server side for you.

You can probably achieve the same thing if you want using any of these if you really want to, but rule of the thumb is you do a BatchGet when you need to bulk dump stuff from DDB and you query when you need to narrow down what you want to retrieve (and you want dynamo to do the heavy lifting filtering the data for you).

Mircea
  • 10,216
  • 2
  • 30
  • 46
5

DynamoDB stores values in two kinds of keys: a single key, called a partition key, like "jupiter"; or a compound partition and range key, like "jupiter"/"planetInfo", "jupiter"/"moon001" and "jupiter"/"moon002".

A BatchGet helps you fetch the values for a large number of keys at the same time. This assumes that you know the full key(s) for each item you want to fetch. So you can do a BatchGet("jupiter", "satrun", "neptune") if you have only partition keys, or BatchGet(["jupiter","planetInfo"], ["satrun","planetInfo"], ["neptune", "planetInfo"]) if you're using partition + range keys. Each item is charged independently and the cost is same as individual gets, it's just that the results are batched and the call saves time (not money).

A Query on the other hand, works only inside a partition + range key combo and helps you find items and keys that you don't necessarily know. If you wanted to count Jupiter's moons, you'd do a Query(select(COUNT), partitionKey: "jupiter", rangeKeyCondition: "startsWith:'moon'"). Or if you wanted the fetch moons no. 7 to 15 you'd do Query(select(ALL), partitionKey: "jupiter", rangeKeyCondition: "BETWEEN:'moon007'-'moon015'"). Here you're charged based on the size of the data items read by the query, irrespective of how many there are.

Sudhir Jonathan
  • 16,998
  • 13
  • 66
  • 90
1

Adding an important difference. Query supports Consistent Reads, while BatchGetITem does not.

BatchGetITem Can use Consistent Reads through TableKeysAndAttributes

Thanks @colmlg for the information.

Sleem
  • 139
  • 1
  • 4
  • 1
    It looks like `BatchGetItem` does support strongly consistent reads: "By default, BatchGetItem performs eventually consistent reads on every table in the request. If you want strongly consistent reads instead, you can set ConsistentRead to true for any or all tables." https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchGetItem.html – colmlg Jun 17 '21 at 09:57