aws cli dynamo db (ValidationException) Error

Question

I'm looking to batch write item to dynamodb using python's boto3 module and i'm getting this. This is the first time i've ever worked with aws cli or boto3. The documentation says validation exception errors occur when there are empty values and possible incorrect data types, but i've played with all those and it doesn't seem to be working.

Does dynamodb only like to have 25 items written to it at a time? how can i control those batches if so?

My request:

client = boto3.client('dynamodb')
response = client.batch_write_item(RequestItems=batch_dict)

Top of batch_dict:

{'scraper_exact_urls': [{'PutRequest': {'Item': {'Sku': {'S': 'T104P3'},
 'pps_id': {'N': '427285976'},
 'scraper_class_name': {'S': 'scraper_class_name'},
 'store_id': {'N': '1197386754'},
 'updated_by': {'S': 'user'},
 'updated_on': {'N': '1480714223'},
 'updated_url': {'S': 'http://www.blah.com'}}}},
 {'PutRequest': {'Item': {'Sku': {'S': 'T104P3'},
 'pps_id': {'N': '427285976'},
 'scraper_class_name': {'S': 'scraper_class_name'},
 'store_id': {'N': '1197386754'},
 'updated_by': {'S': 'user'},
 'updated_on': {'N': '1480714223'},
 'updated_url': {'S': 'http://www.blah.com'}}}},....

Schema:

attributes: "pps_id"=>\Aws\DynamoDb\Enum\Type::NUMBER, "sku"=>\Aws\DynamoDb\Enum\Type::STRING, "scraper_class_name"=>\Aws\DynamoDb\Enum\Type::STRING, "store_id"=>\Aws\DynamoDb\Enum\Type::NUMBER, "updated_url"=>\Aws\DynamoDb\Enum\Type::STRING, "updated_by"=>\Aws\DynamoDb\Enum\Type::STRING, "updated_on"=>\Aws\DynamoDb\Enum\Type::NUMBER, fields: "pps_id", "scraper_class_name",

The Error:

ClientError: An error occurred (ValidationException) when calling the    BatchWriteItem operation: 1 validation error detected: Value .... Map value   must satisfy constraint: [Member must have length less than or equal to 25,   Member must have length greater than or equal to 1]

Think I found the answer here [link](http://stackoverflow.com/questions/31065900/how-to-write-more-than-25-items-rows-into-table-for-dynamodb) — Yale Newman, Dec 02 '16 at 22:41
Possible duplicate of [How to write more than 25 items/rows into Table for DynamoDB?](http://stackoverflow.com/questions/31065900/how-to-write-more-than-25-items-rows-into-table-for-dynamodb) — LuFFy, Mar 09 '17 at 09:09

score 6 · Accepted Answer · edited May 23 '17 at 12:32

6

The BatchWriteItem API works on 25 items at a time. You could use the following code, adapted from the non-copying batching question, to call BatchWriteItem on 25 item chunks

def batch(iterable, n=1):
    l = len(iterable)
    for ndx in range(0, l, n):
        yield iterable[ndx:min(ndx + n, l)]

client = boto3.client('dynamodb')

for x in batch(batch_dict['scraper_exact_urls'], 25):
    subbatch_dict = {'scraper_exact_urls': x}
    response = client.batch_write_item(RequestItems=subbatch_dict)

edited May 23 '17 at 12:32

Community

1
1

answered Mar 05 '17 at 15:56

Alexander Patrikalakis

5,054
1
30
48

1

Had to do a few adjustments for it to work, as otherwise it was incomplete code for a pre-beginner python user like myself – omega_prime Sep 23 '20 at 18:50

score 2 · Answer 2 · answered Dec 17 '20 at 17:34

And here is the Javascript version of batch writing/deleting many items, in case someone needs it:

const batchWriteManyItems = async (tableName, itemObjs, chunkSize = 25) => {

        const buildParams = (table) => JSON.parse(`{"RequestItems": {"${table}": []}}`)

        const queryChunk = (arr, size) => {
            const tempArr = []
            for (let i = 0, len = arr.length; i < len; i += size) {
                tempArr.push(arr.slice(i, i + size));
            }

            return tempArr
        }

        await Promise.all(queryChunk(itemObjs, chunkSize).map(async chunk => {
            let itemParams = buildParams(tableName);
            itemParams.RequestItems[tableName] = chunk
            await dynamoDB.batchWriteItem(itemParams).promise()
        }))
    }

NOTE: the itemObjs attribute can take {PutRequest: {Item: ...} for write items and {DeleteRequest: {Item: ...} to delete items

aws cli dynamo db (ValidationException) Error

2 Answers2