How to use auto increment for primary key id in dynamodb

Question

I am new to dynamodb. I want to auto increment id value when I use putitem with dynamodb.

Is possible to do that?

Answer 1

This is anti-pattern in DynamoDB which is build to scale across many partitions/shards/servers. DynamoDB does not support auto-increment primary keys due to scaling limitations and cannot be guaranteed across multiple servers.

Better option is to assemble primary key from multiple indices. Primary key can be up to 2048 bytes. There are few options:

1. Use UUID as your key - possibly time based UUID which makes it unique, evenly distributed and carries time value
2. Use randomly generated number or timestamp + random (possibly bit-shifting) like: ts << 12 + random_number
3. Use another service or DynamoDB itself to generate incremental unique id (requires extra call)

Following code will auto-increment counter in DynamoDB and then you can use it as primary key.

var documentClient = new AWS.DynamoDB.DocumentClient();
var params = {
  TableName: 'sampletable',
  Key: { HashKey : 'counters' },
  UpdateExpression: 'ADD #a :x',
  ExpressionAttributeNames: {'#a' : "counter_field"},
  ExpressionAttributeValues: {':x' : 1},
  ReturnValues: "UPDATED_NEW" // ensures you get value back
};
documentClient.update(params, function(err, data) {});
// once you get new value, use it as your primary key

My personal favorite is using timestamp + random inspired by Instagram's Sharding ID generation at http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-at-instagram

Following function will generate id for a specific shard (provided as parameter). This way you can have unique key, which is assembled from timestamp, shard no. and some randomness (0-512).

var CUSTOMEPOCH = 1300000000000; // artificial epoch
function generateRowId(shardId /* range 0-64 for shard/slot */) {
  var ts = new Date().getTime() - CUSTOMEPOCH; // limit to recent
  var randid = Math.floor(Math.random() * 512);
  ts = (ts * 64);   // bit-shift << 6
  ts = ts + shardId;
  return (ts * 512) + randid;
}
var newPrimaryHashKey = "obj_name:" + generateRowId(4);
// output is: "obj_name:8055517407349240"

Answer 2

DynamoDB doesn't provide this out of the box. You can generate something in your application such as UUIDs that "should" be unique enough for most systems.

I noticed you were using Node.js (I removed your tag). Here is a library that provides UUID functionality: node-uuid

Example from README

var uuid = require('node-uuid');
var uuid1 = uuid.v1();
var uuid2 = uuid.v1({node:[0x01,0x23,0x45,0x67,0x89,0xab]});
var uuid3 = uuid.v1({node:[0, 0, 0, 0, 0, 0]})
var uuid4 = uuid.v4();
var uuid5 = uuid.v4();

Answer 3

You probably can use AtomicCounters.

With AtomicCounters, you can use the UpdateItem operation to implement an atomic counter—a numeric attribute that is incremented, unconditionally, without interfering with other write requests. (All write requests are applied in the order in which they were received.) With an atomic counter, the updates are not idempotent. In other words, the numeric value increments each time you call UpdateItem.

You might use an atomic counter to track the number of visitors to a website. In this case, your application would increment a numeric value, regardless of its current value. If an UpdateItem operation fails, the application could simply retry the operation. This would risk updating the counter twice, but you could probably tolerate a slight overcounting or undercounting of website visitors.

Answer 4

Came across a similar issue, where I required auto-incrementing primary key in my table. We could use some randomization techniques to generate a random key and store it using that. But it won't be in a incremental fashion.

If you require something in incremental fashion, you can use Unix Time as your primary key. Not assuring, that you can get a accurate incrementation(one-by-one), but yes every record you put, it would be in incremental fashion, with respect to the difference in how much time each record in inserted in.

Not a complete solution, if you don't want to read the entire table and get it's last id and then increment it.

Following is the code for inserting a record in DynamoDB using NodeJS:

.
.
        const params = {
            TableName: RANDOM_TABLE,
            Item: {
                ip: this.ip,
                id: new Date().getTime()
            }
        }
    
        dynamoDb.put(params, (error, result) => {
            console.log(error, result);
        });
.
.

Answer 5

If you are using NoSQL Dynamo DB then using Dynamoose, you can easily set default unique id, here is the simple user create example

// User.modal.js

const dynamoose = require("dynamoose");
const { v4: uuidv4 } = require("uuid");

const userSchema = new dynamoose.Schema(
  {
    id: {
      type: String,
      hashKey: true,
    },
    displayName: String,
    firstName: String,
    lastName: String,
  },
  { timestamps: true },
);

const User = dynamoose.model("User", userSchema);

module.exports = User;

// User.controller.js

exports.create = async (req, res) => {
  const user = new User({ id: uuidv4(), ...req.body }); // set unique id
  const [err, response] = await to(user.save());
  if (err) {
    return badRes(res, err);
  }
  return goodRes(res, reponse);
};

Answer 6

Update for 2022 :

I was looking for the same issue and came across following research.

DynamoDB still doesn't support auto-increment of primary keys.

https://aws.amazon.com/blogs/database/simulating-amazon-dynamodb-unique-constraints-using-transactions/

Also the package node-uuid is now deprecated. They recommend we use uuid package instead that creates RFC4122 compliant UUID's.

npm install uuid

import { v4 as uuidv4 } from 'uuid';
uuidv4(); // ⇨ '9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d'

Answer 7

For Java developers, there is the DynamoDBMapper, which is a simple ORM. This supports the DynamoDBAutoGeneratedKey annotation. It doesn't increment a numeric value like a typical "Long id", but rather generates a UUID like other answers here suggest. If you're mapping classes as you would with Hibernate, GORM, etc., this is more natural with less code.

I see no caveats in the docs about scaling issues. And it eliminates the issues with under or over-counting as you have with the auto-incremented numeric values (which the docs do call out).