6

I am looking for a way to generate a unique ID for nosql database. Unlike relational database there is no idea of rows which means there is no last row to increment from.

The most common way to handle this is to use UUID's. But my problem is I need to add another ID (other than the UUID) which needs to be:

  • Unique
  • Unsigned Int32

Total data could reach around 50,000,000. So how would you generate somewhat unique uint32 ID's?

The UInt32 value type represents unsigned integers with values ranging from 0 to 4,294,967,295.

  • Only generated when a new user registers.
  • 3 Id's are given to each new user.
  • Currently using Couchbase Server.
majidarif
  • 18,694
  • 16
  • 88
  • 133
  • how frequently will they be generated? You might be able to use ms since unix epoch.. also, CRC32? – Paul S. Aug 10 '14 at 15:10
  • Maybe once a few minutes. There could be times that they can happen at the same time, but unlikely. – majidarif Aug 10 '14 at 15:12
  • @PaulS.: Seconds, rather? – Ry- Aug 10 '14 at 15:13
  • @PaulS. I could CRC32 usernames right? as usernames will be unique, and this ids are given to each user on register, problem is, 3 numbers should be given to each new registered user. Just realized crc can contain letters so it wont work? – majidarif Aug 10 '14 at 15:15
  • Which NoSQL database are you using? – Ry- Aug 10 '14 at 15:16
  • @minitech I'm currently using couchbase but I'm looking at others. But technically most NoSQL database have the same patterns for IDS. – majidarif Aug 10 '14 at 15:17
  • https://stackoverflow.com/questions/12994869/how-do-you-increment-or-create-unique-id-in-couchbase – Ry- Aug 10 '14 at 15:18
  • 1
    @minitech [this](http://docs.couchbase.com/prebuilt/node-sdk-2.0-beta/topics/atomic-operations.html) looks promising. Any idea if it might be a fit for my use case? You might want to add it as an answer. I'll accept it. Thanks. – majidarif Aug 10 '14 at 15:21
  • I don’t know, sorry, but feel free to add it as your own answer and ask for commentary. – Ry- Aug 10 '14 at 15:26
  • @minitech I'll look into it more. It seems to only work for a single document. – majidarif Aug 10 '14 at 15:27

3 Answers3

4

This problem has already been solved - I would suggest using the atomic Increment (or Decrement) functions in Couchbase - these are a common pattern to generate unique IDs.

Whenever the incr() method is called, it atomically increments the counter by the specified value, and returns the old value, therefore it's safe if two clients try to increment at the same time.

Pseudocode example (I'm no Node.JS expert!):

// Once, at the beginning of time we init the counter:
client.set("user::count", 0);
...
// Then, whenever a new user is needed: 
nextID = client.incr("user::count", 1); // increments counter and returns 'old' value.
newKey = "user_" + nextID;
client.add(newKey, value);

See the Node.JS SDK for reference, and see Using Reference Doucments for Lookups section in the Couchbase Developer Guide for a complete usage example.

DaveR
  • 9,540
  • 3
  • 39
  • 58
  • Inc and Dec seems to work if you increment a counter inside a document. How about across documents, but I'll have to check. Will get back here as soon as I do as I'm still at work. – majidarif Aug 11 '14 at 07:14
  • @majidarif you just have a document which *is* a counter - for example for userIds you may have a document named `userid::count` which you use `incr()` on to generate the next userID. – DaveR Aug 11 '14 at 08:24
  • hmmm, im not following. Can you explain more? Should I get the last count and incr it when inserting a new document? Wounldn't this be a problem if 2 documents get inserted at the same time. If they get the same last count? – majidarif Aug 11 '14 at 08:28
  • From your updated answer, I see that it does increment the counting document. But how about inserting the next number on the next document. Should I increment the counter first then get the number then insert the new document? – majidarif Aug 11 '14 at 08:30
  • If you look at the complete example I linked, you'll see that calling `incr()` *atomically* increments the counter and returns the previous value - so whenever you want to create a new user you simply do `nextID = counter.incr(); newKey = "prefix_" + nextID` or similar. – DaveR Aug 11 '14 at 08:33
  • I see, can you add that to your answer? Thanks. This makes more sense now. – majidarif Aug 11 '14 at 08:35
1

Here's a function that returns a unique identifier each time it's called. It should be fine as long as the number of items does not exceed the range of 32-bit integers, which seems to be the case given the described requirements. (warning: once the array of UIDs fills up, this enters an infinite loop. You may also want to create some sort of a reset function that can empty the array and thus reset the UID when necessary.)

var getUID = (function() {
    var UIDs = [];

    return function() {
        var uid;

        do {
            uid = Math.random() * Math.pow(2, 32) | 0x0;
        } while (UIDs[uid] !== undefined);

        return UIDs[uid] = uid;
    };
}());
The Paramagnetic Croissant
  • 9,223
  • 3
  • 25
  • 38
  • Wouldn't this forget the ids when the app gets restarted? – majidarif Aug 10 '14 at 15:19
  • If you changed it from `1 << 31` because that ends up negative, note `uid = Math.random() * (-1 >>> 0) | 0`. Also, seems like `UIDs` should be `{}` initially, not `[]`; updating `length` is wasteful when it’s not accurate. – Ry- Aug 10 '14 at 15:19
  • @minitech I changed from `1 << 31` because I realized that it would only result int 31 significant bits. But, thanks for the suggestion anyway. – The Paramagnetic Croissant Aug 10 '14 at 15:21
  • @majidarif practically any JavaScript solution would. That, however, is easy to solve by pre-populating the array with the IDs upon application startup. – The Paramagnetic Croissant Aug 10 '14 at 15:22
0

if you will call this insert method by passing "user" as key then your docId will be auto increment as user_0 user_1 user_2 etc... Please note that couchbase will show one extra row in your bucket with key as meta id and next counter value as doc value. Do not get surprised if you use query like select count(*) total from table; as it will show one more than real count, to avoid use where clause so that this row won't be counted.

public insert(data: any, key: string) {
  return new Promise((resolve, reject) => {
    let bucket = CouchbaseConnectionManager.getBucket(`${process.env.COUCHBASE_BUCKET}`)
    bucket.counter(key, 1, {initial:0}, (err:any, res:any)=>{
      if(err){
        this.responseHandler(err, res, reject, resolve);
      }
      const docId = key + "_" + res.value;
      bucket.insert(docId, data, (error:any, result:any) => {
        this.responseHandler(error, result, reject, resolve);
      });
    });
  });
}
Ravi Bhagat
  • 11
  • 2