Generate Unique Identifier in Very Big Loop - Java

Question

the question i am going to ask is very old and i think it asked on SO 5-10 times.

but mine has a different situation.

Read my Problem before making it duplicate by so many wise (over) SO users.

I am importing CSV sheet containing 10K records in my application.

My logic works in following manner,

(1) Validate & Import Sheet (2) Save to the database if record does not exist

Step 2 is done for each and every record of the sheet.

in the step -2 i have to generate UUID to identify a particular record later ,

in my first solution

// this might be unique in some cases
String id = UUID.randomUUID().toString();

but i checked that it does generate unique id in each case , for example if i import 10 sheet one by one with different records in it, all 10 times i am getting duplicate key error from database for at least 4000 times in each import and save operation,

that means that out of 10,000 key generation it generates only 6000 unique ids.

so then i generate an alphanumeric code which length is 6 , some thing like

eSv3h7

and append it to previously generated id and hence get the following id

d545f2b2-63ab-4703-89b0-f2f8eca02154-eSv3h7

after testing still there is a problem of id duplication.

I also tried several combination mentioned here and on other sites but still there is a same problem of id duplication,

Now i am wondering that this occurs only for 10k records saving in loop , actually i need to import sheet which is having 8 million records in it

so how can i solve my problem of generating a Unique id in my particular case ?

Update 1 - based on all the comments

Try this thing at you end.

loop through 1 to 10,000 generate uuid in the loop store it somewhere in simple text file

then make a simple program to find the duplicates from them , if you do not find any one duplicate in first attempt , repeat all above steps again and again and i am sure you will find duplicates.

in the past i am also strong believer of the same thing that UUID will never generates duplicates, share me your result of above test.

Update 2 - Code

This is the method which is called by each record of the sheet to be saved by caller method' loop.

@Override

public void preSynchronizedServiceExecution(ServiceData sData,
            ValueObject valueObject) throws BlfException {

        PropertyVO pVO = (PropertyVO) valueObject;

        ArrayList<CountyAuctionPropertyVO> capList = pVO
                .getCountyAuctionPropertyList();

        for (CountyAuctionPropertyVO caVO : capList) {

            TadFrameworkUtil.processValueObjectKeyProperty(caVO, true);

            TadFrameworkUtil.processValueObjectKeyProperty(caVO
                    .getPropertyLastOwner(), true);

            TadFrameworkUtil.processValueObjectKeyProperty(caVO
                    .getPropertyLastOwner().getAdd(), true);

        }

        ArrayList<PropertyAminitiesVO> amList = pVO.getPropertyAminitiesList();

        for (PropertyAminitiesVO pamVO : amList) {

            TadFrameworkUtil.processValueObjectKeyProperty(pamVO, true);

        }

        ArrayList<PropertyAttributesVO> atList = pVO
                .getPropertyAttributesList();

        for (PropertyAttributesVO patVO : atList) {

            TadFrameworkUtil.processValueObjectKeyProperty(patVO, true);

        }

        TadFrameworkUtil.processValueObjectKeyProperty(pVO, true);

        TadFrameworkUtil.processValueObjectKeyProperty(pVO.getSiteAdd(), true);
    }

Following is id generation method

public static String generateUUID() throws BlfException {

        // this might be unique in some cases
        String id = UUID.randomUUID().toString();

        // introduce custom random string in mixing of upper and lower
        // alphabets,
        // which is 6 character long
        // and append it to generated GUID.
        String rs = randomString(6);

        id = id.concat("-").concat(rs);

        return id;
    }

Update 3 (Method added)

public static void processValueObjectKeyProperty(ValueObject valueObject,
            boolean create) throws BlfException {

        String key = (String) BlfConverter.getKey(valueObject);

        if (!StringUtility.isStringNonEmpty(key)) {
            throw new BlfException(valueObject.getObjectName()
                    + "-  key property does not exist.");
        }

        if (create) {

            String id = generateUUID();

            valueObject.setProperty(key, id);
        } else {

            String exisitingId = valueObject.getProperty(key);

            if (!StringUtility.isStringNonEmpty(exisitingId)) {

                String id = generateUUID();

                valueObject.setProperty(key, id);
            }
        }
    }

The random string method is just a simple methods of 2 lines which generates alpha numeric random string of length 6.

please ask me if you need anything more so i can post here.

Update 4 (Sample genearted UUID )

d545f2b2-63ab-4703-89b0-f2f8eca02154-eSv3h7
6f06fa28-6f36-4ed4-926b-9fef86d002b3-DZ2LaE
20142d05-f456-4d72-b845-b6819443b480-xzypQr
67b2a353-e7b4-4245-90a0-e9fca8644713-AgSQZm
8213b275-2cb1-4d37-aff0-316a47e5b780-vMIwv9

and i am getting accurate result from database if i need to fetch it from there.

Thanks

Can you share your the entire method? UUID should produce unique enough IDs for this usecase... — Mureinik, Dec 25 '13 at 07:22
What you might want to do is get the seed (integer) for the randomization that will produce the key. Then, check to see if that value is populated in a database. Just an idea.\ — Obicere, Dec 25 '13 at 07:22
How long is your id field when saving to the database? If you truncate the UUID I can imagine there are duplicates, if you're not truncating it sounds like something else is very wrong. — Joachim Isaksson, Dec 25 '13 at 07:25
There is no way that `UUID.randomUUID()` is giving you duplicates. You must have some other bug in your code. If you can show us some code, we might be able to find it for you. — Dawood ibn Kareem, Dec 25 '13 at 07:26
I too do not believe that UUID is producing duplicates. As @JoachimIsaksson pointed, you must be having a truncating scenario for the column in which you are saving these UUIDs you are generating. Check its column length, and.. — Siva Tumma, Dec 25 '13 at 07:27
@JoachimIsaksson please , see my update.more over my id field length is 65 chars no issue of data truncation. — Mihir, Dec 25 '13 at 08:53
@DavidWallace please , see my update.more over my id field length is 65 chars no issue of data truncation. — Mihir, Dec 25 '13 at 08:54
@sivatumma please , see my update.more over my id field length is 65 chars no issue of data truncation. — Mihir, Dec 25 '13 at 08:55
@Mihir, your update describes one of the tests in the test suites of one of the programs that I have worked on. It has run thousands and thousands of times, generating 10000 UUIDs each time and checking for duplicates. This particular test has never failed. However, if you would like to show us some code, we may be able to find the bug in it for you. — Dawood ibn Kareem, Dec 25 '13 at 08:56
@DavidWallace can you share those tests to me so i can run at my end and check to see whether all the generated ids are unique ? — Mihir, Dec 25 '13 at 08:57
No, I can't. They belong to my client. You will have to take my word for it. But if you would like to show us some code, we may be able to find the bug in it for you. — Dawood ibn Kareem, Dec 25 '13 at 08:58
**@Mihir**, I would be trying your test, meanwhile have a look at these things : [so discussion](http://stackoverflow.com/questions/325443/generate-uuid-in-java), and the [core sun doc](http://docs.oracle.com/javase/1.5.0/docs/api/java/util/UUID.html). — Siva Tumma, Dec 25 '13 at 09:08
@DavidWallace as i said in my update this method is called preSynchronizedServiceExecution in loop , if there is 10k times then it will be called 10k times. — Mihir, Dec 25 '13 at 09:31
but where is the call to `generateUUID`? I have read your code line by line and I can't see it. — Dawood ibn Kareem, Dec 25 '13 at 10:07
`Set uuidset = new HashSet(); for(int i=0; i<1000000; i++) { UUID newUUID = UUID.randomUUID(); if(uuidset.contains(newUUID)) throw new Exception("UUID " + newUUID + " is a duplicate"); uuidset.add(newUUID); }` does not cause a single duplicate in a million iterations and a few runs. — Joachim Isaksson, Dec 25 '13 at 10:56
ohh it is in processValueObjectKeyProperty this method.i will update it again. — Mihir, Dec 25 '13 at 13:51
@Mihir can you please add a few generated `UUIDs`? Also what came back from the database after you've inserted it would be quite interesting. — Thomas Jungblut, Dec 25 '13 at 15:13
It's possible that the same object appears more than once in one of the array lists, or even appears in more than one of the lists. If you're doing the key setting, then iterating through each list when you do the insert, then you'll get the most recently assigned key, but possibly several times. — Dawood ibn Kareem, Dec 25 '13 at 20:42
@DavidWallace I answered my question , thanks for your help. — Mihir, Dec 30 '13 at 08:41

score 0 · Accepted Answer · answered Dec 30 '13 at 08:40

Thanks for all your user who seriously study my question and spent some time to help me in solving it.

I found that the error was in Database layer of business logic foundation.

One of the Object needs to Updated but it was created using previous existing id so that i was getting the Duplicate Primary key Error.

I develop a Unit Test for id generation and tested UUID for more than one billion key , it is guaranteed to be unique, it is true in all circumstances.

Thanks again to everyone.

Generate Unique Identifier in Very Big Loop - Java

Update 1 - based on all the comments

Update 2 - Code

Update 3 (Method added)

Update 4 (Sample genearted UUID )

1 Answers1