Image URL Naming Scheme

Question

Prologue: I'm building a sort of CMS/social networking service that will host many images.

I'm intending on using Eucalyptus/Amazon S3 to store the images and was wondering about the significance of the seemingly-random file-names used by sites like Tumblr, Twitter, &c., e.g.

31.media.tumblr.com/d6ba16060ea4dfd3c67ccf4dbc91df92/tumblr_n164cyLkNl1qkdb42o1_500.jpg

and

pbs.twimg.com/media/Bg7B_kBCMAABYfF.jpg

How do they generate these strings, and what benefits does this incur over just incrementing an integer for each file name? Maybe just random characters? Maybe hashing an integer?

Thanks!

Answer 1

Twitter uses an encoding method called 'snowflake'. There is github source

The basic format encodes a timestamp (42 bits), data center id (5 bits), and worker id (the computer at the data center; 5 bits)

For tweet IDs, they write the value as a long decimal number. Tweet ID '508285932617736192' is hex value '070DCB5CDA022000'. The first 42 bits are the timestamp (the time_t value is 070DCB5C + the epoch, 1291675244). The next five bits are the data center (in this case, '1'), and the next five bits are the worker id ('2').

For images, they do the exact same thing, but use base64 encoding (following the RFC 4648 standard for URL encoding; the last two base64 characters are hyphen and underscore).

BwjA8nCCcAAy5zA.jpg decodes as 2014-09-02 20:23:58 GMT, data center #1, worker #7

Answer 2

This is a way to organize media and to guarantee that media will not get written over if another file has the same file name. For example if Twitter had a million photos in its pbs.twimg.com/media/ directory and it is possible that two out of those million photos were named cat.jpg, Twitter would run into an issue uploading the second file with the same name or calling for a file where two exist with the same name. In result, Twitter (amongst other applications) has created a way to prevent the database from getting those two files mixed up and in result renames the file after compressing it to a file name with much more specificity: a set of numbers, letters, and symbols that may seem random but are incrementally generated.

In your CMS, I suggest creating some sort of failsafe to prevent two files from clashing, whether it's one trying to write over another when uploaded or if it's retrieving one file that has the same name as another. You can do this in a few different ways. One method would be as I just described, rename the file and create a system that auto-increments the files' names. Do not generate these file names in an obvious pattern because then all media will be easily accessible through the address bar. This is another reason why the URLs are not as readable.

You can also apply the file_exists() function in your uploader. This is a PHP function that checks whether or not a file with a certain name already exists in a certain directory. Read more about that function here.

Hope this helps.

Answer 3

My guess about the tumblr file naming scheme is as follows:

• d6ba16060ea4dfd3c67ccf4dbc91df92 - hash of the image file, might be MD5 or SHA-1
• tumblr_n164cyLkNl1qkdb42o1_500.jpg - several parts:
• tumblr_ - obvious prefix to advertise the site
• n164cyLkNl1qkdb42o - consists of 2 parts, 10 characters before '1' and 7 after
• n164cyLkNl - some kind of a hash for post ID that the image belongs to. Might be a custom alphabet Base64 value
• qkdb42o - hash of the tumblr blog name.
• Then goes the number, in this case '1' - # of image in a photoset, if it's a single photo then it's just '1'.
• Finally, _500 - maximum width of image in pixels.

Source: I have collected quite a lot of images and tags from tumblr and the pattern turned out to be obvious. You can see how tagging manner is the same for the same blog name hash, while tags of posts with same post number hash are 100% identical.

Now, if only there was a way to decode those hashes back to original value (assuming they're not actually hashes but encoded values, which is unlikely).