20

Scenario

Users can post an item and include up to 5 images with the post, each image that's uploaded needs to be resampled and resized - a total of 4 extra images are created. Meaning if the user uploads 5 images end up with 25 images total to store.

Assumptions

  • The images have been properly checked and they're valid image files
  • The system has to scale (let's assume 1000 posts in the first instance, so maximum 5000 images)
  • Each image is renamed in relation to the auto_incremenet id of the db post entry and includes relevant suffix i.e. 12345_1_1.jpg 12345_2_1.jpg - so there's no issues with duplicates
  • The images aren't of a sensitive nature, so there's no issues with having them directly accessible (although directory listing would be disabled)

Possible approaches

  • Given the ids are unique we could just drop them into one folder (ineffecient after a certain point).
  • Could create a folder for each post and place all the images into that, so ROOT/images/12345 (again, would end up with a multitude of folders)
  • Could do an image store based on date, i.e. each day a new folder is created and the days images are stored in there.
  • Could store the images based on the resized type, i.e. all the original files could be stored in one folder images/orig all the thumbnails in images/thumb (i think Gumtree uses an approach like this).
  • Could allow X amount of files to be stored in one folder before creating another one.

Anyone have experience on the best practices / approaches when it comes to storing images scalably?

Note: I prememt someone will mention S3 - let's assume we want to keep the images locally for the time being.

Thanks for looking

Rarriety
  • 229
  • 2
  • 4
  • 1
    Whether or not putting all images in one folder is `inefficient` or not depends on the filesystem used. On btrFS or Reiserfs, the number of items in a directory is irrelevant for lookup times. Check the documentation for your particular filesystem. – Johan Aug 26 '11 at 10:03
  • 3
    Having lots of images in the same folder can be a pain, especially if you want to list/backup the contents over (S)FTP – Markus Hedlund Aug 26 '11 at 10:08
  • I probably should of been somewhat more careful when claiming "inefficient" - it was less about the filesystems ability to serve up the images (it's what they do, and infact they're pretty damn good at it) - but more about the inherent overheads of having 5000 images stored in one folder and how certain applications will probably collapse in a heap trying to list/load them (or take all day to do it). – Rarriety Aug 26 '11 at 10:32
  • What applications ? struggling to list 5000 files in a directory ? on what system ? thats tiny !!!! – Manse Aug 26 '11 at 10:55

3 Answers3

13

We have such a system in heavy production with 30,000+ files and 20+ GB to date...

   Column    |            Type             |                        Modifiers                         
-------------+-----------------------------+----------------------------------------------------------
 File_ID     | integer                     | not null default nextval('"ACRM"."File_pseq"'::regclass)
 CreateDate  | timestamp(6) with time zone | not null default now()
 FileName    | character varying(255)      | not null default NULL::character varying
 ContentType | character varying(128)      | not null default NULL::character varying
 Size        | integer                     | not null
 Hash        | character varying(40)       | not null
Indexes:
    "File_pkey" PRIMARY KEY, btree ("File_ID")

The files are just stored in a single directory with the integer File_ID as the name of the file. We're over 30,000 with no problems. I've tested higher with no problems.

This is using RHEL 5 x86_64 with ext3 as the file system.

Would I do it this way again? No. Let me share a couple thoughts on a redesign.

  1. The database is still the "master source" of information on the files.

  2. Each file is sha1() hashed and stored in a filesystem hierarchy based on that hash: /FileData/ab/cd/abcd4548293827394723984723432987.jpg

  3. the database is a bit smarter about storing meta-information on each file. It would be a three table system:

    File : stores info such as name, date, ip, owner, and a pointer to a Blob (sha1)
    File_Meta : stores key/value pairs on the file, depending on the type of file. This may include information such as Image_Width, etc...
    Blob : stores a reference to the sha1 along with it's size.

This system would de-duplicate the file content by storing the data referenced by a hash (multiple files could reference the same file data). It would be very easy to backup sync the file database using rsync.

Also, the limitations of a given directory containing a lot of files would be eliminated.

The file extension would be stored as part of the unique file hash. For example, if the hash for an empty file were abcd8765... An empty .txt file and empty .php file would refer to the same hash. Rather, they should refer to abcd8765.php and abcd8765.txt. Why?

Apache, etc.. can be configured to automatically choose the content type and caching rules based on the file extension. It is important to store the files with a valid name and the extension which reflects the content of the file.

You see, this system could really boost performance by delegating the file delivery through nginx. See http://wiki.nginx.org/XSendfile.

I hope this helps in some way. Take care.

Community
  • 1
  • 1
gahooa
  • 131,293
  • 12
  • 98
  • 101
  • 1
    Upvote for a truly elegant architecture. I'm refactoring my static image server to use your ideas. Thanks- – Jens Roland Oct 16 '11 at 17:55
  • Just to clarify your proposed DB structure; separating File and Blob allows you to save the 40 bytes of the Hash field in the case of identical files (blobs), but is there any other reason for it? – Jens Roland Oct 17 '11 at 20:30
  • Secondly, since File and File_Meta have a 1-1 relation, I assume you only separated them to keep the File table as lean as possible? I mean, the File_Meta table can grow pretty big over time, so making sure you can access the basic File attributes without the overhead seems to make sense. But is there another reason for the separation? If you want to do filetype-specific data, how would you structure that? – Jens Roland Oct 17 '11 at 20:38
  • Wow. Just, wow. I do have one question though, you say the `Blob` stores a "reference to the sha1 along with its size". What do you mean by that? – Madara's Ghost Jan 23 '12 at 13:10
  • @JensRoland - I think File and File_Meta have a one-to-many relation, with each File_Meta row only storing one key/value pair (sort of like EAV). – Brilliand Feb 26 '13 at 20:57
1

I would store all the images in a single folder - the database then keeps a track of files names - keep it simple

Manse
  • 37,765
  • 10
  • 83
  • 108
0

First of, I would recommend creating a table for the images. This is a one row / image file table:

| id  | filename | type     | storage |
---------------------------------------
| 123 | 123.png  | original | store1  |
  • id auto incremental int or something equally unique.
  • filename the file base name. This allows you to move the file and just update the code. The filename could be {file_id}.{extension}.
  • type is the type of image: original, thumbnail, resized, whatever. Could also be the dimensions: 100x100, 500x, x500 (where 500x would be unlimited height and x500 would be unlimited width). These are just some examples.
  • storage would an identifier for where the file is, this could be a directory. Say that you store your images in post_images, filename is 123.png and storage is store1, then the path would be post_images/store1/123.png.

I have yet to try this myself, but I have problems with web apps storing 10k+ files in the same directory.

Markus Hedlund
  • 23,374
  • 22
  • 80
  • 109