1

we are working on image gallery where we expect 1 million to 40 million photos but we are thinking to keep them in photo folder but can one photo folder keep 40 million photos. if i directly keep them inside photo folder without creating any subfolder is there any issue of i have to create folder based on date of upload so that for any given date the photo uploaded in that day will go in that day folder like that .

i dont have any issue in creating that structure but for the knowledge point of view i want to know what is the problem if we keep few millions of photo directly in one folder. i have seen few websites who is doing this, for example if you will see this page all images are there under image folder.

something about 5 million images.all images are there under respective id for example under 4132808 so it shows that under images directory there are more than 5 million sub folder.is it ok to keep that much folder under one directory

http://www.listal.com/viewimage/4132808 http://iv1.lisimg.com/image/4132808/600full-the-hobbit%3A-an-unexpected-journey-photo.jpg

sachin
  • 727
  • 1
  • 5
  • 11
  • 1
    40 million photos á 0.5MB (which is not a lot) would come down to roughly 20TB. Are you sure you won't hit HDD limits? – Ingo Bürk Feb 23 '14 at 09:44
  • @IngoBürk yes hdd is no issue – sachin Feb 23 '14 at 09:52
  • Okay, just wanted to point this out. Do keep in mind, though, that a one-folder solution does not scale well without keeping to throw HDD into your machine (which also doesn't scale well). – Ingo Bürk Feb 23 '14 at 09:54

2 Answers2

2

Depends on the filesystem check the file system comparison page on Wikipedia for comparison. However you might want to sort in some structure like 

images/[1st 2 char of some kind of hash/[2nd 2 char of hash]/...

With this you create an easily reproducable path with drastically decreasing the number of files in one folder. You want to do this because in any event if you'd want to list the contents of the folder (or any application would need to do it) it would cause a huge performance problem.

What you can see on other sites is only how you publish those images. Of course they can be served seemingly from the safe url but in the underlying structure you want partition the files somehow.

Some calculations: Let's say you use the sha256 hash of the filename to create the path. That gives you 40 chars of [0-9a-f]. So if you chose to have 2 letters sub folders then you'd have 256 of folders on each level. Now let's assume you do it for 3 levels: ab/cd/ef/1234...png. That's 256^3 folder meaning 16 million. So even if you'll be fine up to couple billion images.

As for serving the files you can do something like this with apache + mod_rewrite:

RewriteEngine On
RewriteCond %{REQUEST_URI} !^/images/../../../.*
RewriteRule ^/images/(..)(..)(..)(.*)$ /images/$1/$2/$3/$4 [L]

This would reroute the requests for the images to the correct place

fejese
  • 4,601
  • 4
  • 29
  • 36
  • i can keep 100000 images in one subfolder so that i will have 4000 subfolder under photo folder – sachin Feb 23 '14 at 09:54
  • I'd recommend deepening you structure so you'd only have few thousands or at most few ten thousand files/folders at any particular level – fejese Feb 23 '14 at 10:01
2

See How many files can I put in a directory?.

Don't put all your files into one folder, it does not scale. If you don't want to start with a deep folder hierarchy, start simple and put the logic where you build the path to the folder in one class or method. This allows to simply rearrange later if needed.

Community
  • 1
  • 1
Bernhard
  • 8,583
  • 4
  • 41
  • 42