4

we have a project that constitutes a large archive of image files... We try to split them into sub-folders within the main archive folder. Each sub-folder contains up to 2500 files in it.

For example:

C:\Archive
C:\Archive\Animals\
C:\Archive\Animals\001 - 2500 files...
C:\Archive\Animals\002 - 2300 files..
C:\Archive\Politics\
C:\Archive\Politics\001 - 2000 files...
C:\Archive\Politics\002 - 2100 files...

Etc... What would be the best way of storing files in such way under Windows ? and why exactly, please ... ?

  • Later on, the files have their EXIF metadata extracted and indexed for keywords, to be added into a Lucene index... (this is done by a Windows service that lives on the server)
Denis
  • 4,718
  • 5
  • 18
  • 20
  • Will you be accessing the folder only locally, or over a network? It makes a big difference. – Mark Ransom Mar 21 '11 at 21:58
  • The folder will be accessed only locally by a Windows service that has a "FileSystemWatcher" component waiting for files to be added or deleted. The component is able to process large batches of new files... – Denis Mar 21 '11 at 22:01

1 Answers1

8

We have an application where we try to make sure we don't store more than around 1000 files in a directory. Under Windows at least, we noticed extreme degradation in performance over this number. The folder can theoretically store up to 4,294,967,295 in Windows 7. Note that because the OS does a scan of the folder, doing lookups and lists very quickly degrades as you add many more files. Once we got to 100,000 files in a folder it was almost completely unusable.

I'd recommend breaking down the animals even further, perhaps by first letter of name. Same with the other files. This will let you separate things out more so you won't have to worry about the directory performance. Best advice I can give is to perform some stress tests on your system to see where the performance starts to tail off once you have enough files in a directory. Just be aware you'll need several thousand files to test this out.

Scott
  • 16,711
  • 14
  • 75
  • 120
  • 1
    Some other good posts: http://stackoverflow.com/questions/1613087/does-storing-a-lot-of-images-in-a-single-directory-slow-down-image-retrieval http://stackoverflow.com/questions/466521/how-many-files-in-a-directory-is-too-many – Scott Mar 21 '11 at 22:03
  • Sure, thanks, the archive, at the moment, has about 2.000.000 files to test it with :) (of course, it is all broken into folders and sub-folders, but each with up to 2500 files in it...) – Denis Mar 21 '11 at 22:04
  • 6
    if you don't enumerate files and just open them, 400K files in a single folder is fine on NTFS. I've did measurements and was quite surprised by this fact. – Eugene Mayevski 'Callback Mar 22 '11 at 06:15
  • I like measurements and facts :) – Denis Jun 20 '12 at 14:10