1

I'm creating an Archiving System that let me store scanned files in a directory. I put my images in one directory called upload.

I made the system using Laravel framework in windows base XAMPP server.

  • One thing that worries me is that images that will be stored in that directory can go about millions.

  • The files will be corrupted or will reach file limit.

  • The searching of images could be very slow

As of this moment I have 12 thousand images stored in that directory and find searching not a problem. But I worry it could be troublesome in the future.

So any recommendation or storing million images in one directory is fine.?

Ikong
  • 2,540
  • 4
  • 38
  • 58

2 Answers2

2

Hopefully that helps in removing your concerns:

One thing that worries me is that images that will be stored in that directory can go about millions.

Hardware right now is capable of handling a huge number of files for a cheap price, and it heavily depends on the platform you are building and the ratio those files are growing. But if you are not into managing this yourself, a solution like Amazon S3 can help.

In the case of directories, preferably you use a mechanism that helps you splitting those files in multiple ones, an example would be to use the first 2-3 characters of the name of the file to build a directory you can store the file in : test.doc will turn into t/e/s/test.doc

The files will be corrupted or will reach file limit.

If the upload process is done successfully, the case where the file is corrupt needs more logic to be handled; depending on the file type, like this

The searching of images could be very slow

An index is your best solution , to have a database of all the files and its paths, this will easily give you a direct access to file without going through searching for them; keeping the name, path, and other info for example.

if you are talking about searching their content, if it is not only images, I recommend taking a look at Lucene based products like Solr or Elastic Search

Community
  • 1
  • 1
Rabea
  • 1,938
  • 17
  • 26
0

Technically you can have that many in one directory, ext2/ext3 filesystems can have up to ~ 1020 files in one directory, but accessing them can be very slow, usually as a rule of thumb - one does not put over 10-20 thousand files in one directory to avoid performance issues.

Plus for example FAT32 does not support over 65k

See How many files can I put in a directory?

So you can divide them into subdirectories vie some pattern, for example by year/month/day/username/etc depending on your loads.

I have a system that stores about 0.5-2k files a day and use filepaths base_dir/year/month/day/fulldate_filename.extension. This way I can later easily divide archive into volumes and search by date will be nearly instant.

Community
  • 1
  • 1
Vasfed
  • 18,013
  • 10
  • 47
  • 53