11

It's well known that in Windows a directory with too many files will have a terrible performance when you try to open one of them. I have a program that is to execute only in Linux (currently it's on Debian-Lenny, but I don't want to be specific about this distro) and writes many files to the same directory (which acts somewhat as a repository). By "many" I mean tens each day, meaning that after one year I expect to have something like 5000-10000 files. They are meant to be kept (once a file is created, it's never deleted) and it is assumed that the hard disk has the required capacity (if not, it should be upgraded). Those files have a wide range of sizes, from a few KB to tens of MB (but not much more than that). The names are always numeric values, incrementally generated. I'm worried about long-term performance degradation, so I'd ask:

  • Is it OK to write all to the same directory? Or should I think about creating a set of subdirectories for every X files?
  • Should I require a specific filesystem to be used for such directory?
  • What would be the more robust alternative? Specialized filesystem? Which?
  • Any other considerations/recomendations?
Fabio Ceconello
  • 15,819
  • 5
  • 38
  • 51
  • One thing to keep in mind though, even though using nested dirs a little (see accepted answer), don't try to evade the filesystem by putting everything in a humongous binary blob file (Windows :wink:) – jørgensen Jan 05 '12 at 00:56
  • Since files come in each day, make a structure like 2012/1/1/3598673958.png etc. It will make it easy to see what's going on in terms of traffic, etc. – Tom Andersen Jan 12 '12 at 15:07
  • Related: http://stackoverflow.com/questions/5019371/storing-accessing-up-to-10-million-files-in-linux – Roman Shapovalov Jan 04 '17 at 14:51

6 Answers6

10

It depends very much on the file system.

ext2 and ext3 have a hard limit of 32,000 files per directory. This is somewhat more than you are asking about, but close enough that I would not risk it. Also, ext2 and ext3 will perform a linear scan every time you access a file by name in the directory.

ext4 supposedly fixes these problems, but I cannot vouch for it personally.

XFS was designed for this sort of thing from the beginning and will work well even if you put millions of files in the directory.

So if you really need a huge number of files, I would use XFS or maybe ext4.

Note that no file system will make "ls" run fast if you have an enormous number of files (unless you use "ls -f"), since "ls" will read the entire directory and the sort the names. A few tens of thousands is probably not a big deal, but a good design should scale beyond what you think you need at first glance...

For the application you describe, I would probably create a hierarchy instead, since it is hardly any additional coding or mental effort for someone looking at it. Specifically, you can name your first file "00/00/01" instead of "000001".

Nemo
  • 70,042
  • 10
  • 116
  • 153
  • S3 has no such limitations, since it really has no folders. You can toss 10 million objects at the root. But then you are 'stuck' on S3, as even a backup down to an earthly drive formatted in ext* would likely not work too well. – Tom Andersen Jan 12 '12 at 15:05
  • I had several >32.000 files directorys in use before ext4 was even heard of. So default filesystem these times where ext3 I guess. Is the limit mentioned a configuration thing? – dronus Feb 22 '14 at 01:40
  • @dronus: I am sure I found that number somewhere, but apparently my information is out of date for ext3 (see http://serverfault.com/a/187196). Personally, I still would not try to put millions of files in a directory except with XFS. – Nemo Feb 22 '14 at 02:25
  • I think the number was meant as the limit of folders inside a folder, this limit is mentioned sometimes on the web. Still in question if it holds today.. – dronus Feb 23 '14 at 15:15
  • @dronus: Where a Windows person says "folder", a Unix person says "directory". We are discussing the number of files that you can put in a directory... – Nemo Feb 23 '14 at 15:49
  • @Nemo: I used an Amiga in the old times, which had "directorys" and use linux everyday. I just don't know how I came about the word "folder" here. Still a marginal discussion theme... – dronus Feb 25 '14 at 10:35
  • In https://medium.com/@hartator/benchmark-deep-directory-structure-vs-flat-directory-structure-to-store-millions-of-files-on-ext4-cac1000ca28 dated Dec 2018, it is stated that EXT4 has a limit of a little more than 10M files per directory. – Law29 Jul 06 '19 at 15:54
  • "ext2 and ext3 have a hard limit of 32,000 files per directory" - no. https://stackoverflow.com/questions/466521/how-many-files-can-i-put-in-a-directory#466596 – symcbean Dec 09 '20 at 16:58
  • Please provide some citation for XFS being designed for this purpose. My experience is it slows down significantly after a few million files, as does another poster described here: https://serverfault.com/questions/870355/how-to-benchmark-the-performance-of-xfs-with-a-large-number-of-files#comment1479998_870355 – Chris Stryczynski Jun 30 '23 at 14:46
  • @ChrisStryczynski "Scalabilty of the structures and indexes on disk and the algorithms for iterating them are adequate for supporting PB scale filesystems with billions of inodes" https://mirrors.edge.kernel.org/pub/linux/utils/fs/xfs/docs/xfs_filesystem_structure.pdf#page=12 Every data structure in XFS was designed for scalability from the start, and my experience is that it performs well. However, my experience is with terabyte-scale files, not large directories. – Nemo Jun 30 '23 at 17:18
7

If you use a filesystem without directory-indexing, then it is a very bad idea to have lots of files in one directory (say, > 5000).

However, if you've got directory indexing (which is enabled by default on more recent distros in ext3), then it's not such a problem.

However, it does break quite a few tools to have many files in one directory (For example, "ls" will stat() all the files, which takes a long time). You can probably easily split it into subdirectories.

But don't overdo it. Don't use many levels of nested subdirectory unnecessarily, this just uses lots of inodes and makes metadata operations slower.

I've seen more cases of "too many levels of nested directories" than I've seen of "too many files per directory".

MarkR
  • 62,604
  • 14
  • 116
  • 151
3

The best solution I have for you (rather than quoting some values from a micro-filesystem-benchmark) is to test it yourself.

Just use the file system of your choice. Create some random test data for 100, 1000 and 10000 entries. Then, measure the time it takes your system to perform the action you are concerned about time-wise (opening a file, reading 100 random files, etc).

Then, you compare the times and use the best solution (put them all into one directory; put each year into a new directory; put each month of each year into a new directory).

I do not know in detail what you are using, but creating a directory is a one time (and probably quite easy) operation, so why not do it instead of changing filesystems or trying some other more time-consuming stuff?

Carsten
  • 17,991
  • 4
  • 48
  • 53
1
  • Is it OK to write all to the same directory? Or should I think about creating a set of subdirectories for every X files?

In my experience the only slow down a directory with many files will give is if you do things such as getting a listing with ls. But that mostly is the fault of ls, there are faster ways of listing the contents of a directory using tools such as echo and find (see below).

  • Should I require a specific filesystem to be used for such directory?

I don't think so with regards to amount of files in one directory. I am sure some filesystems perform better with many small files in one dir whilst others do a better job on huge files. It's also a matter of personal taste, akin to vi vs. emacs. I prefer to use the XFS filesystem so that'd be my advice. :-)

  • What would be the more robust alternative? Specialized filesystem? Which?

XFS is definitely robust and fast, I use it in many places, as boot partition, oracle tablespaces, space for source control you name it. It lacks a bit on delete performance, but otherwise it's a safe bet. Plus it supports growing the size whilst it is still mounted (that's a requirement actually). That is you just delete the partition, recreate it at the same starting block and whatever ending block that's larger than the original partition, then you run xfs_growfs on it with the filesystem mounted.

  • Any other considerations/recomendations?

See above. With the addition that having 5000 to 10000 files in one directory should not be a problem. In practice it doesn't arbitrarily slow down the filesystem as far as I know, except for utilities such as "ls" and "rm". But you could do:

find * | xargs echo
find * | xargs rm

The benefit that a directory tree with files, such as directory "a" for file names starting with an "a" etc., will give you is that of looks, it looks more organised. But then you have less of an overview... So what you're trying to do should be fine. :-)

I neglected to say you could consider using something called "sparse files" http://en.wikipedia.org/wiki/Sparse_file

aseq
  • 451
  • 1
  • 6
  • 23
  • 2
    Except for the suggestion to use XFS, this is mostly wrong. Every time you attempt to open a file on a typical Linux file system, the system will scan the entire directory listing from the beginning to locate that file by name. So, for example, to "stat" every file in the directory is an O(n^2) operation. Very noticeable at tens of thousands of files. – Nemo Jan 05 '12 at 00:32
  • 2
    Directory indexing is enabled in ext3 by default in some distributions (Centos5, certainly). Use tune2fs to inspect and change the directory indexing option (dir_index). – MarkR Jan 05 '12 at 01:42
  • 1
    "Except for the suggestion to use XFS, this is mostly wrong." What's wrong is not elaborating. Just saying "this is wrong" is not helpful at all. In my practical experience the slow down or even inability of "ls" and "rm to work well or at all with many 1000s of files is much more noticeable than any possible slowdowns otherwise. Which can even be tuned. – aseq Jan 05 '12 at 08:23
1

In addition to the other answers, if the huge directory is managed by a known application or library, you could consider replacing it by something else, e.g:

  • a GDBM index file; GDBM is a very common library providing indexed file, which associates to an arbitrary key (a sequence of bytes) an arbitrary value (another sequence of byte).
  • perhaps a table inside a database like MySQL or PostGresQL. Be careful about indexing.
  • some other way to index data

The advantages of the above approaches include:

  1. space performance for a large collection of small items (less than a kilobyte each). A filesystem need an inode for each item. Indexed systems may have much less granularity
  2. time performance: you don't access the filesystem for every item
  3. scalability: indexed approaches are designed to fit large needs: either a GDBM index file, or a database can handle many millions of items. I'm not sure your directory approach will scale as easily.

The disadvantage of such approach is that they don't show as files. But as MarkR's answer remind you, ls is behaving quite poorly on huge directories.

If you stick to a filesystem approach, many software using large number of files are organizing them in subdirectories like aa/ ab/ ac/ ...ay/ az/ ba/ ... bz/ ...

Community
  • 1
  • 1
Basile Starynkevitch
  • 223,805
  • 18
  • 296
  • 547
  • I considered using BLOBs in a database (since I already have the program accessing a MySQL database) but was worried about the long-term stability (sometimes I hear scary stories about things that happen when you load too much data in MySQL). I also considered MongoDB (even implementend a POC in it), but it has some limitations when the machine isn't 64 bits, and I don't want to make this into one more requirement for my program. But GDBM seems to be a very good alternative, thanks for the tip. – Fabio Ceconello Jan 05 '12 at 12:32
  • Any database should be periodically backed up (e.g. with `mysqldump`). I am not familiar of horror stories for well administrated and well designed MySQL databases. And GDBM is simpler than MySQL. But making backups is always essential (hardware disks are failing!). – Basile Starynkevitch Jan 05 '12 at 12:35
  • 1
    Of course, you should backup the data, not only the file containing it. for MySQL, using `mysqldump`; for GDBM, write your own backup routine or use `gdbmexport` – Basile Starynkevitch Jan 05 '12 at 13:17
0

It is bad for performance to have a huge number of files in one directory. Checking for the existence of a file will typically require an O(n) scan of the directory. Creating a new file will require that same scan with the directory locked to prevent the directory state changing before the new file is created. Some file systems may be smarter about this (using B-trees or whatever), but the fewer ties your implementation has to the filesystem's strengths and weaknesses the better for long term maintenance. Assume someone might decide to run the app on a network filesystem (storage appliance or even cloud storage) someday. Huge directories are a terrible idea when using network storage.

Kyle Jones
  • 5,492
  • 1
  • 21
  • 30
  • Directory indexing is enabled in ext3 by default in some distributions (Centos5, certainly). Use tune2fs to inspect and change the directory indexing option (dir_index). – MarkR Jan 05 '12 at 01:43