13

I have a project written for Android devices. It generates a large number of files, each day. These are all text files and images. The app uses a database to reference these files.

The app is supposed to clear up these files after a little use (perhaps after a few days), but this process may or may not be working. This is not the subject of this question.

Due to a historic accident, the organization of the files are somewhat naive: everything is in the same directory; a .hidden directory which contains a zero byte .nomedia file to prevent the MediaScanner indexing it.

Today, I am seeing an error reported: 

java.io.IOException: Cannot create: /sdcard/.hidden/file-4200.html
  at java.io.File.createNewFile(File.java:1263)

Regarding the sdcard, I see it has plenty of storage left, but counting 

$ cd /Volumes/NO_NAME/.hidden
$ ls | wc -w
9058

Deleting a number of files seems to have allowed the file creation for today to proceed.

Regrettably, I did not try touching a new file to try and reproduce the error on a commandline; I also deleted several hundred files rather than a handful.

However, my question is:

  • are there hard limits on filesize or number of files in a directory?
  • am I even on the right track here?

Nota Bene: The SD card is as-is - i.e. I haven't formatted it, so I would guess it would be a FAT-* format.

The FAT-32 format has hard limits of filesize of 2GB (well above the filesizes I am dealing with) and a limit of number of files in the root directory. I am definitely not writing files in the root directory.

Jonas
  • 121,568
  • 97
  • 310
  • 388
jamesh
  • 19,863
  • 14
  • 56
  • 96

4 Answers4

27

There's a limit on 512 entries in the root directory of FAT filesystem. This limit comes about because the root directory lives in a particular place on FAT filesystems.

For other directories this limit is not in place. Additionally, FAT32 removed the 512 entry limit for the root filesystem by treating the root directory the same as any other directory.

Using long filenames - i.e. not in 8.3 format - means than a single file uses multiple directory entries.

Some Googling finds some people claiming that a FAT32 directory can have a maximum of 65,536 entries (which would be fewer files if they had long file names). However, none of the sources that mentioned this limit seemed that reliable so I thought I'd test this.

I wrote a script which creates files with 30 character filenames, meaning each file would need 4 directory entries. When the script got to file 16,384 it failed with an IO error and I couldn't create more files in my test directoy. So this does seem to validate the 65,536 entry limit.

If you're hitting this limit at 9,000 files then your files must be using at least 7 entries each which corresponds to filenames that are at least 66 characters long. Does this match what you're doing? (Or you could have some short filenames and some very, very long ones, but you get the idea.)

David Webb
  • 190,537
  • 57
  • 313
  • 299
  • Great answer, including primary research. This sounds very feasible. The bug has v. recently re-occurred so will test this theory, and report back. – jamesh Apr 20 '10 at 23:22
  • 1
    Absolutely bang on the money. 6k files of approx 108 chars in the filename. 3.5k files of approx 9 chars in the filename. Thanks. – jamesh Apr 21 '10 at 20:05
  • 1
    I'm impressed you did the research. Question: did you notice any performance hits as the number of files got large? – Peri Hartman Nov 30 '13 at 01:28
6

I have done some more testing ( reliable testing :-) ), trying to write 100,000 short-named directories in a single directory. The limit was reached at 65,536.

The test was done on Nexus One running Android 2.2, but I believe the result holds for any FAT32 SD card.

OferR
  • 1,634
  • 19
  • 21
  • Great! thanks! This concurs with @Dave Webb's answer, but with primary research on a device. Good job! – jamesh Aug 02 '10 at 22:25
5

I think the limit of files in a directory in Fat32 also depends on the length of the filenames

http://www.microsoft.com/whdc/system/platform/firmware/fatgen.mspx

Victor
  • 9,210
  • 3
  • 26
  • 39
  • Indeed, the FAT32 specification says that: "The maximum valid directory size is 2^21 bytes." Directory is a special file of 32-byte directory entries, each containing a single 8.3 filename. That implies the limit of 65,536 short-names files, while long filename occupy multiple directory entries, effectively decreasing the limit. I believe the specification should be the primary source of information rather than ad-hoc experiments on a concrete, single setting. The accepted answer is still valuable, even though it actually does not prove anything general. – Ondřej Bouda May 07 '13 at 17:00
-1

There is a practical limit much lower than 512, depending on the Android device you connect :
When I connect my phone (Samsung A5 2015) to my computer (Ubuntu) : from my computer, I can easily read/write files on my phone, except for a directory which contains 300 files. It is infinitely slow for any simple action : list the content, copy a file, delete a file . . . So, I think that there is a practical limit to the number of files for my phone. I have also tablets with which I do not have this problem.

flaja
  • 1
  • 1
  • I have encountered a problem of very very slow copy, I have read the previous answers that did not help me. I have finally located the problem **on this device, on this directory**. And it answered my question. I think it can be useful for others. – flaja Jan 27 '20 at 17:02
  • Please don't add "Me too" as answers. It does not provide an answer to the question. If you have a different but related question, then ask it (reference the current question if it provides context). If you're interested in this specific question, you can upvote it, leave a comment, or start a bounty once you have enough reputation. – hongsy Jan 28 '20 at 00:51
  • my question was exactly the same :Is there a limit for the number of files in a directory on an SD card? (on my phone). I have found that the number of 300 files was too much for a practical use with this device. If you want me to delete my answer tell me. I will delete it. – flaja Jan 29 '20 at 08:37