9

I need to read the contents of several thousands of small files at startup. On linux, just using fopen and reading is very fast. On Windows, this happens very slowly.

I have switched to using Overlapped I/O (Asynchronous I/O) using ReadFileEx, where Windows does a callback when data is ready to read.

However, the actual thousands of calls to CreateFile itself are still a bottleneck. Note that I supply my own buffers, turn on the NO_BUFFERING flag, give the SERIAL hint, etc. However, the calls to CreateFile take several 10s of seconds, whereas on linux everything is done much faster.

Is there anything that can be done to get these files ready for reading more quickly?

The call to CreateFile is:

            hFile = CreateFile(szFullFileName,
                GENERIC_READ,
                FILE_SHARE_READ | FILE_SHARE_WRITE,
                NULL,
                OPEN_EXISTING,
                FILE_ATTRIBUTE_NORMAL | FILE_FLAG_OVERLAPPED | FILE_FLAG_NO_BUFFERING | FILE_FLAG_SEQUENTIAL_SCAN,
                NULL);
Macker
  • 1,498
  • 1
  • 12
  • 17
  • 1
    It's possible that the bottleneck is the filesystem, is it possible to use a flat file instead of reading thousands of small files? – tenfour Sep 15 '11 at 13:18
  • Windows is notoriously bad in dealing with a large number of files in one directory, if that's your case. Is multithreading the thing an option, so you'll have 10 threads doing the reads in concur? – Eran Sep 15 '11 at 13:22
  • 1
    You should have a look at this question http://stackoverflow.com/questions/197162/ntfs-performance-and-large-volumes-of-files-and-directories – Carey Gregory Sep 15 '11 at 13:58
  • Hi, we have 12,000 files, not millions. We only care about reading the info, not writing it. Yes, the filesystem may well be the bottleneck. No, the files are not all in one directory, they're scattered over about 300 directories. – Macker Sep 15 '11 at 14:42
  • @Macker: In my experience it doesn't take millions at all. Tens of thousands is sufficient to degrade the CreateFile function very badly. I've seen a directory of 50,000 files take 5+ minutes to do a simple open/close on all of them. As best I've been able to determine, the cause is the reasons explained in the link I provided. – Carey Gregory Sep 15 '11 at 18:20
  • 2
    Do you have antivirus software running? Have you tried excluding the folder in question from scanning? – Harry Johnston Sep 16 '11 at 00:31

2 Answers2

12

CreateFile in kernel32.dll has some extra overhead compared to the kernel syscall NtCreateFile in ntdll.dll. This is the real function that CreateFile calls to ask the kernel to open the file. If you need to open a large number of files, NtOpenFile will be more efficient by avoiding the special cases and path translation that Win32 has-- things that wouldn't apply to a bunch of files in a directory anyway.

NTSYSAPI NTSTATUS NTAPI NtOpenFile(OUT HANDLE *FileHandle, IN ACCESS_MASK DesiredAccess, IN OBJECT_ATTRIBUTES *ObjectAttributes, OUT IO_STATUS_BLOCK *IoStatusBlock, IN ULONG ShareAccess, IN ULONG OpenOptions);

HANDLE Handle;
OBJECT_ATTRIBUTES Oa = {0};
UNICODE_STRING Name_U;
IO_STATUS_BLOCK IoSb;

RtlInitUnicodeString(&Name_U, Name);

Oa.Length = sizeof Oa;
Oa.ObjectName = &Name_U;
Oa.Attributes = CaseInsensitive ? OBJ_CASE_INSENSITIVE : 0;
Oa.RootDirectory = ParentDirectoryHandle;

Status = NtOpenFile(&Handle, FILE_READ_DATA, &Oa, &IoSb, FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, FILE_SEQUENTIAL_ONLY);

Main downside: this API is not supported by Microsoft for use in user mode. That said, the equivalent function is documented for kernel mode use and hasn't changed since the first release of Windows NT in 1993.

NtOpenFile also allows you to open a file relative to an existing directory handle (ParentDirectoryHandle in the example) which should cut down on some of the filesystem overhead in locating the directory.

In the end, NTFS may just be too slow in handling directories with large numbers of files as Carey Gregory said.

Chris Smith
  • 5,326
  • 29
  • 29
  • 2
    Neither is the statement about NTFS being "slow" nor the one about NtCreateFile being faster than CreateFile true - both are entirely made up. Unless someone posts some proof for that i will have to call bullshit - except for a few CPU-cycles both functions will behave night-identical and NTFS is a moderately fast filesystem. – specializt Jan 08 '15 at 19:12
  • See http://stackoverflow.com/questions/197162/ntfs-performance-and-large-volumes-of-files-and-directories – Chris Smith Jan 08 '15 at 22:28
  • 1
    Yeah .… thats no proof whatsoever. No data. – specializt Jan 08 '15 at 22:31
  • Agreed, one can use Process Monitor to find the [matching set of parms](https://social.msdn.microsoft.com/Forums/vstudio/en-US/5a6f9f80-b4d2-4047-8004-26f163508d0d/setcurrentdirectoryw-wont-drop-to-c-with-or-?forum=vcgeneral) for both functions. CreateDirectoryW (same thing) did a little more processing, but the extra overheads used in setting up and managing the structures for NTCreatedFIle amount to just about the same effect. – Laurie Stearn Mar 09 '16 at 13:29
0

Try paging in the MFT efficiently before issuing the Create file. This can be done by issuing FSCTL_ENUM_USN_DATA.

deadly
  • 1,194
  • 14
  • 24
GReg
  • 1