2

I have an application that has 10000 threads running at a time. Each thread opens the same file. The problem is whenever I launch the application with 10K threads, the application terminates after creating 500 threads(file handles). I have tried the same application on Linux and is running fine after I tweaked the ulimit option. Is there any limit on the file handles a process can open in Windows? I have been googling and all I get is to change the entries in config.nt file in C\Windows\System32....

But I found out that the said file does not exist for 64 bit OS. Is there any way that I can change the limit in Windows?

My OS is WINDOWS 7 64 bit.

Shubham Jain
  • 16,610
  • 15
  • 78
  • 125
rahuljain1313
  • 141
  • 1
  • 1
  • 8
  • 1
    I'm struggling to comprehend what sort of problem you would be working with where the *correct* solution is to run 10000 threads. Especially one where every thread is accessing the same file. – Damien_The_Unbeliever Jun 29 '15 at 06:50
  • @Damien_The_Unbeliever But apart from the problem, is there any way by which we can change the limit in Windows? – rahuljain1313 Jun 29 '15 at 07:12
  • There is no fixed limit to the number of file handles in Windows. Your problem is elsewhere - for example, if the application is 32-bit, it is probably running out of address space. Details about the way in which the application fails might be useful in identifying the problem. – Harry Johnston Jun 29 '15 at 07:28
  • @HarryJohnston - according to [Mark Russinovich](http://blogs.technet.com/b/markrussinovich/archive/2009/09/29/3283844.aspx), there is: "In one of the rare cases where Windows sets a hard-coded upper limit on a resource, the Executive defines 16,777,216 (16*1024*1024) as the maximum number of handles a process can allocate" (of course, that may have changed since 2009 but I'm not aware of any reason why it would have done - and, of course, that's total handles, not just file handles - if that was your point) – Damien_The_Unbeliever Jun 29 '15 at 07:33
  • 1
    This is the error: **[Errno 24] Too many open files** The application is in python. And every time a thread accesses the file, it closes it as well. – rahuljain1313 Jun 29 '15 at 07:35
  • @Damien_The_Unbeliever: I stand corrected. Still, not the OPs problem. :-) – Harry Johnston Jun 29 '15 at 07:35
  • 5
    That's not a Windows error. At a guess, you're running out of Microsoft C runtime library file descriptors. – Harry Johnston Jun 29 '15 at 07:42
  • Possible duplicate of [Is there a limit on number of open files in Windows](https://stackoverflow.com/questions/870173/is-there-a-limit-on-number-of-open-files-in-windows) – Adrian McCarthy Apr 19 '18 at 20:48
  • 1
    OP did not claim a windows error. I can confirm the error message and the approx number of 500 (working from a console app on W10 x64). A follow up on Harry Johnston's suggestion abou MS C runtime library file descriptors would be wellcome. See https://learn.microsoft.com/en-us/cpp/c-runtime-library/file-handling. The number is 512. The solution therefore is: _setmaxstdio(newmax); – Jan May 21 '19 at 10:28

2 Answers2

7

To view the total number of handles (not just file handles) your application opened at a given time: Just to make sure it's the handle limit.

Download Process Explorer from https://technet.microsoft.com/en-us/sysinternals/processexplorer.aspx Make sure to set appropiate refresh speed. Open it and go to View -> Select Columns -> press on the tab "Process Performance"and click on "Handle Count".

For Windows 7 x64 bit, a process can have 16.711.680 handles opened simultaneously. If you want to check limits for yourself then read below. Check that by using a tool from Windows Internals Book (https://technet.microsoft.com/en-us/sysinternals/bb963901.aspx). Tool's name is TestLimit and you will find it in the lower part of the page under the Book Tools header.

There are no ways to increase this limit for Windows Operating Systems as far as I know, and I looked also.

As others stated, think of a method to minimize the large number of threads. Maybe your application closes the file, but not the handle. My advice, if you really need using very large handle count, start a new process every time handle count is about 16m.

xrayder
  • 71
  • 1
  • 3
1

running out of Microsoft C runtime library file descriptors (Harry Johnston) appears to be the right diagnosis. https://learn.microsoft.com/en-us/cpp/c-runtime-library/file-handling. The default max is 512 open file descriptors. The solution therefore is a line of code early in main:

_setmaxstdio(newmax);

How to do that in Python is another question.

Jan
  • 447
  • 6
  • 8
  • But the max appears 2048, which will not help the OP quite enough – Jan May 21 '19 at 10:45
  • 1
    Actually, on W10 x64, by experimenting, I found a max of 8192 – Jan May 21 '19 at 10:48
  • Quote MS: "Technically, the upper limit is defined by the preprocessor macro _NHANDLE_ defined in corecrt_internal_lowio.h in the Windows SDK sources for the UCRT. IntelliSense tells me its definition is (128 * (1 << 6)), or 8192" – Jan May 22 '19 at 08:03