change the maximum number of open files in Windows

Question

I would like to know how I can change the maximum number of open files in Windows. Similar to ulimit command under unix. Under my c:\windows\system32, I cannot find the CONFIG.NT file, thus changing the file number in CONFIG.NT file won't work for me.

The problem is I wrote a python script, in which I need to call subprocess.Popen(cmd, stdout=sp.PIPE, stdin=sp.PIPE, shell = True) 256 or 512 times. That means I need to launch 256 or 512 background processes. The script works fine whenever the number of launched process is smaller than 255.

As far as I know, there isn't any fixed maximum. Whatever is going wrong, this probably isn't the cause. — Harry Johnston, Dec 03 '14 at 23:38
What problem are you facing? It doesn't sound as though you have come up with the right solution. — David Heffernan, Dec 04 '14 at 12:20
Sounds like a limitation in Python. Nothing in Windows is likely to impose a one-byte limit on the number of child processes - although you should note that Windows may not perform well with that many processes running simultaneously. — Harry Johnston, Dec 06 '14 at 22:48

Ira Baxter · Answer 1 · 2014-12-04T12:24:32.270

2

The general claim is that Windows does not have an upper limit on files; just what you can manage in memory.

I think you will find useful: Russinvich's discussion of Windows handle resource limits useful.

We build parallel programs, that can process thousands of files. In spite of the claims, we seem to run into resource limits when we attempt to open (and hold open simultaneously) several thousand files.

We have found it to be useful to limit the number of opens to a few hundred. This doesn't seem to affect performance much over pushing the limits. This is pretty easily managed with a counting semaphore in your parallel application.

edited Dec 04 '14 at 12:24

answered Dec 04 '14 at 12:16

Ira Baxter

93,541
22
172
341

The problem is I wrote a python script, in which I need to call subprocess.Popen(cmd, stdout=sp.PIPE, stdin=sp.PIPE, shell = True) 256 or 512 times. That means I need to launch 256 or 512 background processes. The script works fine whenever the number of launched process is smaller than 255. – bunny Dec 04 '14 at 15:24
So why did you ask the question that you did, given this problem? – David Heffernan Dec 04 '14 at 21:15

score 1 · Answer 2 · edited May 23 '17 at 12:34

This is an old question, but I'll contribute. There is a limit, but it's not a limit of subprocesses per-say. Python opens file descriptors for each pipe on each subprocess. Your code is breaking at 255 because Window has a default limit of 512 open file descriptors (pretty close to the 2 pipes x 255 subprocesses).

See answer 3 in this question: Why python has limit for count of file handles?. This answer shows how to increase Window's default limit. You can change open limit to a max of 2048 per Windows limits.

change the maximum number of open files in Windows

2 Answers2