Too many open files error with Popen of subprocess

Question

I'm using Python's subprocess module to call a command to write values from a file to memory. It looks like:

import subprocess

f = open('memdump', 'r')
content = [line.split()[1] for line in f]
f.close()
tbl_pt0 = 0x4400
tbl_pt1 = 0x4800
process = 0
i = 0
for value in content:
    p1 = subprocess.Popen("echo \"jaguar instance=0; jaguar wr offset=0x%x value=%s\" | pdt" \
% (tbl_pt0, value), shell = True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
    p2 = subprocess.Popen("echo \"jaguar instance=0; jaguar wr offset=0x%x value=%s\" | pdt" \
% (tbl_pt1, value), shell = True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
    tbl_pt0 += 0x4
    tbl_pt1 += 0x4

Here the pdt is a program that writes the value. There're 256 values in the file, so the for loop runs 256 loops. But when I get to loop 253, I got too many open files error in subprocess.py. Any one has any idea? Thanks.

don't use `subprocess.PIPE` unless you read/write from the corresponding pipe (you don't in your example). — jfs, Aug 06 '14 at 14:53

Maciej Gol · Accepted Answer · 2014-07-10T17:16:39.867

The reason is that subprocess.Popen call is asynchronous in that it returns immediately without waiting for the spawned process to exit. That said, you are quickly creating 2x256 processes. Each of them has 2 pipes each taking up 1 file descriptor. A single process can have a limited amount of file descriptors opened at any one time, and you are reaching it because you don't wait for the processes and pipes to close. You can either wait for them to exit, e.g. p.communicate() where p is the return value of subprocess.Popen, or increase the maximum file descriptors opened at once:

permamently - add fs.file-max = 100000 to /etc/sysctl.conf
temporarily (until reboot) - sysctl -w fs.file-max=100000

score 1 · Answer 2 · answered Jul 10 '14 at 17:09

With every Popen you create 6 new filehandles. Perhaps you can run the processes one by one, and use communicate instread of echo:

import subprocess
from itertools import count

JAGUAR = "jaguar instance=0; jaguar wr offset=0x%x value=%s"

with open('memdump', 'r') as f:
    content = [line.split()[1] for line in f]

for tbl_pt0, tpl_pt1, value in zip(count(0x4400,4), count(0x4800,4), content):
    subprocess.Popen(["pdt"], stdout=subprocess.PIPE, stderr=subprocess.PIPE,
        close_fds=True).communicate(JAGUAR % (tbl_pt0, value))
    subprocess.Popen(["pdt"], stdout=subprocess.PIPE, stderr=subprocess.PIPE,
        close_fds=True).communicate(JAGUAR % (tbl_pt1, value))

Too many open files error with Popen of subprocess

2 Answers2

Linked