3

I have to check how much time do_something() takes in total for a pair of lists containing 30k elements. Below is my code

def run(a, b, data):
    p = datetime.datetime.now()
    val = do_something(a, b, data[0], data[1])
    q = datetime.datetime.now()
    res = (q - p).microseconds
    return res

Next, I call this using the following code:

func = functools.partial(run, a, b)

x = np.linspace(500, 1000, 30000).tolist()
y = np.linspace(20, 500, 30000).tolist()

data = zip(x, y)

with multiprocessing.Pool(processes=multiprocessing.cpu_count()) as pool:
    d = pool.map(func, data)
res = sum(d)

Whenever I run this, I keep getting OSError: [Errno 24] Too many open files. How do I fix this?

Somnath Rakshit
  • 555
  • 1
  • 6
  • 17

1 Answers1

4

You can use ulimit -u 2048 to raise the process limit.

Use ulimit -a to check the current limits.

smanna
  • 483
  • 7
  • 19
  • Current limit was around 325k. Tried this and raised limit to 10 million, still the same error at the same time. – Somnath Rakshit Dec 15 '20 at 08:29
  • 1
    Use timeout or gc.collect() to make sure the processes are garbage collected. Looks like it's saving the proc file descriptors which is overflowing. Also use `pool.join()` after map. – smanna Dec 15 '20 at 08:48