1

I'm getting OSError: [Errno 24] Too many open files error when using multiprocessing.Pool. The export function connects to a database, exports data into a file, uploads to S3, and removes the file from the filesystem.

with Pool() as pool:
    results = pool.map(export, work, 1)

for result in results:
    logger.info(result)

Any ideas why this is happening?

nobody
  • 270
  • 2
  • 14
  • 1
    Without seeing your worker code it is impossible to answer the question. You leave somewhere files, database connections, network connections or something of that ilk open and eventually you encounter the OS hard limit per process. Try closing all your handles as soon as you don't need them anymore and ensure nothing is open when you exit your worker function. – Hannu Aug 04 '21 at 19:17

1 Answers1

1

The error you are seeing is caused because there are too many open filedescriptors (these can be open filehandles or open sockets) or it could be caused by workers which are not disposed of properly. In order to see what these limits are you can use ulimit:

$ ulimit -Hn
1048576
$ ulimit -Sn
1024

Or there is a hard limit of over 1 million but a process will receive Errno 24 (EMFILE) as soon as 1024 files are opened. The most logical solution is that you have to many open files and you should close them as soon as you're done with them, if this is not the case you can also (temporarily) modify the soft limit by calling (the example below doubles the limit):

$ ulimit -n 2048

This post explains the underlying error and offers some more background: Difference between Linux errno 23 and Linux errno 24

amo-ej1
  • 3,279
  • 26
  • 35