1

I'm using Python on Windows to split a large file (few GBs) into many smaller files. The file is splitted by line's content, so I need all the small files to be opened in the same time, and the content to be written into them.

I got IOError: [Errno 24] Too many open files when trying to open one of the files, and according to the debugger there are already 507 open files.

Is there a way to raise the maximum allowed open files?

I read about the ulimit option in macOS, but couldn't find an equivalent option on Windows.

Also- why do I get this error after 507 files? Is this specifically the maximum?

Thanks

stovfl
  • 14,998
  • 7
  • 24
  • 51
Shir
  • 1,157
  • 13
  • 35
  • 1
    Why do you need all the files to be opened at the same time? Why not split the big file in memory and then write out the smaller ones sequentially? – vencaslac Nov 12 '18 at 11:30
  • The large file (which actually might be few large files one after the other) might be *very* large (overall including all files- few GBs) – Shir Nov 12 '18 at 11:36
  • please have a look at this Q/A, your question seems to be a duplicate of it https://stackoverflow.com/questions/6475328/read-large-text-files-in-python-line-by-line-without-loading-it-in-to-memory – vencaslac Nov 12 '18 at 11:38
  • Possible duplicate of [Read large text files in Python, line by line without loading it in to memory](https://stackoverflow.com/questions/6475328/read-large-text-files-in-python-line-by-line-without-loading-it-in-to-memory) – vencaslac Nov 12 '18 at 11:39
  • I know exactly how to read the file, my problem is with writing it to other files simultaneously, it's not related to the answer in the link – Shir Nov 12 '18 at 11:41
  • Possible duplicate of [fopen problem - too many open files](https://stackoverflow.com/questions/3184345/fopen-problem-too-many-open-files) – stovfl Nov 12 '18 at 12:11
  • I agree it is a similar question, but the question in the link doesn't give a good solution @stovfl – Shir Nov 12 '18 at 12:30

1 Answers1

5

Apparently 512 is the maximum in python. I found the solution here- https://stackoverflow.com/a/28212496/8875017

import win32file
win32file._setmaxstdio(2048)
Shir
  • 1,157
  • 13
  • 35