Downsides of keeping file handles open?

Question

I am parsing some XML and write data to different files depending on the XML element that is currently being processed. Processing an element is really fast, and writing the data is, too. Therefore, files would need to open and close very often. For example, given a huge file:

for _, node in lxml.etree.iterparse(file):
    with open(f"{node.tag}.txt", 'a') as fout:
        fout.write(node.attrib['someattr']+'\n'])

This would work, but relatively speaking it would take a lot of time opening and closing the files. (Note: this is a toy program. In reality the actual contents that I write to the files as well as the filenames are different. See the last paragraph for data details.)

An alternative could be:

fhs = {}
for _, node in lxml.etree.iterparse(file):
    if node.tag not in fhs:
        fhs[node.tag] = open(f"{node.tag}.txt", 'w')

    fhs[node.tag].write(node.attrib['someattr']+'\n'])

for _, fh in fhs.items(): fh.close()

This will keep the files open until the parsing of XML is completed. There is a bit of lookup overhead, but that should be minimal compared to iteratively opening and closing the file.

My question is, what is the downside of this approach, performance wise? I know that this will make the open files inaccessible by other processes, and that you may run into a limit of open files. However, I am more interested in performance issues. Does keeping all file handles open create some sort of memory issues or processing issues? Perhaps too much file buffering is going on in such scenarios? I am not certain, hence this question.

The input XML files can be up to around 70GB. The number of files generated is limited to around 35, which is far from the limits I read about in the aforementioned post.

Answer 1

The obvious downsides you have already mentioned, is that there will be a lot of memory required to keep all the file handles open, depending of course on how many files. This is a calculation you have to do on your own. And don't forget the write locks.

Otherwise there isn't very much wrong with it per say, but it would be good with some precaution:

fhs = {}
try:
    for _, node in lxml.etree.iterparse(file):
        if node.tag not in fhs:
            fhs[node.tag] = open(f"{node.tag}.txt", 'w')

        fhs[node.tag].write(node.attrib['someattr']+'\n'])
finally:
    for fh in fhs.values(): fh.close()

Note: When looping over a dict in python, the items you get are really only the keys. I'd recommend doing for key, item in d.items(): or for item in d.values():

Answer 2

You don't didn't say just how many files the process would end up holding open. If it's not so many that it creates a problem, then this could be a good approach. I doubt you can really know without trying it out with your data and in your execution environment.

In my experience, open() is relatively slow, so avoiding unnecessary calls is definitely worth thinking about-- you also avoid setting up all the associated buffers, populating them, flushing them every time you close the file, and garbage-collecting. Since you ask, file pointers do come with large buffers. On OS X, the default buffer size is 8192 bytes (8KB) and there is additional overhead for the object, as with all Python object. So if you have hundreds or thousands of files and little RAM, it can add up. You can specify less buffering or no buffering at all, but that could defeat any efficiency gained from avoiding repeated opens.

Edit: For just 35 distinct files (or any two-digit number), you have nothing to worry about: The space that 35 output buffers will need (at 8 KB per buffer for the actual buffering) will not even be the biggest part of your memory footprint. So just go ahead and do it they way you proposed. You'll see a dramatic speed improvement over opening and closing the file for each xml node.

PS. The default buffer size is given by io.DEFAULT_BUFFER_SIZE.

Answer 3

As a good rule,try to close a file as soon as possible.

Note that also your operating system has limits - you can open only certain number of files. So you might soon hit this limit and you will start getting "Failed to open file" exceptions.

Memory and file handles leaking are obvious problem ( if you fail to close the files for some reason ).

Answer 4

If you are generating thousands of files the way you might considder writing them to a directory structure to get them separately stored in different directories to have easier access afterwards. For example: a/a/aanode.txt , a/c/acnode.txt, etc.

In case the XML contains consecutive nodes you can write while that condition is True. You only close the moment another node for another file appears. What you gain from it largely depends on the structure of your XML file.