Why opening and iterating over file handle over twice as fast in Python 2 vs Python 3?

Question

I can't work out why it's so much faster to parse this file in Python 2.7 than in Python 3.6. I've found this pattern both on macOS and Arch-Linux independently. Can others replicate it? Any explanation?

Warning: the code snippet writes a ~2GB file

Timings:

$ python2 test.py 
5.01580309868
$ python3 test.py 
10.664075019994925

Code for test.py:

import os

SEQ_LINE = 'ATCGN'* 80 + '\n'

if not os.path.isfile('many_medium.fa'):
    with open('many_medium.fa', 'w') as out_f:
        for i in range(1000000):
            out_f.write('>{}\n'.format(i))
            for _ in range(5):
                out_f.write(SEQ_LINE)

from timeit import timeit

def f():
    with open('many_medium.fa') as f:
        for line in f:
            pass

print(timeit('f()', setup='from __main__ import f', number=5))

Answer 1

Did some research and came across this article by Nelson Minar that explains what the difference is between python2 and python3 file reading.

• Python 3 is ~1.7x little slower reading bytes line by line than Python 2

• In Python 2, reading lines with Unicode is hella slow. About 7x slower than reading Unicode all at once. And Unicode lines are 70x slower than byte lines!

• In Python 3, reading lines with Unicode is quite fast. About as fast as reading the file all at once. But only if you use the built-in open, not codecs.

• In Python 3, codecs is really slow for reading line by line. Avoid.

And continues to say:

Python 3 UTF-8 decoding is significantly faster than Python 2. And it’s probably best to stick with the stock open() call in Py3, not codecs. It may be slower in some circumstances but it’s the recommended option going further and the difference isn’t enormous.

According to the SO answer that @user2357112 linked:

When you open a file in Python in text mode (the default), it uses what it calls "universal newlines" (introduced with PEP 278, but somewhat changed later with the release of Python 3). What universal newlines means is that regardless of what kind of newline characters are used in the file, you'll see only \n in Python. So a file containing foo\nbar would appear the same as a file containing foo\r\nbar or foo\rbar (since \n, \r\n and \r are all line ending conventions used on some operating systems at some time).

The solution mentioned in this answer is to open the file in byte mode to avoid the conversion:

open('many_medium.fa', "r+b")

My tests indicated a massive difference in the speed, but python2 still seemed to be slightly faster. There does not seem to be a way to avoid this, as it's handled by python's interpreter.

Answer 2

Because in Python 2, the standard open() call creates a far simpler file object than the Python 3 open() call does. The Python 3 open call is the same thing as io.open(), and the same framework is available on Python 2.

To make this a fair comparison, you'd have to add the following line to the top of your test:

from io import open

With that change, the timings on Python 2 go from 5.5 seconds, to 37 seconds. Compared to that figure, the 11 seconds Python 3 takes on my system to run the test really is much, much faster.

So what is happening here? The io library offers much more functionality than the old Python 2 file object:

• File objects returned by open() consist of up to 3 layers of composed functionality, allowing you to control buffering and text handling.
• Support for non-blocking I/O streams
• A consistent interface across a wide range of streams
• Much more control over the universal newline translation feature.
• Full Unicode support.

That extra functionality comes at a performance price.

But your Python 2 test reads byte strings, newlines are always translated to \n, and the file object the code is working with is pretty close to the OS-supplied file primitive, with all the downsides. In Python 3, you usually want to process data from files as text, so opening a file in text mode gives you a file object that decodes the binary data to Unicode str objects.

So how can you make things go 'faster' on Python 3? That depends on your specific use case, but you have some options:

• For text-mode files, disable universal newline handling, especially when handling a file that uses line endings that differ from the platform standard. Set the newline parameter to the expected newline character sequence, like \n. Binary mode only supports \n as line separator.
• Process the file as binary data, and don't decode to str. Alternatively, decode to Latin-1, a straight one-on-one mapping from byte to codepoint. This is an option when your data is ASCII-only too, where Latin-1 omits an error check on the bytes being in the range 0-127 rather than 0-255.

When using mode='rb', Python 3 can easily match the Python 2 timings, the test only takes 5.05 seconds on my system, using Python 3.7.

Using latin-1 as the codec vs. UTF-8 (the usual default) makes only a small difference; UTF-8 can be decoded very efficiently. But it could make a difference for other codecs. You generally want to set the encoding parameter explicitly, and not rely on the default encoding used.