How long should ElementTree iterparse take?

Question

In answering another question, someone showed me the following tutorial, in which the author claims to have used iterparse to parse a ~100 MB XML file in under 3 seconds:

http://eli.thegreenplace.net/2012/03/15/processing-xml-in-python-with-elementtree/

I am trying to parse an ~90 MB XML file, and I have the following code:

from xml.etree.cElementTree import *
count = 0

for event, elem in iterparse('foo.xml'):        
    if elem.tag == 'identifier' and elem.text == 'bar':
        count += 1
    elem.clear() # discard the element

print count

It is taking about thirty seconds... not even the same order of magnitude as reported in the tutorial I read using a similarly sized file, a similar algorithm, and the same package.

Could someone please inform me what might be wrong with my code, or what differences I might not be noticing between my situation and the tutorial?

I am using Python 2.7.3.

Addendum:

I am also using a reasonably powerful machine, in case anyone thinks that might be it.

I don't think there's much value in comparing XML files by size alone. I'm sure complexity of internal structure factors in there. — TJD, Jun 28 '12 at 18:57
How much output is there? Displaying output is surprisingly slow. — Karl Knechtel, Jun 28 '12 at 19:04
Well in my snippet code obviously there is none (and it still takes a long time to run) but in the code I'm actually trying to work with I have disabled all output (specifically to see if that was slowing it down). It isn't. — russell, Jun 28 '12 at 19:07

score 4 · Accepted Answer · answered Jun 28 '12 at 19:38

As TJD mentioned, comparing XMLs in size only may not be very informative. However, I happen to have files of the same structure but different size:

With a 79M file:

$ python -m timeit -n 1 -c 'from xml.etree.cElementTree import iterparse
count = 0
for event, elem in iterparse("..../QT20060217_S_18mix23-2500_01.mzML"):
    if elem.tag.endswith("spectrum"): count += 1
    elem.clear()
print count'
6126
6126
6126
1 loops, best of 3: 950 msec per loop

With a 3.8G file the timeit output is:

1 loops, best of 3: 22.3 sec per loop

Also, compare with lxml: changing xml.etree.cElementTree in the first line to lxml.etree I get:

for the first file: 730 msec per loop

for the second file: 11.4 sec per loop

How long should ElementTree iterparse take?

1 Answers1