OverflowError: size does not fit in an int while parsing big XML with DOM

Question

I have a pretty big XML file and I need to get all the nodes (different companies information) that contain a specific parameter. XML is about 12 GB unpacked.

    <Companies  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ...>

 <Company id="782634892" source="abcd">
   <attribution>abcde</attribution>
   <name xml:lang="en">company name</name>
   <Phones>
     <Phone type="phone" hide="0">
       <formatted>+1800111</formatted>
       <country>1</country>
       <prefix>800</prefix>
       <number>111</number>
     </Phone>
   </Phones>
   <Rubrics>
     <rubric ref="184107947"/>
   </Rubrics>

There is a bunch of stuff more but that doesn't matter.

My code is pretty simple:

file = open('companies2.xml')
data = file.read()
dom = parseString(data)
key = dom.getElementsByTagName("Company")
for elements in key:
    rubricsArray =  elements.getElementsByTagName("Rubrics")[0].getElementsByTagName("rubric")
    for rub in rubricsArray:
        if rub.attributes["ref"].value == '32432793389':
            print elements.toxml()

It works on a smaller file I made for testing. But here it doesn't.

   Traceback (most recent call last):
  File "./XMLparse.py", line 29, in <module>
    dom = parseString(data)
  File "/usr/lib/python2.7/xml/dom/minidom.py", line 1930, in parseString
    return expatbuilder.parseString(string)
  File "/usr/lib/python2.7/xml/dom/expatbuilder.py", line 940, in parseString
    return builder.parseString(string)
  File "/usr/lib/python2.7/xml/dom/expatbuilder.py", line 223, in parseString
    parser.Parse(string, True)
OverflowError: size does not fit in an int

Any ideas how to make it work? I tried to use gz file but zmore creates some random first line:

------> companies2.xml.gz <------

And DOM won't parse it. So i gunzipped it. Thanks in advance for any help.

As file sizes grow above 2Gb, we're going to see more and more cases where software can't cope because it uses 32-bit offsets. Most of our programming environments are constrained at the language level to use 32-bit int's as offsets into arrays, strings, etc. No easy answers. — Michael Kay, Jul 16 '14 at 21:33
@MichaelKay: I agree. However, I have had some quite pleasant surprises lately. Many old tools do not support more than 32-bit indexing, but equally many tools have been updated. Most of the time you can manage by identifying the problem and then finding the updated tool. — DrV, Jul 18 '14 at 10:11

score 0 · Answer 1 · edited May 23 '17 at 11:58

0

The error message tells it. Something somewhere uses a 32-bit int to store the size of the file or the position in the file.

First, please check that you are running a 64-bit Python. A 32-bit Python very likely chokes if you feed it with a 12 GiB data file if you intend to keep the contents in RAM.

Second, you might want to try another parser. The easiest to try is xml.etree.cElementTree (of course you could try the non-C version, but that'll be too slow). If it chokes, then try lxml and make your code use iterparse. cElementTree is a part of the standard distribution, lxml has to be installed separately.

You may get some ideas by looking at this question and the answers: using lxml and iterparse() to parse a big (+- 1Gb) XML file

edited May 23 '17 at 11:58

Community

1
1

answered Jul 16 '14 at 21:02

DrV

22,637
7
60
72

I checked the python, it was ok, working in 64 bits and looooong integer :) Thanks for the advices. elementTree couldn't handle my file either, it took him 10 minutes to swallow all of the 49 GB of RAM and the process was killed. lxml wasn't installed on the server and I didn't want to ask for it. xml.sax did the trick though, I could parse my file line-by-line and extract the needed info. elementTree.iterparser could do the job, sax just had a better manual and examples. – antonavy Jul 21 '14 at 16:39

OverflowError: size does not fit in an int while parsing big XML with DOM

1 Answers1

Linked