4

I am trying parse a large xml file (around 100,000 records) using XML::Twig but perl parsing fails with error:

perl.exe - Application Error: 
The instruction at "0x28086920" referenced memory at "0x00000004". The memory could not be "written"...

I read that XML::Twig parses large xml files without any issues, but in my case it fails with the above error.

My .pl file has a forloop code that rotates 100,000 times as below:

foreach my $d1(@detailData)   {
    if ($d1->first_child('a')->getElementsByTagName('b')) {
        $id= $d1->first_child('a')->first_child('x')->field('b');
    }
    ....
    .....
    ......

}

Inside forloop I have around 20 if loops as above. Is it causing the memory issues?

Can anyone suggest me how to overcome this memory issues?

mirod
  • 15,923
  • 3
  • 45
  • 65
user1019072
  • 299
  • 1
  • 7
  • 17
  • 2
    Can you show how the `XML::Twig` object is being instantiated and spec'd? – Zaid Jan 31 '12 at 06:04
  • 1
    It seems likely that you aren't taking advantage of `XML::Twig`'s methods for limiting memory usage, but like Zaid implies, it's impossible to say without more details. – flesk Jan 31 '12 at 09:00

1 Answers1

6

After googling for perl "The memory could not be written" I would guess that the problem comes either from loading the file all at once in memory, see for example http://www.perlmonks.org/?node_id=457265 or (less likely) from mixing modules compiled with different compilers (eg using Activestate packages with a Cygwin perl, see http://cygwin.com/ml/cygwin/2006-12/msg00798.html).

For XML::Twig to deal with huge files, you need to tell it at which level to process parts of the file (usually by defining handlers that process a sub-tree, then discard it), see the docs of the module.

mirod
  • 15,923
  • 3
  • 45
  • 65