Perl: Out of Memory

Question

I receive about 7000 objects from a database and write each item to a JSON file. After about 2000 objects perl terminates with a Out of memory error. In the beginning all database objects are received and obviously they fit perfectly into memory. The error happends during the loop.

I'm quite new to perl and I don't know why this happens. From my point of view, there is no memory allocation in the loop that cannot be free-ed after each iteration.

Here is my code:

use lib '../AcePerl/blib/lib','../AcePerl/blib/arch';
use Ace;
use JSON;
use Data::Structure::Util qw( unbless );
use List::MoreUtils;
use strict vars;

use constant HOST => $ENV{ACEDB_HOST} || 'mining.wormbase.org';
use constant PORT => $ENV{ACEDB_PORT} || 2005;

$|=1;

my $json = JSON->new->allow_nonref;

print "Opening the database....";
my $db = Ace->connect(-host=>HOST,-port=>PORT
              -cache => {
                         cache_root         => '/usr/tmp/acedb',
                         cache_depth        => 4,
                         default_expires_in => '1 week'
                         }
) || die "Connection failure: ",Ace->error;
print "done.\n";

#my @anatomies = $db->list('Anatomy_Term','WBbt:000547*');
my @anatomies = $db->fetch(-name    => '*',
                           -class   => 'Anatomy_Term',
                           -fill    => true,
#                           -count   => 10
                          );

#my @anatomies = $db->list('Anatomy_Term','*');

print scalar(@anatomies), " anatomy terms found!\n";

my $i = 0;
my $c;

sub wbstr {
  my $arg = $_[0];
  if(!defined $arg) {
    return undef;
  }else{
    return $arg->name;}
}

# parse anatomy terms for cells
my %anatomy_data;

open (MYFILE, '>anatomy-data2.txt');
print MYFILE '{';

my $parent;
my $first;
my $ancestor;
my $expr_ancestor;
my $is_a;

foreach $c (@anatomies) {
  if($i!=0) {
    print MYFILE ',';
  }

  $i++;
  print MYFILE $json->encode(wbstr($c)), ':{';

  print MYFILE '"term":', $json->encode(wbstr($c->Term)), ',';
  print MYFILE '"definition":', $json->encode(wbstr($c->Definition)), ',';
  print MYFILE '"synonym":', $json->encode(wbstr($c->Synonym)), ',';

  print MYFILE '"parents":[';
  $first = true;
  foreach $parent ($c->CELL_p) {
    if($first eq false) {
      print MYFILE ',';
    }
    print MYFILE $json->encode(wbstr($parent));
    $first = false;
  }
  print MYFILE '],';

  print MYFILE '"ancestors":[';
  $first = true;
  foreach $ancestor ($c->Ancestor) {
    if($first eq false) {
      print MYFILE ',';
    }
    print MYFILE $json->encode(wbstr($ancestor));
    $first = false;
  }
  print MYFILE '],';


  print MYFILE '"expr_ancestors":[';
  $first = true;
  foreach $expr_ancestor ($c->Expr_ancestor) {
    if($first eq false) {
      print MYFILE ',';
    }
    print MYFILE $json->encode(wbstr($expr_ancestor));
    $first = false;
  }
  print MYFILE '],';

  print MYFILE '"is_a":[';
  $first = true;
  foreach $is_a ($c->IS_A_p) {
    if($first eq false) { 
      print MYFILE ',';
    }
    print MYFILE $json->encode(wbstr($is_a));
    $first = false;
  }
  print MYFILE ']}';
}
print MYFILE '}';

print "Done\n";

close (MYFILE);

You should be running with `use warnings; use strict;`. There could be many many bugs you're hiding. As it is, the places where you're saying `$first = true` should be throwing a warning because that is an archaic equivalent for `$first = "true"`. Add the warnings/strict and see what comes out. It probably won't fix your memory problems but it will turn up bad things you're doing. — Andy Lester, Jan 23 '13 at 15:51
Ok, I better use fetch_many() instead of fetch(). Then I don't run into memory trouble. ;) — user2004469, Jan 23 '13 at 15:56
I wonder if the behavior would change if you `shift` entries out of `@anatomies` after processing them? — aschepler, Jan 23 '13 at 15:57
@user2004469 So is this solved by using fetch_many()? Then please answer yourself. — Perleone, Jan 23 '13 at 21:30

score -1 · Answer 1 · edited May 23 '17 at 10:24

-1

try to flush your file (or insert a "\n" at some point in your print MYFILE statements). This SO question may helps you

edited May 23 '17 at 10:24

Community

1
1

answered Jan 23 '13 at 16:05

Miguel Prz

13,718
29
42

Perl: Out of Memory

1 Answers1