4

I have a Mac server (snow leopard) with 32GB RAM. When I try to allocate more than 1.1GB RAM in Perl (v 5.10.0) I get an out of memory error. Here is the script that I used: 

#!/usr/bin/env perl

# My snow leopard MAC server runs out of memory at >1.1 billion bytes.  How
# can this be when I have 32 GB of memory installed?  Allocating up to 4
# billion bytes works on a Dell Win7 12GB RAM machine.

# perl -v
# This is perl, v5.10.0 built for darwin-thread-multi-2level
# (with 2 registered patches, see perl -V for more detail)

use strict;
use warnings;

my $s;
print "Trying 1.1 GB...";
$s = "a" x 1100000000;   # ok
print "OK\n\n";

print "Trying 1.2 GB...";
$s = '';
$s = "a" x 1200000000;   # fails
print "..OK\n";

Here is the output that I get:

Trying 1.1 GB...OK

perl(96685) malloc: *** mmap(size=1200001024) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Out of memory!
Trying 1.2 GB...

Any ideas why this is happening?

UPDATE 4:42pm 11/14/13

As per Kent Fredric (see 2 posts below), here are my ulimits. Virtual memory defaults to unlimited

$ ulimit -a | grep bytes
data seg size           (kbytes, -d) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
virtual memory          (kbytes, -v) unlimited

$ perl -E 'my $x = "a" x 1200000000; print "ok\n"'
perl(23074) malloc: *** mmap(size=1200001024) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Out of memory!

$ perl -E 'my $x = "a" x 1100000000; print "ok\n"'
ok

I tried setting virtual memory to 10 billion but to no avail.

$ ulimit -v 10000000000   # 10 billion

$ perl -E 'my $x = "a" x 1200000000; print "ok\n"'
perl(24275) malloc: *** mmap(size=1200001024) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Out of memory!
ROMANIA_engineer
  • 54,432
  • 29
  • 203
  • 199
pcantalupo
  • 2,212
  • 17
  • 27

4 Answers4

6

You're using a 32-bit build of Perl (as shown by perl -V:ptrsize), but you need a 64-bit build. I recommend installing a local perl using perlbrew.

This can be achieved by passing -Duse64bitall to Configure when installing Perl.

This can be achieved by passing --64all to perlbrew install when installing Perl.

(For some odd reason, perl -V:use64bitall says this was done, but it clearly wasn't.)

ikegami
  • 367,544
  • 15
  • 269
  • 518
  • How do you know I have a 32-bit perl from my perl -V output (pastebin.com/Bd3VLdpM)? I thought that -arch x86_64 is 64-bit. Thank you – pcantalupo Nov 14 '13 at 17:01
  • 2
    hum, I see `use64bitall=define`, but you should also have `USE_64_BIT_ALL` under Compile-time options. Something's weird about your build. My advice stands. – ikegami Nov 14 '13 at 17:04
  • This is the Mac factory build. I guess I would have to try perlbrew to see if it works. I hope others with similar Mac server Perl configurations will chime in. Thanks again – pcantalupo Nov 14 '13 at 17:09
  • 2
    @Virushunter The config is indeed very weird. This may be some form of cross-platform build: `-arch x86_64 -arch i386 -arch ppc`. Anyway, `intsize=4, longsize=4, ptrsize=4` is a certain giveaway that this was compiled for a 32-bit system. Those are the sizes of the C types, not the Perl data structures, so you can't address more than 4GB. (This still doesn't fully explain the 1,1GB cutoff) – amon Nov 14 '13 at 17:38
  • Indeed, 1.2G is far far short of the 4G cut-off. So 64-bit doesn't seem like a direct cause. – Kent Fredric Nov 14 '13 at 18:38
  • 32-bit processes don't have 4GB. I believe they have 2GB on Windows. No idea about Darwin. Keep in mind the OP's code has a 1.12 GB scalar on the stack which he's trying to assign to a scalar with a 1.02GB buffer. (@amon, @Kent Fredric) – ikegami Nov 14 '13 at 19:15
  • That really shouldn't matter, as long as perl GC's before the second assignment. However, if its not GCing before the assginment, then you're temporarily using 2.3G of memory, which may be close to hitting the limit imposed on 32bit. Just as you see in my other answer, OSs may apply global limits on process memory, which might mean 32bit/64bit is not even an issue. – Kent Fredric Nov 14 '13 at 19:37
  • Note: He does assign an empty string to $s before assigning the second value, so as long as compile-time constant folding isn't happening, the first assignment should be GC'd before the second. However, he's pasted output showing that it executes the first half of the code, which means constant folding isn't happening on the `x` operator. ... Unless he's lied about his output ;) – Kent Fredric Nov 14 '13 at 19:39
  • @Kent Fredric, Re: "He does assign an empty string to $s before assigning the second value", That has no effect whatsoever on memory. Use `undef $s;` (as opposed to `$s = undef;` or `$s = '';`) to free the buffer. – ikegami Nov 14 '13 at 20:24
  • @Kent Fredric, Re: "so as long as compile-time constant folding isn't happening", I thought it did, but I checked and it doesn't. `x` is presumably exempt. – ikegami Nov 14 '13 at 20:24
  • I commented to that effect, because for me, `"a" x some_big_number` was causing compile-time memory overflows. Granted, I'm testing on 5.19 betas, where this may be a new optimisation. But I literally had to re-write my test in terms of `"a" x $var` to delay the overflow to runtime. – Kent Fredric Nov 14 '13 at 23:32
  • Maybe this helps explain what I see on 5.19: You'll see here, it OOM's before *any* printing occurs https://gist.github.com/kentfredric/7476401 , however, if I adjust it as seen here: https://gist.github.com/kentfredric/7476454 , then it manages to print something before it OOMs, because the former attracts constant-folding at compile time, while the latter does not. – Kent Fredric Nov 14 '13 at 23:42
  • Oh, he's updated the top post, you'll see that even on its own, the 1_200_000_000 target OOMs :/ – Kent Fredric Nov 14 '13 at 23:51
  • Ah, I THOUGHT there was a serious regression on older perls. Turns out, memusage for the 1.1G test consumes *2* G of ram! So 5.10.0 is only half as efficient as 5.19! – Kent Fredric Nov 15 '13 at 00:16
  • According to this, the OSX Kernel itself is only 32bit .... I'm not sure what that means, but it scares me. http://stackoverflow.com/questions/5486476/how-do-we-create-a-32-bit-build-of-perl-5-12-3-in-usr-local-on-mac-os-x-snow-le – Kent Fredric Nov 15 '13 at 02:13
  • Though heres a similar problem with java, which is blamed on 32bit java http://stackoverflow.com/questions/13782840/unable-to-allocate-memory-error-on-mac-os-x-javaxx-xx-malloc-mmapsize – Kent Fredric Nov 15 '13 at 02:22
3

Seems like this could be related to the problem. This only really is worthy of a comment, but its too complex to put as one without it being entirely illegible

perlbrew exec --with=5.10.0 memusage perl -e '$x = q[a] x 1_000_000_000; print length($x)'
5.10.0
==========
1000000000
Memory usage summary: heap total: 2000150514, heap peak: 2000141265, stack peak: 4896

Yes, thats 2 G of memory for 1 G of text.

Now with 2G ... 

perlbrew exec --with=5.10.0 memusage perl -e '$x = q[a] x 1_000_000_000; $y = q[a] x 1_000_000_000; print length($x)+length($y)'
5.10.0
==========
2000000000
Memory usage summary: heap total: 4000151605, heap peak: 4000142092, stack peak: 4896

Yikes. That would certainly hit the 32Bit limit if you had one.

I was spoiled and doing my testing on 5.19.5, which has a notable improvement, namedly copy-on-write strings, which greatly reduces memory consumption:

perlbrew exec --with=5.19.5 memusage perl -e '$x = q[a] x 1_000_000_000; $y = q[a] x 1_000_000_000; print length($x)+length($y)'
5.19.5
==========
2000000000
Memory usage summary: heap total: 2000157713, heap peak: 2000150396, stack peak: 5392

So either way, if you're using any version of Perl at all other than a development one, you need to expect it to eat twice the memory you need.

If there's a memory limit for some reason around the 2G window for 32bit processes, then you will hit that with a 1G string.

Why does Copy On Write matter?

Well, when you do

$a = $b

$a is a copy of $b

So when you do

$a = "a" x 1_000_000_000

First, it expands the right hand side, creating a variable, and then makes a copy to store in $a.

You can prove this by eliminating the copy as follows:

perlbrew exec --with=5.10.0 memusage perl -e 'print length(q[a] x 1_000_000_000)'
5.10.0
==========
1000000000
Memory usage summary: heap total: 1000150047, heap peak: 1000140886, stack peak: 4896

See, all I did was removed the intermediate variable, and the memory usage halved!

:S

Though because 5.19.5 only makes references to the original string, and copies it when written to, it is efficient by default, so removal of the intermediate variable has negligible benefits

perlbrew exec --with=5.19.5 memusage perl -e 'print length(q[a] x 1_000_000_000)'
5.19.5
==========
1000000000
Memory usage summary: heap total: 1000154123, heap peak: 1000145146, stack peak: 5392
Kent Fredric
  • 56,416
  • 14
  • 107
  • 150
1

It could also be a Mac imposed limitation on per-process memory to prevent processes consuming too much system memory.

I don't know how valid this will be, but I assume Mac, being a Unix, has unix-like ulimits:

There are a few such memory limits, some excerpts from /etc/security/limits.conf

- core - limits the core file size (KB)
- data - max data size (KB)
- fsize - maximum filesize (KB)
- memlock - max locked-in-memory address space (KB)
- rss - max resident set size (KB)
- stack - max stack size (KB)
- as - address space limit (KB)

bash provides ways to limit and read these (somewhat), info bash --index-search=ulimit

For instance, ulimit -a | grep bytes emits this on my Linux machine:

data seg size           (kbytes, -d) unlimited
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 8192
virtual memory          (kbytes, -v) unlimited

And I can arbitrarily limit this within a scope:

$ perl -E 'my $x = "a" x 100000000;print "ok\n"'
ok
$ ulimit -v 200000
$ perl -E 'my $x = "a" x 100000000;print "ok\n"'
Out of memory!
panic: fold_constants JMPENV_PUSH returned 2 at -e line 1.

So ulimits are certainly something to look into.

Kent Fredric
  • 56,416
  • 14
  • 107
  • 150
  • ( Also, the reason its failing in fold_constants for me, is because "a" x CONSTANT_NUMBER will get compile-time translation, so having that in code anywhere will make the previous code also fail ;) ) – Kent Fredric Nov 14 '13 at 17:51
1

I think I figured it out. I could not accept that Apple shipped a 32-bit Perl when their documentation says otherwise. From 'man perl':

64-BIT SUPPORT
Version 5.10.0 supports 64-bit execution (which is on by default).  Version 5.8.8
only supports 32-bit execution.

Then I remembered, I installed Fink on my Mac server and it being, er, finicky with 32 and 64 bit issues. So, I commented out

#test -r /sw/bin/init.sh && . /sw/bin/init.sh

from my .profile. Now I can at least allocate 14 GB RAM (yeah!) on my 32 GB RAM server

$ perl -E 'my $x = "a" x 14000000000; print "ok\n"'
ok

I tried 16GB but it hung for 5 minutes before I gave up. Now, a diff between perl -V for 32-bit and 64-bit tells the tale (but why still intsize=4?).

$ diff perlv.32 perlv.64
16c16
<     intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
---
>     intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
18c18
<     ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
---
>     ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
34,35c34,36
<                         PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP USE_ITHREADS
<                         USE_LARGE_FILES USE_PERLIO USE_REENTRANT_API
---
>                         PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP USE_64_BIT_ALL
>                         USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES
>                         USE_PERLIO USE_REENTRANT_API

Thank you all for you help,

Paul

pcantalupo
  • 2,212
  • 17
  • 27
  • Because ints are always 32 bits ;). But IV, which contains perl representations of integers, is 8 instead of 4, which is what matters here. ptrsize is also 8, which increases memory addressing capacity from the 4G limit to the 16 PB limit. Just if you have code that assumes ptr's are ints, that code is wrong =) – Kent Fredric Nov 15 '13 at 13:51
  • Though I'm glad you found a good answer that solves your specific problem, I'm just still boggled why the root problem cuts off around 2G, that in itself seems to be unexplainable so far. Lots of hand-waving that its OS enforced, but nothing documented anywhere. – Kent Fredric Nov 15 '13 at 13:54