Memory optimization in PHP array

Question

I'm working with a large array which is a height map, 1024x1024 and of course, i'm stuck with the memory limit. In my test machine i can increase the mem limit to 1gb if i want, but in my tiny VPS with only 256 ram, it's not an option.

I've been searching in stack and google and found several "well, you are using PHP not because memory efficiency, ditch it and rewrite in c++" and honestly, that's ok and I recognize PHP loves memory.

But, when digging more inside PHP memory management, I did not find what memory consumes every data type. Or if casting to another type of data reduces mem consumption.

The only "optimization" technique i found was to unset variables and arrays, that's it.

Converting the code to c++ using some PHP parsers would solve the problem?

Thanks!

Answer 1

If you want a real indexed array, use SplFixedArray. It uses less memory. Also, PHP 5.3 has a much better garbage collector.

Other than that, well, PHP will use more memory than a more carefully written C/C++ equivalent.

Memory Usage for 1024x1024 integer array:

• Standard array: 218,756,848
• SplFixedArray: 92,914,208

as measured by memory_get_peak_usage()

$array = new SplFixedArray(1024 * 1024); // array();
for ($i = 0; $i < 1024 * 1024; ++$i)
  $array[$i] = 0;

echo memory_get_peak_usage();

Note that the same array in C using 64-bit integers would be 8M.

As others have suggested, you could pack the data into a string. This is slower but much more memory efficient. If using 8 bit values it's super easy:

$x = str_repeat(chr(0), 1024*1024);
$x[$i] = chr($v & 0xff); // store value $v into $x[$i]
$v = ord($x[$i]);        // get value $v from $x[$i]

Here the memory will only be about 1.5MB (that is, when considering the entire overhead of PHP with just this integer string array).

For the fun of it, I created a simple benchmark of creating 1024x1024 8-bit integers and then looping through them once. The packed versions all used ArrayAccess so that the user code looked the same.

                   mem    write   read
array              218M   0.589s  0.176s
packed array       32.7M  1.85s   1.13s
packed spl array   13.8M  1.91s   1.18s
packed string      1.72M  1.11s   1.08s

The packed arrays used native 64-bit integers (only packing 7 bytes to avoid dealing with signed data) and the packed string used ord and chr. Obviously implementation details and computer specs will affect things a bit, but I would expect you to get similar results.

So while the array was 6x faster it also used 125x the memory as the next best alternative: packed strings. Obviously the speed is irrelevant if you are running out of memory. (When I used packed strings directly without an ArrayAccess class they were only 3x slower than native arrays.)

In short, to summarize, I would use something other than pure PHP to process this data if speed is of any concern.

Answer 2

In addition to the accepted answer and suggestions in the comments, I'd like to suggest PHP Judy array implementation.

Quick tests showed interesting results. An array with 1 million entries using regular PHP array data structure takes ~200 MB. SplFixedArray uses around 90 megabytes. Judy uses 8 megs. Tradeoff is in performance, Judy takes about double the time of regular php array implementation.

Answer 3

A little bit late to the party, but if you have a multidimensional array you can save a lot of RAM when you store the complete array as json.

$array = [];

$data = [];
$data["a"] = "hello";
$data["b"] = "world";

To store this array just use:

$array[] = json_encode($data);

instead of

$array[] = $data;

If you want to get the arrry back, just use something like:

$myData = json_decode($array[0], true);

I had a big array with 275.000 sets and saved about 36% RAM consumption.

EDIT: I found a more better way, when you zip the json string:

$array[] = gzencode(json_encode($data));

and unzip it when you need it:

$myData = json_decode(gzdecode($array[0], true));

This saved me nearly 75% of RAM peak usage.