2

The Problem

Lets say I have this function:

function hog($i = 1) // uses $i * 0.5 MiB, returns $i * 0.25 MiB
{
    $s = str_repeat('a', $i * 1024 * 512); return substr($s, $i * 1024 * 256);
}

I would like to call it and be able to inspect the maximum amount of memory it uses.

In other words: memory_get_function_peak_usage($callback);. Is this possible?

What I Have Tried

I'm using the following values as my non-monotonically increasing $i argument for hog():

$iterations = array_merge(range(0, 50, 10), range(50, 0, 5));
$iterations = array_fill_keys($iterations, 0);

Which is essentially:

(
    [0] => 0
    [10] => 0
    [20] => 0
    [30] => 0
    [40] => 0
    [50] => 0
    [45] => 0
    [35] => 0
    [25] => 0
    [15] => 0
    [5] => 0
)

Enclosing with memory_get_usage()

foreach ($iterations as $key => $value)
{
    $alpha = memory_get_usage(); hog($key);
    $iterations[$key] = memory_get_usage() - $alpha;
}

print_r($iterations);

Output:

(
    [0] => 96
    [10] => 0
    [20] => 0
    [30] => 0
    [40] => 0
    [50] => 0
    [45] => 0
    [35] => 0
    [25] => 0
    [15] => 0
    [5] => 0
)

If I store the return value of hog(), the results start to look more realistic:

foreach ($iterations as $key => $value)
{
    $alpha = memory_get_usage(); $s = hog($key);
    $iterations[$key] = memory_get_usage() - $alpha; unset($s);
}

print_r($iterations);

Output:

(
    [0] => 176
    [10] => 2621536
    [20] => 5242976
    [30] => 7864416
    [40] => 10485856
    [50] => 13107296
    [45] => 11796576
    [35] => 9175136
    [25] => 6553696
    [15] => 3932256
    [5] => 1310816
)

As expected, now it's showing me the amount of memory returned, but I need the total memory used.

Using register_tick_function():

I didn't knew, but it turns out that when you do:

declare (ticks=1)
{
    $a = hog(1);
}

It won't tick for every line, statement or block of code inside of hog() function, only for the code inside the declare block - so, unless the function is defined within it, this option is a no go.

Mixing with gc_* functions:

I tried (without much hope I must say) using combinations of gc_disable(), gc_enable() and gc_collect_cycles() with both experiments above to see if anything changed - it didn't.

Community
  • 1
  • 1
Alix Axel
  • 151,645
  • 95
  • 393
  • 500
  • 1
    No clue why someone did a downvote.. Hate it when people downvote without a comment. I'll just upvote for the heck of it... – Damien Overeem May 12 '13 at 20:19
  • @damienovereem: I'm pretty pessimistic about finding a solution, but I don't know... Hence the question. I guess the downvote is because this apparently has no solution. – Alix Axel May 12 '13 at 20:25
  • Some people @ stack are a bit... odd.. Downvoting without giving a reason hardly helps anyone. People should take that into consideration.. Oh well.. can't have it all right :) – Damien Overeem May 12 '13 at 20:29
  • @OneTrickPony: Outside I can, inside I can't (imagine I want to profile the memory usage of ... `preg_match_all()` - couldn't do it). – Alix Axel May 12 '13 at 20:49
  • memory_get_peak_usage() ? (If PHP>=5.2.0) http://www.php.net/manual/en/function.memory-get-peak-usage.php – bestprogrammerintheworld May 13 '13 at 17:03
  • @bestprogrammerintheworld: That's global for the whole script, once it reaches the maximum you won't be able to figure out the remaining blocks. – Alix Axel May 13 '13 at 17:05
  • @AlixAxel - Aha ok. I will try to supply a solution within a few days. I don't have the time right now, but it's seems like a very challenging and fun issue to deal with :-) – bestprogrammerintheworld May 14 '13 at 10:48
  • 1
    Maybe this could be something? https://github.com/kampaw/profiler – bestprogrammerintheworld May 14 '13 at 10:52
  • @bestprogrammerintheworld: That looks very similar to what I was doing, except that I was wrapping my code in the `declare(1) {...}` block and the `hog()` function definition was outside of that block. Maybe that's the reason why it didn't work. Anyway, I'll try this later on when I have a change, meanwhile post that as an answer so I can upvote it. =) The project is certainly nice. – Alix Axel May 14 '13 at 11:29
  • @bestprogrammerintheworld: I checked the code and the example of that project, I suspect it *seems* to trap the **output** consumption of `array_push` and `array_pop` because he's actually storing it an array. I would like it to *tap* into the function inner/private execution and return the memory peak of the function itself. – Alix Axel May 14 '13 at 11:36
  • @bestprogrammerintheworld: Basically, I'm writing a profiler that during `s` seconds calls function / method `a`, `b`, `c` the same (variable) number of times (cycling through the CPU time to be more consistent) and gathers their absolute (and relative) execution times, for example `profile('crc32|md5|sha1', $seconds = 15, 'variadic args here');`. This is a convenient way to quickly pick one over the other. But, imagine that one callback is 5% faster than the other, however it eats 300% more RAM. Can you see why tapping into the functions usage would be useful / necessary in this case? – Alix Axel May 14 '13 at 11:43
  • 1
    I guess your best bet would be a trial and error kind of way. Let a seperate script run the function and just before you call the function flood the memory with `str_repeat` to leave eg just 100Kb left. Then call this script with curl or something and if it returns status 200 you run it again with just 50kb left. If its a status 500 you get a message like '.. trying to allocate xxxx bytes' and then run the script again with the 100Kb - xxxx bytes. Obvisouly this will only work when you have results you can cache , have little variation in the input and dont require the exact number of bytes. – Hugo Delsing May 14 '13 at 12:59
  • @HugoDelsing: That's very clever idea! However, it's very unpractical and time consuming. =\ – Alix Axel May 14 '13 at 13:33
  • 1
    Yeah it wouldnt be the most dynamic solution. Probably easier to change some of the PHP source files and run the profiler on a seperate custom PHP built. – Hugo Delsing May 14 '13 at 13:53

3 Answers3

1

I was digging in the PHP manual and I found the memtrack extension, not perfect but it's something.

EDIT: I had heard about it, but never actually tried it before. Turns out XHProf is all I needed:

$flags = array
(
    XHPROF_FLAGS_CPU,
    XHPROF_FLAGS_MEMORY,
    XHPROF_FLAGS_NO_BUILTINS,
);

$options = array
(
    'ignored_functions' => array
    (
        'call_user_func',
        'call_user_func_array',
        'xhprof_disable',
    ),
);

function hog($i = 1) // uses $i * 0.5 MiB, returns $i * 0.25 MiB
{
    $s = str_repeat('a', $i * 1024 * 512); return substr($s, $i * 1024 * 256);
}

Test #1:

xhprof_enable(array_sum($flags), $options);

hog(4);

$profile = xhprof_disable();

print_r($profile);

Output:

    [main()==>hog] => Array
        (
            [ct] => 1
            [wt] => 54784
            [mu] => 384
            [pmu] => 3142356
        )

    [main()] => Array
        (
            [ct] => 1
            [wt] => 55075
            [mu] => 832
            [pmu] => 3142356
        )

mu is memory usage, pmu is peak memory usage, 3142356 / 1024 / 1024 / 0.5 = 4 = $i.

Test #2 (without XHPROF_FLAGS_NO_BUILTINS):

    [hog==>str_repeat] => Array
        (
            [ct] => 1
            [wt] => 21890
            [cpu] => 4000
            [mu] => 2097612
            [pmu] => 2094200
        )

    [hog==>substr] => Array
        (
            [ct] => 1
            [wt] => 17202
            [cpu] => 4000
            [mu] => 1048992
            [pmu] => 1048932
        )

    [main()==>hog] => Array
        (
            [ct] => 1
            [wt] => 45978
            [cpu] => 8000
            [mu] => 1588
            [pmu] => 3143448
        )

    [main()] => Array
        (
            [ct] => 1
            [wt] => 46284
            [cpu] => 8000
            [mu] => 2132
            [pmu] => 3143448
        )

Whoohoo! Thanks Facebook!

From the XHProf docs:

It is worth clarifying that that XHProf doesn't strictly track each allocation/free operation. Rather it uses a more simplistic scheme. It tracks the increase/decrease in the amount of memory allocated to PHP between each function's entry and exit. It also tracks increase/decrease in the amount of peak memory allocated to PHP for each function.

Alix Axel
  • 151,645
  • 95
  • 393
  • 500
0

That has everything to do with the variable scope. Everything inside the function will be cleared once the function ends. So if you need to know how much memory is used in total you need to declare them outside the function scope. My vote would go to a single variable array for ease incase you need multiple vars. If you need just one you obviously dont need the array.

<?php
$outervar = array();

function hog($i = 1) // uses $i * 0.5 MiB, returns $i * 0.25 MiB 
{ 
  global $outervar;
  $outervar['s'] = str_repeat('a', $i * 1024 * 512); return substr($outervar['s'], $i * 1024 * 256); 
}

foreach ($iterations as $key => $value) 
{ 
  $alpha = memory_get_usage(); 
  hog($key); 
  $iterations[$key] = memory_get_usage() - $alpha;
  $outervar = array(); 
 }

print_r($iterations);

Since we dont store the result of hog this will use 0.5mb*$i. If you also need the return value even if its not stored, first save it to $outervar['result'] or something and then return it. But then it will be counted double if you do save it.

A second option would be to gave a second parameter by reference &$memusage and use the memory_get_usage() part inside the function and store the result in the byref variable

Hugo Delsing
  • 13,803
  • 5
  • 45
  • 72
  • 1
    I still have to place `$outervar['s'] = ...` on every logical block to get the function peak usage, which is highly unpractical. Plus, if `hog()` uses any other function like `preg_replace()` on a big array, I still won't be able to account for the `preg_replace()` memory usage. – Alix Axel May 12 '13 at 20:55
0

I found this: github.com/kampaw/profiler that seem to use the "tick/register/decare-concept", that wasn't an option for you. I've also read that the register_tick_functionality is going to removed in PHP 6. (But that might just be a rumour)

I understand totally what you mean by checking memory INSIDE of the function.

I've tested your code based on just calling the function and then returning memory usage afterward using that function. I made a checkMemoryFunction() just make it more general (of course checkMemoryFunction would take a bit of memory to, but that might be possible to subtract if necessary). I think you were thinking right about getting used memory, BUT I found another very weird thing...

...with this code:

<?php
function hog($i) 
{
    $s = str_repeat('a', $i * 1024 * 512); 
    return substr($s, $i * 1024 * 256);
}

function checkMemoryFunction($functionName, $i) {
    $startMemory = memory_get_usage();
    call_user_func($functionName, $i);
    return memory_get_usage() - $startMemory;
}

$iterations = array_merge(range(0, 50, 10), range(50, 0, 5));
$iterations = array_fill_keys($iterations, 0);

foreach ($iterations as $key => $value)
{
    $mem = checkMemoryFunction('hog', $key);
    $iterations[$key] = $mem;
}

$peak = max($iterations);
echo '<hr />Iteratins array:';
print_r($iterations);
echo 'memory peak=' . $peak;
?>

I got about same results as you did: (First element set, but not the rest)

Output Iterations array:

Array ( [0] => 312 [10] => 0 [20] => 0 [30] => 0 [40] => 0 [50] => 0 [45] => 0 [35] => 0 [25] => 0 [15] => 0 [5] => 0 ) memory peak=312

However, when I add rows to set each key-value to 0 (or whatever value)...

$iterations = array_merge(range(0, 50, 10), range(50, 0, 5));
$iterations = array_fill_keys($iterations, 0);

// set each value to 0 in array
foreach ($iterations as $key => &$value)
{
    $value = 0;
}

foreach ($iterations as $key => $value)
{
    $mem = checkMemoryFunction('hog', $key);
    $iterations[$key] = $mem;
}

...I get these values (some memory usage for all function calls): 

Array ( [0] => 312 [10] => 24 [20] => 24 [30] => 24 [40] => 24 [50] => 24 [45] => 24 [35] => 24 [25] => 24 [15] => 24 [5] => 24 ) memory peak=312

So it seems like the problem lied within the call to array_fill_keys(). (It seems like the array-elements aren't initilized in some weird way)

Taking a close look at $iterations-array directly in your code after merging the arrays, it look like this: (duplicate values)

Array ( [0] => 0 [1] => 10 [2] => 20 [3] => 30 [4] => 40 [5] => 50 [6] => 50 [7] => 45 [8] => 40 [9] => 35 [10] => 30 [11] => 25 [12] => 20 [13] => 15 [14] => 10 [15] => 5 [16] => 0` )

but I think what you really wanted was something like this:

Array ( [0] => 0 [1] => 10 [2] => 20 [3] => 30 [4] => 40 [5] => 50 [6] => 45 [8] => 35 [10] => 25 [12] => 15 [14] => 5)

My suspicion was that the duplicates made array_fill_keys() act in strange manner, so I tried:

$iterations = array(0, 10, 20, 30, 40, 50, 45, 35, 25, 15);
$iterations = array_fill_keys($iterations, 0);
foreach ($iterations as $key => $value)
{
        $mem = checkMemoryFunction('hog', $key);
        $iterations[$key] = $mem;
}

But it still didn't work as expected:

Array ( [0] => 312 [10] => 0 [20] => 0 [30] => 0 [40] => 0 [50] => 0 [45] => 0 [35] => 0 [25] => 0 [15] => 0 ) memory peak=312

When I add 

foreach ($iterations as $key => &$value)
{
    $value = 0;
}

again it workds like expected:

Array ( [0] => 312 [10] => 0 [20] => 24 [30] => 24 [40] => 24 [50] => 24 [45] => 32 [35] => 48 [25] => 24 [15] => 24 [5] => 24 ) memory peak=312

I think it's strange because array_fill_keys($iterations, 0); should do the same thing as the foreach above. I can't figure out WHY it doesn't work as expected. Maybe it's a bug, but probably it's something "stupid" that I haven't thought of.

Another approach could be like storing "content inside a function" from theh PHP-source-file and then save it as profile/hog.php and after that execute the hog.php code.

I hope this could help you out a bit! :-)

bestprogrammerintheworld
  • 5,417
  • 7
  • 43
  • 72