How to reduce virtual memory by optimising my PHP code?

Question

My current code (see below) uses 147MB of virtual memory! My provider has allocated 100MB by default and the process is killed once run, causing an internal error. The code is utilising curl multi and must be able to loop with more than 150 iterations whilst still minimizing the virtual memory. The code below is only set at 150 iterations and still causes the internal server error. At 90 iterations the issue does not occur.

How can I adjust my code to lower the resource use / virtual memory?

Thanks!

<?php

    function udate($format, $utimestamp = null) {
      if ($utimestamp === null)
        $utimestamp = microtime(true);
      $timestamp = floor($utimestamp);
      $milliseconds = round(($utimestamp - $timestamp) * 1000);
      return date(preg_replace('`(?<!\\\\)u`', $milliseconds, $format), $timestamp);
    }

$url = 'https://www.testdomain.com/';
$curl_arr = array();
$master = curl_multi_init();

for($i=0; $i<150; $i++)
{
    $curl_arr[$i] = curl_init();
    curl_setopt($curl_arr[$i], CURLOPT_URL, $url);
    curl_setopt($curl_arr[$i], CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($curl_arr[$i], CURLOPT_SSL_VERIFYHOST, FALSE);
    curl_setopt($curl_arr[$i], CURLOPT_SSL_VERIFYPEER, FALSE);
    curl_multi_add_handle($master, $curl_arr[$i]);
}

do {
    curl_multi_exec($master,$running);
} while($running > 0);

for($i=0; $i<150; $i++)
{
    $results = curl_multi_getcontent ($curl_arr[$i]);
    $results = explode("<br>", $results);
      echo $results[0];
      echo "<br>";
      echo $results[1];
      echo "<br>";
      echo udate('H:i:s:u');
      echo "<br><br>";
      usleep(100000);
}

?>

Answer 1

As per your last comment..

Download RollingCurl.php.

Hopefully this will sufficiently spam the living daylights out of your API.

<?php

$url = '________';
$fetch_count = 150;
$window_size = 5;


require("RollingCurl.php");

function request_callback($response, $info, $request) {
    list($result0, $result1) = explode("<br>", $response);
    echo "{$result0}<br>{$result1}<br>";
    //print_r($info);
    //print_r($request);
    echo "<hr>";
}


$urls = array_fill(0, $fetch_count, $url);

$rc = new RollingCurl("request_callback");
$rc->window_size = $window_size;
foreach ($urls as $url) {
    $request = new RollingCurlRequest($url);
    $rc->add($request);
}
$rc->execute();

?>

---

Looking through your questions, I saw this comment:

If the intention is domain snatching, then using one of the established services is a better option. Your script implementation is hardly as important as the actual connection and latency.

I agree with that comment.

Also, you seem to have posted the "same question" approximately seven hundred times:

https://stackoverflow.com/users/558865/icer
https://stackoverflow.com/users/516277/icer

How can I adjust the server to run my PHP script quicker?
How can I re-code my php script to run as quickly as possible?
How to run cURL once, checking domain availability in a loop? Help fixing code please
Help fixing php/api/curl code please
How to reduce virtual memory by optimising my PHP code?
Overlapping HTTPS requests?
Multiple https requests.. how to?

Doesn't the fact that you have to keep asking the same question over and over tell you that you're doing it wrong?

This comment of yours:

@mario: Cheers. I'm competing against 2 other companies for specific ccTLD's. They are new to the game and they are snapping up those domains in slow time (up to 10 seconds after purge time). I'm just a little slower at the moment.

I'm fairly sure that PHP on a shared hosting account is the wrong tool to use if you are seriously trying to beat two companies at snapping up expired domain names.

Answer 2

The result of each of the 150 queries is being stored in PHP memory and by your evidence this is insufficient. The only conclusion is that you cannot keep 150 queries in memory. You must have a method of streaming to files instead of memory buffers, or simply reduce the number of queries and processing the list of URLs in batches.

To use streams you must set CURLOPT_RETURNTRANSFER to 0 and implement a callback for CURLOPT_WRITEFUNCTION, there is an example in the PHP manual:

http://www.php.net/manual/en/function.curl-setopt.php#98491

function on_curl_write($ch, $data)
{
  global $fh;
  $bytes = fwrite ($fh, $data, strlen($data));
  return $bytes;
}

curl_setopt ($curl_arr[$i], CURLOPT_WRITEFUNCTION, 'on_curl_write');

Getting the correct file handle in the callback is left as problem for the reader to solve.

Answer 3

<?php

echo str_repeat(' ', 1024); //to make flush work

$url = 'http://__________/';
$fetch_count = 15;
$delay = 100000; //0.1 second
//$delay = 1000000; //1 second


$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);


for ($i=0; $i<$fetch_count; $i++) {

    $start = microtime(true);

    $result = curl_exec($ch);

    list($result0, $result1) = explode("<br>", $result);
    echo "{$result0}<br>{$result1}<br>";
    flush();

    $end = microtime(true);

    $sleeping = $delay - ($end - $start);
    echo 'sleeping: ' . ($sleeping / 1000000) . ' seconds<hr />';
    usleep($sleeping);

}

curl_close($ch);

?>