1

I have a mysql database. The table contains the urls that I need to curl and columns that show how often and how many times these urls are called.

My table like that:

URL TIMEOUTMS FREQUENCY HOW MANY TIME ERROR COUNT OF
url1 2000 5 3 3
url2 3000 10 5 5
url3 2000 30 3 3

For example, I have to curl url1 3 times and wait 5 seconds in between. If all three are wrong I should create an alarm.

I wrote the multi curl function, but I got a little confused because of the values I specified in the table

My code is:

<?php
function control_paralel($url_array,$timeout_array){
   $nodes = $url_array;
   $node_count = count($nodes);

   $curl_arr = array();
   $master = curl_multi_init();

    
   for($i = 0; $i < $node_count; $i++)
   {
       $url =$nodes[$i];
       $curl_arr[$i] = curl_init($url);
       curl_setopt($curl_arr[$i], CURLOPT_RETURNTRANSFER, true);
       curl_setopt($curl_arr[$i], CURLOPT_SSL_VERIFYHOST, FALSE);
       curl_setopt($curl_arr[$i], CURLOPT_SSL_VERIFYPEER, FALSE);
       curl_setopt($curl_arr[$i], CURLOPT_CONNECTTIMEOUT_MS, 
       $timeout_array[$i]);
     curl_setopt($curl_arr[$i], CURLOPT_TIMEOUT_MS, $timeout_array[$i]);
    
       curl_multi_add_handle($master, $curl_arr[$i]);
  }

   do {
      curl_multi_exec($master,$running);
   } while($running > 0);


   for($i = 0; $i < $node_count; $i++)
   {
       $results[] = curl_multi_getcontent  ( $curl_arr[$i]  );
       $httpcode[]= curl_getinfo($curl_arr[$i], CURLINFO_HTTP_CODE);
   }
   return $httpcode;
   }


$result_alerts = get_all_alerts();
$url_array = [];
$timeout_array = [];
while($row = $result_alerts->fetch_assoc()) {
    array_push($url_array,$row['URL']);
    array_push($timeout_array,$row['TIMEOUT']);
}

print_r(control_paralel($url_array,$timeout_array));
?>

I thought to group by according to the FREQUENCY and HOW_MANY_TIME columns and send each one to multi_curl in separate groups, but do you think this is the right way? Do you have any other suggestions?

senjizu
  • 103
  • 12
  • What exactly are you stuck on? You are clear on PHP not being asynchronous? Want to make sure what you are trying to accomplish is feasible... – ficuscr Jan 11 '22 at 07:24
  • You know that neither CURLOPT_CONNECTTIMEOUT_MS nor CURLOPT_TIMEOUT_MS make anything "sleep" to begin with, yes? – CBroe Jan 11 '22 at 07:51
  • That multi-exec loop will busy-loop and consume 100% CPU for the duration of the downloads == not a good solution. – Daniel Stenberg Jan 11 '22 at 10:02
  • @CBroe yes I know. I'm thinking of using the sleep command in between. – senjizu Jan 11 '22 at 10:25
  • @DanielStenberg so what do you recommend? Is the python language capable of this? Do you know an example usage? – senjizu Jan 11 '22 at 10:26
  • don't use comments to ask extra questions! The PHP example on https://www.php.net/manual/en/function.curl-multi-exec.php already shows how the loop should be done – Daniel Stenberg Jan 11 '22 at 10:59
  • **[You should not switch off `CURLOPT_SSL_VERIFYHOST` or `CURLOPT_SSL_VERIFYPEER`](https://paragonie.com/blog/2017/10/certainty-automated-cacert-pem-management-for-php-software)**. It could be a security risk! [Here is how to get the certificate bundle if your server is missing one](https://stackoverflow.com/a/32095378/1839439) – Dharman Jan 11 '22 at 12:26

0 Answers0