PHP Curl requests Timeout / Page stop loading?

Question

I am sending about 600 Curl requests to different websites and at some point my page stop/break and here is the error I am getting.

Website.com unexpectedly closed the connection. ERR_INCOMPLETE_CHUNKED_ENCODING

I am looping the function below through all my 600 websites.

function GetCash($providerUrl, $providerKey){

$url = check_protocol($providerUrl);
$post = [
    'key' => Decrypt($providerKey),
    'action' => 'balance'
];

// Sets our options array so we can assign them all at once
$options = [
    CURLOPT_URL        => $url,
    //CURLOPT_POST       => false,
    CURLOPT_POSTFIELDS => $post,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_CONNECTTIMEOUT => 5,
    CURLOPT_TIMEOUT => 5,
];

// Initiates the cURL object
$curl = curl_init();
curl_setopt_array($curl, $options);
$json = curl_exec($curl);

curl_close($curl);

//Big variable of all the values
$services = json_decode($json, true);
//Check for invalid API response
if($services['error'] == "Invalid API key"){
    return FALSE;
}else{
    return $services['balance'];
}
return FALSE;

}

Answer 1

If you are sending requests to 600 different websites in synchronous fashion, it is very likely that the request is simply exceeding PHP's time limit. Depending on what the page was outputting, it may abruptly truncate the data, resulting in this error. To see if this is the case, try only querying a few websites.

You may be able to run set_time_limit(0) in your PHP code to remove the time limit, but it still might hit some sort of browser timeout. For that reason, it is generally best to run long-running tasks from the command line, which has no time limits, like php /path/to/script.php.

If you still need the results to show up on an HTML page, you may want to consider spawning a background task, having it save its progress to a text file or database of some sort, and use AJAX requests to continually check the progress.