Why does a cURL connection fail (without error) if no timeout is set?

Question

I have a PHP script that connects to an URL through cURL and then does something, depending on the returned HTTP status code:

$ch = curl_init();
$options = array(
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_URL            => $url,
            CURLOPT_USERAGENT      => "What?!?"
);
curl_setopt_array($ch, $options);
$out = curl_exec($ch);
$code = curl_getinfo($ch)["http_code"];
curl_close($ch);

if ($code == "200") {
    echo "200";
} else {
   echo "not 200";
}

Some webservers are slow to reply, and although the page is loaded in my browser after a few seconds my script, when it tries to connect to that server, tells me that it did not receive a positive ("200") reply. So, apparently, the connection initiated by cURL timed out.

But why? I don't set a timeout in my script, and according to other answers on this site the default timeout for cURL is definitely longer than the three or four seconds it takes for the page to load in my browser.

So why does the connecion time out, and how can I get it to last longer, if, apparently, it is already set to infinite?

---

Notes:

• The same URL doesn't always time out. So sometimes cURL can connect.
• It is not one specific URL that sometimes times out, but different URLs at different times.
• I'm on a shared server, so I don't have root access to any files.
• I tried to look at curl_getinfo($ch) and curl_error($ch) – as per @drew010's suggestion in the comments – but both were empty whenever the problem happened.
• The whole script runs for a little more than one minute. In this time it connects to 300+ URLs successfully. Even when one of the URLs fails, the other connections are successfully made. So the script does not time out.
• cURL does not time out either, because when I try to connect to an URL with a script sleeping for 59 seconds, cURL successfully connects. So apparently the slowness of the failing URL is not a problem in itself for cURL.

---

Update

Following @Karlos' suggestion in his answer, I used:

CURLOPT_VERBOSE        => 1,
CURLOPT_STDERR         => $curl_log

(using code from this answer) and found the following in $curl_log when an URL failed (URL and IP changed):

* About to connect() to www.somesite.com port 80 (#0)
*   Trying 104.16.37.249... * connected
* Connected to www.somesite.com (104.16.37.249) port 80 (#0)
GET /wp_german/?feed=rss2 HTTP/1.1
User-Agent: myURL
Host: www.somesite.com
Accept: */*

* Recv failure: Connection reset by peer
* Closing connection #0

So, I have found the why – thank you @Karlos! – and apparently @Axalix was right and it is a network problem. I'll now follow suggestions given on this site for that kind of failure. Thanks to everyone for their help!

Answer 1

My experience working with curl showed me that sometimes when using the option:

CURLOPT_RETURNTRANSFER => true

the server might not give a successful reply or, at least, a successful reply within the timeframe that curl has to receive the response and cache it, so the results are returned by the curl into the variable you assign. In your code:

$out = curl_exec($ch);

In this stackoverflow question CURLOPT_RETURNTRANSFER set to true doesnt work on hosting server, you can see that that the option CURLOPT_RETURNTRANSFER is directly affected by the requested host web server implementation.

As you are using explicitly the response body, and your code relies on the response headers, a good way to solve this might be to:

CURLOPT_RETURNTRANSFER => false

and execute the curl code to work on the response headers.

Once you have the header with the code you are interested, you could run a php script that echoes the curl response and parse it by yourself:

<?php
    $url=isset($_GET['url']) ? $_GET['url'] : 'http://www.example.com';
    $ch= curl_init();
    $options = array(
            CURLOPT_RETURNTRANSFER => false,
            CURLOPT_URL            => $url,
            CURLOPT_USERAGENT      => "myURL"
    );
    curl_setopt_array($ch, $options);
    curl_exec($ch);
    curl_close($ch);
?>

In any case the reply to your question why your request does not get an error, I guess that the use of the option CURLOPT_NOSIGNAL and the different timeout options explained in the set_opt php manual might get you closer to it.

In order to dig further, the option CURLOPT_VERBOSE might help you to have extra information about the request behavior through the STDERR.

Answer 2

The reason may be your hosting provider is imposing some limits on outgoing connections.

Here is what can be done to secure your script:

1. Create a queue in DB with all the URLs that need to be fetched.

2. Run cron every minute or 5 minutes, take a few URLs from DB - mark them as in progress.

3. Try to fetch those URLs. Mark every fetched URL as success in DB.

4. Increment failure count for unsuccessful ones.

5. Continue going through queue until its empty.

If you implement such a solution you will be able to process every single URL under any unfavourable conditions.