2

Good Day!

cURL is acting very slow when requesting a page. I know its not the page requested, because the page returns in the browser instantly.

2 things I've noticed

  • The starttransfer_time is regularly almost 20
  • The local_port seems to change every time. Is that normal?
  • Occasionally, cURL will respond instantly

I have the following code:

$ch = curl_init(); 
curl_setopt($ch, CURLOPT_URL, $url ); 
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, false);
$output = curl_exec($ch); 
curl_close($ch);

Echoing curl_getinfo() gives me the following

[url] => http://127.0.0.1:80/wpengine/?json=t
[content_type] => text/html; charset=iso-8859-1
[http_code] => 302
[header_size] => 215
[request_size] => 64
[filetime] => -1
[ssl_verify_result] => 0
[redirect_count] => 0
[total_time] => 17.238
[namelookup_time] => 0
[connect_time] => 0
[pretransfer_time] => 0
[size_upload] => 0
[size_download] => 221
[speed_download] => 12
[speed_upload] => 0
[download_content_length] => 221
[upload_content_length] => 0
[starttransfer_time] => 17.238
[redirect_time] => 0
[certinfo] => Array
    (
    )

[primary_ip] => 127.0.0.1
[primary_port] => 80
[local_ip] => 127.0.0.1
[local_port] => 51875
[redirect_url] =>

Can anyone give me some pointers on how to figure out what is going on?

Here are a few lines from the Apache access log

127.0.0.1 - - [06/Dec/2013:12:01:22 -0500] "GET /wpengine/?json=t HTTP/1.1" 302 221
127.0.0.1 - - [06/Dec/2013:12:01:12 -0500] "GET /community HTTP/1.1" 200 6266
127.0.0.1 - - [06/Dec/2013:12:01:22 -0500] "GET /public/js/jquery.js?b=10 HTTP/1.1" 304 -
Ben
  • 377
  • 2
  • 6
  • 26
  • 1
    If you're running this locally, what server do you have set up? Can you check the server logs? – brandonscript Dec 06 '13 at 16:19
  • @r3mus I am running Wamp with Apache. I edited my post to include a few lines from the access log – Ben Dec 06 '13 at 17:03
  • 2
    Curious if maybe CURL is trying IPv6 first, and failing to resolve? Try adding `curl_setopt($ch, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 );` – brandonscript Dec 06 '13 at 17:27
  • Hmm, that did not make a difference. I find this very puzzling. Could it have something to do with sessions as it is on the same domain? Just a shot in the dark... – Ben Dec 06 '13 at 18:55
  • 1
    Not really sure, but looking at your logs, once CURL actually *hits* the server, it's super fast. Any way you can try this on a public server somewhere to eliminate the localhost variable? – brandonscript Dec 06 '13 at 19:06
  • Sure! Ill give it a shot, and let you know what happens. Thanks helping! – Ben Dec 06 '13 at 20:07
  • @remus option helps me with connection by OAuth protocol – mikatakana Oct 12 '14 at 09:41

4 Answers4

5

This is mostly because of the Expect: 100-continue header CURL send to the server when it handles large POST data, and the server just happened not support this feature. You can confirm by add -vv for curl command line and see the outputs.

To solve this, you can either

  1. fix your server to support Expect: 100-continue header, or
  2. force curl not to send it by add -H "Expect:" option (in php, you can refer to this answer: How can I stop cURL from using 100 Continue?).

Links about Expect: 100-continue:

https://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.2.3 https://support.urbanairship.com/entries/59909909--Expect-100-Continue-Issues-and-Risks

Simon East
  • 55,742
  • 17
  • 139
  • 133
Reorx
  • 2,801
  • 2
  • 24
  • 29
2

I've found a similar issue when hitting Google Analytics' MP API ... the starttransfer time is consistently 1.0 seconds plus a tiny amount, from which I infer there is a 1 second delay somewhere, I am guessing in cURL. It's not using CPU at my end, and I somehow doubt Google is putting in the delay... otuput from curl_getinfo() ...

Array
(
    [url] => http://www.google-analytics.com/collect
    [content_type] => image/gif
    [http_code] => 200
    [header_size] => 388
    [request_size] => 201
    [filetime] => -1
    [ssl_verify_result] => 0
    [redirect_count] => 0
    [total_time] => 1.021528
    [namelookup_time] => 8.7E-5
    [connect_time] => 8.8E-5
    [pretransfer_time] => 0.000324
    [size_upload] => 1736
    [size_download] => 35
    [speed_download] => 34
    [speed_upload] => 1699
    [download_content_length] => 35
    [upload_content_length] => 1736
    [starttransfer_time] => 1.002496
    [redirect_time] => 0
    [certinfo] => Array
        (
        )

    [redirect_url] => 
)
Dave Crooke
  • 670
  • 1
  • 6
  • 12
  • Yeah, a quick command line script using lwp-request (starting a new Perl interpreter for every POST) is miles faster than cURL looped in one PHP process. – Dave Crooke Jan 30 '14 at 01:01
  • 5
    I found out a way to fix this, at least for my needs ... I was passing the above argument list to CURLOPT_POSTFIELDS expecting it to generate a urlencoded post body, but in fact it generates multipart. In order to get it to send a plain URLencoded POST you have to encode the body parameters yourself and pass a ready-to-use string. For some reason, the multipart process has a one second delay, the plain POST works at full speed. – Dave Crooke Jan 31 '14 at 00:32
  • YES. Even though I don't understand why multipart takes so long to start, it worked, delay has gone! You should really update your answer with this info – Alexander Malakhov Sep 18 '15 at 09:24
1

One of the comments by @brandonscript in this thread helped me solve the problem.

His answer:

Curious if maybe CURL is trying IPv6 first, and failing to resolve?

Try adding curl_setopt($ch, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 );

My problem was the opposite, I had to set the following option:

curl_setopt($curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V6 );

Thank you, wonderful people.

Community
  • 1
  • 1
k dimitrov
  • 369
  • 3
  • 5
0

I used Fiddler app from Telerik to debug connections, and did not find any problem, I think it may be related to web server child process workers. To debug using Fiddler paste the following line before curl_exec($ch):

curl_setopt($ch, CURLOPT_PROXY, '127.0.0.1:8888');

Thank you all for the pointers

togobites
  • 161
  • 1
  • 5