Difference between command line cURL and PHP cURL

Question

I have a cURL command like this:

curl 'https://www.example.com' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36' \
  -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3' \
  -H 'accept-language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7' \
  -H 'authority: www.example.com'

Executing this in a command line like in Terminal app on my Mac, results to the expected output.

(In case you test it yourself: If this output contains the word Sicherheitsüberprüfung it's geo blocked and you have to use a German IP to test it.)

I transferred the exact command to PHP cURL like this:

<?php
$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, 'https://www.example.com');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');

$headers = array();
$headers[] = 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36';
$headers[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3';
$headers[] = 'Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7';
$headers[] = 'Authority: www.example.com';
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$result = curl_exec($ch);
curl_close($ch);
echo $result;
?>

When I run this code I'm getting a message that my request was recognized as automated request/robot: It says Sicherheitsüberprüfung, means security check.

Of course, I'm using the same IP for both, command line and PHP cURL request.

Why that? Isn't command line cURL the same as PHP cURL?

Or is there anything wrong with my PHP script?

UPDATE

I fortuitously found out the following: I'm using Coda as code editor on my Mac. This has a build-in PHP rendering engine. Using this with my PHP script, the result is as expected. It's the same result I'm getting in the command line.

UPDATE 2

I made what Jannes Botis suggested in his answer. I then ran the PHP script in my Coda code editor app (what output the expected) and with MAMP as localhost (what is always recognized as automated request).

I figured out that the the code executed with MAMP was using HTTP/2 while the code executed in Coda is using HTTP/1.1. To solve this, I added the following to the script:

curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);

Now, both output exact the same string:

GET / HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
Authority: www.example.com

But, it's still the same: The one is working, the other is recognized as automated request.

Answer 1

Try to debug the request in both cases:

a) Terminal: use curl verbose mode: curl -v and check the http request sent, especially check the header list

b) php curl: print the http request using CURLINFO_HEADER_OUT:

curl_setopt($ch, CURLINFO_HEADER_OUT, true);

curl_exec($ch);

$info = curl_getinfo($ch);
print_r($info['request_header']);

Testing the different headers, what made it work was adding "Pragma: no-cache" header to the request:

$headers[] = 'Pragma: no-cache';

On the other hand, in terminal curl, I had to uppercase the request headers, e.g. User-Agent etc.

Try to create a tcp connection with fsockopen:

$fp = fsockopen("ssl://"."www.example.com", 443, $errno, $errstr, 30);
if (!$fp) {
    echo "$errstr ($errno)<br />\n";
} else {
    $out = "GET / HTTP/1.1\r\n";
    $out .= "Host: www.example.com\r\n";
    $headers = array();
    $headers[] = 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36';
    $headers[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3';
    $headers[] = 'Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7';
    $headers[] = 'Authority: www.example.com';
    $out .= $headers;
    $out .= "Connection: Close\r\n\r\n";
    fwrite($fp, $out);
    while (!feof($fp)) {
        echo fgets($fp, 1024);
    }
    fclose($fp);

and test if this works. Maybe the issue is either that php curl adds some info to the http request or the problem is on the tcp connection level, some info added there.

References

• cURL works from Terminal, but not from PHP
• PHP cURL: modify/overwrite Connection header
• Sending TCP Data with PHP

Answer 2

Command line curl :

It is a tool to transfer data to or from a server, using any of the supported protocols (HTTP, FTP, IMAP, POP3, SCP, SFTP, SMTP, TFTP, TELNET, LDAP or FILE). curl is powered by Libcurl. This tool is preferred for automation, since it is designed to work without user interaction. curl can transfer multiple file at once. For more details for Command line curl

Syntax:

curl [options] [URL...]

Example:

curl http://site.{one, two, three}.com

PHP cURL

$ch = curl_init('http://example.com/wp-login.php');
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 3);

if($this->getRequestType() == 'POST')
{
    curl_setopt($ch, CURLOPT_POST, true);
    curl_setopt($ch, CURLOPT_POSTFIELDS, 
        array(
            'user[name]'    => 'Generic+Username',
            'user[email]'   => 'mahekpatel04@gmail.com'
        );
    );
}

$response   = curl_exec($ch);

Answer 3

The issue is with ciphers selected by PHP's cURL by default.

Running curl command with -Ivs options allows us to see what ciphers it uses:

* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH

Setting them in PHP allows it to bypass this mysterious check:

curl_setopt($ch,
  CURLOPT_SSL_CIPHER_LIST,
  'ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH'
);

Also, it seems that Host header and using HTTPv2 should be added:

$headers[] = 'Host: www.11880.com';
// ...
curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_2_0);