cURL works from Terminal, but not from PHP

Question

I'm running into a rather strange issue.

I'm trying to log into a remote moodle install using curl from PHP.

I have a curl command, which works perfectly in the Terminal.

When I translate the same thing into PHP, it works, but it just doesn't login. The exact same value which successfully login via terminal, somehow trips up the login system via PHP and it doesn't login. Instead, it returns the login page again.

My cURL command (data section ommitted as it has my username and password):

curl 'http://moodle.tsrs.org/login/index.php'
-H 'Pragma: no-cache'
-H 'Origin: http://moodle.tsrs.org'
-H 'Accept-Encoding: gzip, deflate'
-H 'Accept-Language: en-US,en;q=0.8'
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537.36'
-H 'Content-Type: application/x-www-form-urlencoded'
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
-H 'Cache-Control: no-cache'
-H 'Referer: http://moodle.tsrs.org/login/index.php'
-H 'Cookie: MoodleSession=ngcidh028m37gm8gbdfe07mvs7; MOODLEID_=%25F1%25CD%2519D%25B2k%25FE%251D%25EFH%25E5t%25B1%2503%258E; MoodleSessionTest=NhzaTNij6j; _ga=GA1.2.925953522.1416155774; _gat=1; __utmt=1; __utma=147409963.925953522.1416155774.1416642544.1416692798.3; __utmb=147409963.1.10.1416692798; __utmc=147409963; __utmz=147409963.1416155774.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)'
-H 'Connection: keep-alive'

The corresponding PHP code:

function login() {
    $username = $_POST['username'];
    $password = $_POST['password'];

    if(!isset($_POST['username']) || !isset($_POST['password'])) {
        echo "No login data received";
        return;
    }

    $creq = curl_init();

    $data = array('username' => $username, 'password' => $password, 'testcookies'=> '1');

    $headers = array('Pragma: no-cache', 'Origin: http://moodle.tsrs.org', 'Accept-Encoding: ', 'Accept-Language: en-US,en;q=0.8', 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537.36', 'Content-Type: application/x-www-form-urlencoded', 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'Cache-Control: no-cache', 'Cookie: MoodleSession=ngcidh028m37gm8gbdfe07mvs7; MOODLEID_=%25F1%25CD%2519D%25B2k%25FE%251D%25EFH%25E5t%25B1%2503%258E; MoodleSessionTest=NhzaTNij6j; _ga=GA1.2.925953522.1416155774; _gat=1; __utmt=1; __utma=147409963.925953522.1416155774.1416642544.1416692798.3; __utmb=147409963.1.10.1416692798; __utmc=147409963; __utmz=147409963.1416155774.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)', 'Connection: keep-alive' );
        curl_setopt_array($creq, array(
        CURLOPT_URL => 'http://moodle.tsrs.org/login/index.php',
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_POST => true,
        CURLOPT_ENCODING => '',
        CURLINFO_HEADER_OUT => true,
        CURLOPT_POSTFIELDS => $data,
        CURLOPT_HTTPHEADER => $headers,
        CURLOPT_FOLLOWLOCATION => false
    ));

    $output = curl_exec($creq);

    echo print_r(curl_getinfo($creq));

    echo "\n" . $output . "\n";
}

And the output of curlinfo:

Array
(
    [url] => http://moodle.tsrs.org/login/index.php
    [content_type] => text/html; charset=utf-8
    [http_code] => 200
    [header_size] => 541
    [request_size] => 945
    [filetime] => -1
    [ssl_verify_result] => 0
    [redirect_count] => 0
    [total_time] => 1.462409
    [namelookup_time] => 0.002776
    [connect_time] => 0.330766
    [pretransfer_time] => 0.330779
    [size_upload] => 365
    [size_download] => 8758
    [speed_download] => 5988
    [speed_upload] => 249
    [download_content_length] => -1
    [upload_content_length] => 365
    [starttransfer_time] => 0.694866
    [redirect_time] => 0
    [certinfo] => Array
        (
        )

    [primary_ip] => 125.22.33.149
    [redirect_url] =>
    [request_header] => POST /login/index.php HTTP/1.1
Host: moodle.tsrs.org
Pragma: no-cache
Origin: http://moodle.tsrs.org
Accept-Language: en-US,en;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Cache-Control: no-cache
Cookie: MoodleSession=ngcidh028m37gm8gbdfe07mvs7; MOODLEID_=%25F1%25CD%2519D%25B2k%25FE%251D%25EFH%25E5t%25B1%2503%258E; MoodleSessionTest=NhzaTNij6j; _ga=GA1.2.925953522.1416155774; _gat=1; __utmt=1; __utma=147409963.925953522.1416155774.1416642544.1416692798.3; __utmb=147409963.1.10.1416692798; __utmc=147409963; __utmz=147409963.1416155774.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
Connection: keep-alive
Content-Length: 365
Expect: 100-continue
Content-Type: application/x-www-form-urlencoded; boundary=----------------------------83564ee60d56


)

Does anyone know any possible reason for this? I've tried swapping out the hard coded cookie with COOKIEFILE and COOKIEJAR, but it doesn't change anything.

I've done the same before but can't find the code- I think its because it redirects after logging in, so you need to allow it to redirect - use something like curl_setopt($curl, CURLOPT_MAXREDIRS, 10); — Russell England, Nov 24 '14 at 08:24
The headers are **NOT** the same in the question. Command line cURL includes a `Referer` header and a value for `Accept-Encoding`. The PHP cURL does not include the Referer at all and an empty space for Accept-Encoding. @RichardTheKiwi, just to clarify, is your problem also moddle specific? — HPierce, Dec 22 '16 at 18:34

score 11 · Accepted Answer · answered Dec 22 '16 at 19:34

This could have been debugged better by seeing everything that was actually done by cURL. This is done by adding the verbose flag to the command: -v.

$ curl localhost/login [...] -v

We can get the same output from PHP's curl by adding the CURLOPT_VERBOSE option. Note that by adding this line you are instructing cURL to output the same information to STDOUT - it will not be returned and content will not be sent to the browser, so this must be debugged in the terminal.

curl_setopt($curl, CURLOPT_VERBOSE, 1);

By doing it this way, you can get a consistent and comparable output of both HTTP requests, it should look sommthing like this:

POST / HTTP/1.1
Host: localhost:3000
Pragma: no-cache
Origin: http://moodle.tsrs.org
Accept-Language: en-US,en;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Cache-Control: no-cache
Cookie: MoodleSession=ngcidh028m37gm8gbdfe07mvs7; MOODLEID_=%25F1%25CD%2519D%25B2k%25FE%251D%25EFH%25E5t%25B1%2503%258E; MoodleSessionTest=NhzaTNij6j; _ga=GA1.2.925953522.1416155774; _gat=1; __utmt=1; __utma=147409963.925953522.1416155774.1416642544.1416692798.3; __utmb=147409963.1.10.1416692798; __utmc=147409963; __utmz=147409963.1416155774.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
Connection: keep-alive
Content-Length: 250
Expect: 100-continue
Content-Type: application/x-www-form-urlencoded; boundary=------------------------b4d79f17a3887f2d

< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Content-Type: application/json; charset=utf-8
< Content-Length: 2
< ETag: W/"2-mZFLkyvTelC5g8XnyQrpOw"
< Date: Thu, 22 Dec 2016 19:13:40 GMT
< Connection: keep-alive

Left: Command line cURL as provided in the question (with extra -v flag)

Right: PHP cURL as posted in the question (with CURLOUT_VERBOSE enabled)

As you can see, the headers aren't the same, and this makes that clear. The PHP invocation is missing Accept-Encoding and Referer headers.

If that didn't turn up anything, let's try changing some more cURL settings in PHP back to the original cURL defaults.

Internally, PHP opts to override some defaults in cURL without telling you. While these settings should be fine, let's change them back by explicitly reseting them back to cURL defaults:

curl_setopt($curl, CURLOPT_DNS_CACHE_TIMEOUT, 60);
curl_setopt($curl, CURLOPT_DNS_USE_GLOBAL_CACHE, 0);
curl_setopt($curl, CURLOPT_MAXREDIRS, -1);
curl_setopt($curl, CURLOPT_NOSIGNAL, 0);

Moodle has an option to verify HTTP_REFERER. I was going to post the same answer before I saw yours. The Referer header is **definitely** something to check, even if more problems might be present too. — LSerni, Dec 23 '16 at 14:14

score 4 · Answer 2 · answered Dec 22 '16 at 15:14

Use http_build_query on the $data array before passing to curl to avoid Content-Type: application/x-www-form-urlencoded; boundary=---. This also ensures to encode any special characters from the password.

curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($data));

Reshape your curl requests as follows:

Make a GET request to the login page with pointing a cookie file at $cookies = '/tmp/some/dir/xyz.cookie.txt'. Make sure using full path for cookie name. And then close the curl handle. This will store the cookie in cookie file.

$creq = curl_init();
curl_setopt_array($creq, array(
  CURLOPT_URL => 'http://moodle.tsrs.org/login/index.php',
  CURLOPT_RETURNTRANSFER => true,
  CURLOPT_ENCODING => '',
  CURLINFO_HEADER_OUT => true,
  CURLOPT_HTTPHEADER => $headers,
  CURLOPT_FOLLOWLOCATION => false,
  CURLOPT_COOKIEJAR => $cookies // save cookie
));
$output = curl_exec($creq);
curl_close($creq);

Now make the POST request with second curl request. This time point the same cookie file with COOKIEFILE option.

$creq = curl_init();
curl_setopt_array($creq, array(
  CURLOPT_URL => 'http://moodle.tsrs.org/login/index.php',
  CURLOPT_RETURNTRANSFER => true,
  CURLOPT_POST => true,
  CURLOPT_ENCODING => '',
  CURLINFO_HEADER_OUT => true,
  CURLOPT_POSTFIELDS => http_build_query ($data),
  CURLOPT_HTTPHEADER => $headers,
  CURLOPT_FOLLOWLOCATION => false,
  CURLOPT_COOKIEJAR => $cookies, // save cookie
  CURLOPT_COOKIEFILE => $cookies // load cookie
);
$output = curl_exec($creq);
curl_close($creq);

It can happen sometimes the server look for the cookie when a login request made (to ensure that the request came after visiting the login page).

`This also ensures to encode any special characters from the password` that's misleading, `multipart/form-data` encoded data is binary safe, when passing an array, curl will automatically encode it. additionally, when transferring large non-ascii data, it uses considerably less bandwidth than `application/x-www-form-urlencoded`, form-data has a bigger header overhead (thus why it doesn't use less bw for small data), but it does not encode the data at all. with urlencoded, (almost) every non-ascii byte is 3 bytes encoded. in form-data, all bytes (including non-ascii bytes) is exactly 1 byte. — hanshenrik, Dec 25 '16 at 14:40
but yeah, this is probably why it's not working. the curl command line uses `application/x-www-form-urlencoded` encoding, while the php curl (using his code) uses `multipart/form-data` encoding, which the sever probably rejects. using http_build_query will make the php curl code also use `application/x-www-form-urlencoded` — hanshenrik, Dec 25 '16 at 14:44

score 2 · Answer 3 · answered Nov 23 '14 at 16:07

Most likely your problem is related to HTTP header Expect: 100-continue that cURL sends by default for each POST request.

The Expect: 100-continue header is used in POST requests containing big data when client is not sure that server will accept such request. In this case client first sends request with only headers including Expect: 100-continue and, if the server's response is successful, send the same request with body (POST data).

The problem is that not all web servers handle this header correctly. In such cases sending this header is undesired.

The solution is manually remove Expect header from sending headers by passing array('Expect:') to CURLOPT_HTTPHEADER option. In your case you can simply add 'Expect:' string to $headers array:

$headers[] = 'Expect:';

@Raghav Sood You have to show response headers. This might help to find the source of problem. Add `CURLOPT_HEADER` to the array of options and output response headers. Also you have to provide response headers of your CLI cURL request — hindmost, Nov 24 '14 at 15:26

score 1 · Answer 4 · answered Feb 21 '22 at 08:52

1

I solved the issue by setting a User-Agent

$headers = array(
        'Accept: */*',
        'User-Agent: curl/7.68.0',
        'Accept-Encoding: deflate,gzip,br',
        'Content-Type:application/json',
);

answered Feb 21 '22 at 08:52

Sunding Wei

1,803
17
12

score 0 · Answer 5 · answered Nov 23 '14 at 10:52

0

I suspect your first attempt using the curl command is using the GET method in the index.php file. I suggest you enable --trace-ascii on your first curl request in the command line and see whether a GET request is being made by the page or not. If yes, you should change your PHP script which is using the POST method. If you change the CURLOPT_POST to false, the PHP script should work.

answered Nov 23 '14 at 10:52

Ehsan

4,334
7
39
59

I'm fairly certain it's a POST request, since I lifted it from the Chrome Dev tools and Moodle's documentation strictly says login is only via POST. Also, the page I'm curling does seem to receive my POST data, as the login page it returns to PHP has my username prefilled, which was sent as a POST variable – Raghav Sood Nov 23 '14 at 10:56

cURL works from Terminal, but not from PHP

5 Answers5

Linked