GET Request works with python 'requests' library, but does not work with curl

Question

I need to probably modify my cURL request to make it work. You can find the needed url from searching on Qantas for example (Melbourne to Sydney, one way, the next day flights), you get a page with the results. Then if you mouseover the flight number, you get a dialogue box. In FF network, you can see a get request happen as you mouseover which retrieves a json document.

The cURL code that does not work:

$flight_data_url = 'http://book.qantas.com.au/pl/QFOnline/wds/FlifoInfoServlet;jsessionid=eFVNx9ORnACoaRyRp_Iu675LrmM6JfRkDHpa2c6I60bhJABPBeW3!764143094!1637886517!1440118985617?AIRLINE_CODE=QF&B_DATE=201508220600&B_LOCATION=MEL&E_LOCATION=SYD&FLIGHT_NUMBER=400&LANGUAGE=GB&SITE=QFQFQFBD'

$handle = curl_init();
$options = array(
                CURLOPT_RETURNTRANSFER => true,
                CURLOPT_FOLLOWLOCATION => true,
                CURLOPT_CAINFO => $path . '\cacert.pem',
                CURLOPT_URL => $flight_data_url,
                CURLOPT_HTTPHEADER => array('Host'=>'book.qantas.com.au','User-Agent'=>'Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)','Accept'=>'application/json, text/javascript, */*; q=0.01','Accept-Language'=>'en-US,en;q=0.5','Accept-Encoding'=>'gzip, deflate','Content-Type'=>'text/javascript; charset=utf-8','X-Requested-With'=>'XMLHttpRequest','Referer'=>'http://book.qantas.com.au/pl/QFOnline/wds/OverrideServlet','Cookie'=>$flight_cookie_file,'Connection'=>'keep-alive')
            );

            curl_setopt_array($handle, $options);
            $flight_model_data = curl_exec($handle);

            curl_close($handle);

The python code that does work:

#!/usr/bin/env python
import requests


def getaurl():

    url = 'http://book.qantas.com.au/pl/QFOnline/wds/FlifoInfoServlet;jsessionid=LuJYL8OJHk_DUIPcYbcfT343gYMEw3b9ej4Vt_MATLerQG0kNIZR!932397328!976639289!1440293569417?AIRLINE_CODE=QF&B_DATE=201508240630&B_LOCATION=MEL&E_LOCATION=SYD&FLIGHT_NUMBER=404&LANGUAGE=GB&SITE=QFQFQFBD'

    headers = { "Host":"book.qantas.com.au",'user-agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:36.0) Gecko/20100101 Firefox/36.0', "Accept":"application/json, text/javascript, */*; q=0.01", "Accept-Language":"en-US,en;q=0.5", "Accept-Encoding":"gzip, deflate","Content-Type":"text/javascript; charset=utf-8", "X-Requested-With":"XMLHttpRequest", "Referer":"http://book.qantas.com.au/pl/QFOnline/wds/OverrideServlet", "Connection":"keep-alive" }
    r = requests.get(url, headers=headers)

    print r.text

getaurl()

You are saying it is not working but without replicating the code locally and using your certificates maybe we cannot debug your code. Can you check if you get a curl error ? http://php.net/manual/en/function.curl-error.php — Bizmate, Aug 23 '15 at 20:42
There is no curl error, it is returning a page with a 404 not found error rfc 2068. Whatever that means. — Lawrence DeSouza, Aug 23 '15 at 21:07

score 1 · Accepted Answer · answered Aug 23 '15 at 23:55

Ok i debugged your code. You are doing two things wrong to get it to work.

1 ) you need to tell curl that it needs to handle a GZIP encoded connection

curl_setopt($handle, CURLOPT_ENCODING, "gzip");

2 ) set your headers using as per manual (it is not a key => value list) search CURLOPT_HTTPHEADER in http://www.php.net/manual/en/function.curl-setopt.php

A full working example. Just change the url and the cookie value to make sure the session is working

<?php header('Content-Type: text/html; charset=utf-8'); ?>

<html>
<head>
    <title>Fixed by Bizmate.biz</title>
</head>
<body>
<?php

$flight_data_url = "http://book.qantas.com.au/pl/QFOnline/wds/FlifoInfoServlet;jsessionid=Wzpc3MHW1Z1dQVnKNgjL1zGOFYs5jTi0yey2JJB3OsumjMe42a8C!-111476454!1015406667!1440372015574?AIRLINE_CODE=QF&B_DATE=201508261625&B_LOCATION=YVR&E_LOCATION=LAX&FLIGHT_NUMBER=3352&LANGUAGE=GB&SITE=QFQFQFBW";

//$flight_data_url = 'http://book.qantas.com.au/pl/QFOnline/wds/FlifoInfoServlet;jsessionid=eFVNx9ORnACoaRyRp_Iu675LrmM6JfRkDHpa2c6I60bhJABPBeW3!764143094!1637886517!1440118985617?AIRLINE_CODE=QF&B_DATE=201508220600&B_LOCATION=MEL&E_LOCATION=SYD&FLIGHT_NUMBER=400&LANGUAGE=GB&SITE=QFQFQFBD';

$handle = curl_init();

$flight_cookie_file = 'DWM_XSITECODE=QFQFQFBW; wt_21=false; QSession=D884F2710404EC6F60E402761C3296A8; TLTUID=8B4FCC6D7E9CD0AC141750417B7DDB37; qantas_isDevice=type#desktop|os#Linux; _wt.user-676736=WT3vpi-xU8uREVQbLyHDUf1TwZLPluq_yfjSdWFE4rxld2gVwcxlwtwsrUqhlQL-MMM-zC-j0Nk1MDugbvMEcuRZWkUrevpXCBQnAHuAQ~~; amdfa=done; pastSearchCookie=REGION:am:TRIPTYPE:O:FROM:Vancouver:TO:Sydney:DEPDAY:Wednesday:DEPMONTH:8:DEPYEAR:2015:RETDAY:Wednesday:RETMONTH:10:RETYEAR:2015:FLEXIBLE:false:ADULTS:1:CHILDREN:0:INFANTS:0:TRAVELCLASS:ECO|REGION:am:TRIPTYPE:O:FROM:Vancouver:TO:Sydney:DEPDAY:Wednesday:DEPMONTH:8:DEPYEAR:2015:RETDAY:Wednesday:RETMONTH:10:RETYEAR:2015:FLEXIBLE:true:ADULTS:1:CHILDREN:0:INFANTS:0:TRAVELCLASS:ECO|REGION:am:TRIPTYPE:R:FROM:Vancouver:TO:Sydney:DEPDAY:Monday:DEPMONTH:9:DEPYEAR:2015:RETDAY:Wednesday:RETMONTH:10:RETYEAR:2015:FLEXIBLE:false:ADULTS:1:CHILDREN:0:INFANTS:0:TRAVELCLASS:ECO|; aam_uuid=09862152334998442092616937494749808238; s_cc=true; s_fid=77B7E8432AD3E6A0-170B4CF864D887BC; s_qntttc=1440372017198; s_sq=%5B%5BB%5D%5D; s_vi=[CS]v1|2AED21DE051D0C8B-40001905C00064CC[CE]; aam_uuid=09862152334998442092616937494749808238; _wt.mode-676736=WT3PVZU041xU6Y~; webserver=server44; 3_uc=region#AM|tvldate#20150826|arr#SYD|locale#en|dep#YVR|country#ca';

$headerArray = array(
    'Host: book.qantas.com.au',
    'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36',
    'Accept: application/json, text/javascript, */*; q=0.01',
    'Accept-Language: en-GB,en;q=0.8,en-US;q=0.6,it-IT;q=0.4,it;q=0.2',
    'Accept-Encoding: gzip, deflate, sdch',
    'Content-Type: text/javascript; charset=utf-8',
    'X-Requested-With: XMLHttpRequest',
    'Referer: http://book.qantas.com.au/pl/QFOnline/wds/OverrideServlet',
    'Cookie: ' . $flight_cookie_file,
    'Connection: keep-alive',
    'Pragma: no-cache'
);

$options = array(
    CURLOPT_RETURNTRANSFER => true,
    CURLINFO_HEADER_OUT => true,
    CURLOPT_ENCODING => 'gzip',
    CURLOPT_URL => $flight_data_url,
    CURLOPT_HTTPHEADER => $headerArray

);

#
curl_setopt_array($handle, $options);

$flight_model_data = curl_exec($handle);



if(curl_exec($handle) === false)
{
    echo 'Curl error: ' . curl_error($handle);
}
else
{
    echo 'Char encoding ' . mb_detect_encoding($flight_model_data) . ' <br/><br/>';

    echo 'Operation Completed without any errors : <br/>' . $flight_model_data . " ENDED\n\n <br/><br/>";

    file_put_contents("test.txt",$flight_model_data);

    echo "<br/>header : " . json_encode(curl_getinfo ($handle) );
}

curl_close($handle);
?>
</body>
</html>

GET Request works with python 'requests' library, but does not work with curl

1 Answers1

Linked