Send ajax request through CURL

Question

An API request needs to be sent. For some reason, the server is blocking CURL request, but it approves an XHR ajax request. I could send an ajax request, but another problem arises - Mixed content my website is served over HTTPS but the request that needs to be sent is over HTTP so I cannot use ajax.

I am looking for a way to simulate ajax request through CURL, in some way, trick the server to believe that the CURL request is indeed an ajax request.

Here's what I have tried.

This is my CURL request.

    $ch = curl_init();
        curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; Linux x86_64)');
        curl_setopt($ch, CURLOPT_REFERER, 'server's url');
        curl_setopt($ch, CURLOPT_AUTOREFERER, true);
        curl_setopt($ch, CURLOPT_HTTPHEADER, array(
        'Accept:application/json, text/javascript, */*; q=0.01',
        'Accept-Encoding:gzip, deflate',
        'Accept-Language:en-US,en;q=0.9',
        'Connection:keep-alive',
        'Content-Type: application/json; charset=utf-8',
        'X-Requested-With: XMLHttpRequest',
        '__RequestVerificationToken: $token'
        ));
        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
        curl_setopt($ch, CURLOPT_MAXREDIRS, 10);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
        curl_setopt($ch, CURLOPT_COOKIEJAR, base_path().'/cookies.txt');
        curl_setopt($ch, CURLOPT_COOKIEFILE, base_path().'/cookies.txt');
        $buffer = curl_exec($ch);
        if(curl_error($ch))
        {
        $buffer =   curl_error($ch);
        }
        curl_close($ch);

return $buffer;

This curl request is blocked

But, this ajax request goes through my localhost, but since my live website uses HTTPS I cannot really use it.

     $.ajax({
          type: "get",
          xhrFields: { withCredentials:true },
          url: http://apiendpoint.com,
          success: function(data)
          {
           // console.log(data);
          }
    })

How do you know it's blocked? (I.e., versus simply not working?) What the HTTP status you're getting? — Alex Howansky, Jan 11 '18 at 19:25
The response received through ajax request is `success` , while curl request response is `invalid` — Dhiraj, Jan 11 '18 at 19:26
Try this -> curl_setopt($ch, CURLOPT_HTTPHEADER, array("X-Requested-With: XMLHttpRequest", "Content-Type: application/json; charset=utf-8")); — d_void, Jan 11 '18 at 19:27
If you're seeing the text "invalid" in the response, then you're not being blocked. The request completed and you're getting a failure message. Try running curl with `-v` (for command line) or run `curl_getinfo()` (for PHP curl) to dump extra info. — Alex Howansky, Jan 11 '18 at 19:29
If I had to guess, there's a login system involved here and the cURL request is not "logged in" but the browser is because it has the cookie. @zuif, can you verify? — Tim G, Jan 11 '18 at 19:34
@TimG CURL has the cookie associated with it as well. But, there's no login system. There is a verification code sent on the phone, and if the correct verification is passed it goes through ajax but does not go through curl — Dhiraj, Jan 11 '18 at 19:37
@zuif cURL has no information about the cookies from the browser in the request it sends - it only tracks cookies for the connection it opens - i.e. it's like running chrome and firefox - main website in chrome, then loading the ajax url in firefox. The verification code is most likely setting a cookie, or a php session variable to signify that the user has been verified. This cookie, or session is not available to cURL. — Tim G, Jan 11 '18 at 19:46
@TimG In that case, what could be the reason of a xhr request being verified, but not the curl request? — Dhiraj, Jan 11 '18 at 19:48
@zuif xhr is coming from the browser and has session and cookies that are sent to the server. open up developer tools and view the request headers for the ajax call - I suspect you'll see cookies there. as a test, you should be able to (in chrome I know you can) `copy as curl` to run the command on the command line. it will add in the relevant cookies and headers from the browser. If you add these in your php code it will probably work, but that's not the solution long term as it changes for each person. — Tim G, Jan 11 '18 at 19:51
@TimG Only thing I am missing is Cookie , but I am setting cookie here `curl_setopt($ch, CURLOPT_COOKIEJAR, base_path().'/cookies.txt'); curl_setopt($ch, CURLOPT_COOKIEFILE, base_path().'/cookies.txt');` in my curl request. The API goes through other requests, but only fails when verification comes up. Copying the headers of verification, it's the same as other requests. — Dhiraj, Jan 11 '18 at 20:08
@zuif read my last comment carefully - cURL is a different browser - cookies aren't shared. — Tim G, Jan 11 '18 at 20:10
@TimG confusing stuff! curl shares the same cookie `cookies.txt` since opening the initial connection and the verification code also uses the same cookie. I just cannot figure out how to make this work. — Dhiraj, Jan 11 '18 at 20:20
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/163018/discussion-between-zuif-and-tim-g). — Dhiraj, Jan 11 '18 at 21:10
Duplicate of https://stackoverflow.com/questions/5972443/php-simulate-xhr-using-curl ? — huysentruitw, Jan 15 '18 at 18:06
@WouterHuysentruit I have followed it, doesn't solve my problem — Dhiraj, Jan 15 '18 at 18:07
@WouterHuysentruit It's not in the question, but I have tried every possible header combinations there might be, doesn't make a difference. — Dhiraj, Jan 15 '18 at 18:12
First of all, why do you duplicate instructions? curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); — Michele Carino, Jan 21 '18 at 12:48
Can you try re-using the curl object `$ch` instead of closing it? — Tarun Lalwani, Jan 21 '18 at 13:31

lintabá · Accepted Answer · 2018-01-15T17:57:13.310

20

in chrome, you can copy a working curl expression from developer toolbar. Try with that one from cli. If that works, you can figure out which parts are required and which parts are not. Then you can transcript it to php.

If you have doubts if the same thing happens from php than from curl, just try it with requestbin.

edited Jan 15 '18 at 17:57

answered Jan 15 '18 at 16:51

lintabá

733
9
18

I did this, I tried using the same exact headers, but no good. – Dhiraj Jan 15 '18 at 17:49
Never used requestbin, how can I use it? And how do I compare my curl request and browser request? – Dhiraj Jan 15 '18 at 18:21
- create a bin on requestbin. (this will give you an url) - replace the original url with the given one in the curl request and execute it - check the result on requestbin - send a request from your site with ajax to the requestbin url - check the new one too - compare the two results – lintabá Jan 15 '18 at 18:51
Thank you, nice tool. I tried sending the exact same header as received by requestbin, but still doesn't work :( – Dhiraj Jan 15 '18 at 21:15
I have updated the question with the URL in question, can you please check it and let me know if you can work it out – Dhiraj Jan 20 '18 at 16:52

Felipe Valdes · Answer 2 · 2018-01-16T13:08:12.737

0

I think it is possible the header for token may not be what you think it is, since given $a==1, '$a' converts to $a, but "$a" converts to "1" (notice single quotes vs double quotes).

in your example, try replacing:

'__RequestVerificationToken: $token'

with:

"__RequestVerificationToken: $token"

and let us know if that solves the problem.

Consider using passthru("curl command here..."); using the suggestion from lintabá

edited Jan 16 '18 at 13:08

answered Jan 16 '18 at 04:10

Felipe Valdes

1,998
15
26

Thank you. But there is no `$token` variable, I saw an example mentioning this header with `$token` explicitly stated, so I tried that one too. – Dhiraj Jan 16 '18 at 04:58
1

so this isn't the real code that is giving the issue then, but instead some form of pseudocode? – Felipe Valdes Jan 16 '18 at 13:06
@zuif you should post the code that is not working (hide any sensible data) so we can try to debug it – Cliff Burton Jan 20 '18 at 14:46
@CliffBurton I have updated the question with the URL in question, can you please check it and let me know if you can work it out – Dhiraj Jan 20 '18 at 16:52
why not put spaces after the : in the headers? – Felipe Valdes Jan 21 '18 at 03:25

score -1 · Answer 3 · answered Jan 21 '18 at 14:07

-1

You can use 2-stage request

1 curl .... -c ${cookie_file}

2 curl .... -b ${cookie_file} -c ${cookie_file}

This should work. first is basicly gets the cookie with session id 2nd do the real request

answered Jan 21 '18 at 14:07

Yehuda

457
2
6
16

Send ajax request through CURL

3 Answers3

Linked