21

I am working on ZOHO API and trying to update the record using cURL. I tried different cURL variations, but it always returns "false". But when I call the same URL using a browser, it works.

Is there any way they can block cURL requests? Is there any other way I can call that URL using a POST or maybe a GET request?

The cURL code I have tried is as below:

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, 0);
$data = curl_exec($ch);
curl_close($ch);
elixenide
  • 44,308
  • 16
  • 74
  • 100
www.amitpatil.me
  • 3,001
  • 5
  • 43
  • 61

4 Answers4

23

Many web servers want to block HTTP requests forged by something else than a browser, to prevent bots abuses. If you want to simulate/pretend your request from a browser, you at least have to:

  1. Pass the exact same headers than your browsers (use ie Firebug to get them)

    curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

  2. Change the user agent (name of the browser)

    curl_setopt($ch, CURLOPT_USERAGENT, $agent);

  3. Enable cookies (for eg redirection and session handling)

    curl_setopt ($ch, CURLOPT_COOKIEJAR, $file);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);

  4. Add referers 

    curl_setopt($curl, CURLOPT_REFERER, 'http://www.google.com');
curl_setopt($curl, CURLOPT_AUTOREFERER, true);

And pray you haven't missed anything!

Tom Desp
  • 911
  • 6
  • 19
19

Servers cannot block cURL requests per se, but they can block any request that they do not like. If the server checks for some parameters that your cURL request does not satisfy, it could decide to respond differently.

In the vast majority of cases, this difference in behavior is triggered by the presence (or absence) and values of the HTTP request headers. For example, the server might check that the User-Agent header is present and has a valid value (it could also check lots of other things).

To find out what the HTTP request coming from the browser looks like, use an HTTP debugging proxy like Fiddler or your browser's developer tools.

To add your own headers to your cURL request, use

curl_setopt($ch, CURLOPT_HTTPHEADER, array('HeaderName: HeaderValue'));
elixenide
  • 44,308
  • 16
  • 74
  • 100
Jon
  • 428,835
  • 81
  • 738
  • 806
  • Agree with ur statement "parameters that your curl request does not satisfy" but in there official documentation, they didnt mentioned anything such. One more thing, there API has to be called over HTTPS....CURL manages internally or do i have to add something more to deal with HTTPS ? – www.amitpatil.me Feb 22 '12 at 09:02
  • @AmitPatil: Yes, curl does HTTPS. Can't really speak about the ZOHO API, I 've never even looked at it. – Jon Feb 22 '12 at 09:14
  • *"In the vast majority of cases, this difference in behavior is triggered by the presence (or absence) and values of the HTTP request headers"* Modern request forges blockers are much smarter than that nowadays, imho. – Tom Desp Feb 22 '12 at 09:15
  • @TomDesp: I 'd very much like to read more details on that, especially considering that we 're talking about an API (which means that legitimate requests cannot really be expected to "look human" by *any* criteria). – Jon Feb 22 '12 at 09:35
  • @Jon: I **totally agree** with you considering the API perspective. I'm just saying that headers are not always sufficient. For instance, I already had to enable cookie jars for enabling redirections with temporary tokens sent by the server, stored in the cookies and passed to the redirected location (`CURLOPT_COOKIEJAR` and `CURLOPT_AUTOREFERER`). Anyway, that's the solution I found to the headers-are-not-enough issues I encountered. There may be simpler solutions. – Tom Desp Feb 22 '12 at 11:00
  • @TomDesp: Cookies are also passed on as HTTP headers though. ;-) – Jon Feb 22 '12 at 11:01
  • Guys TomDesp & Jon : i got a solution....added "CURLOPT_SSL_VERIFYPEER" and it worked...There is "invalid data error" from zoho server....but atleast my request has been heard by server...many thanks for looking in to it – www.amitpatil.me Feb 22 '12 at 12:16
  • @Jon: passed to server via headers for sure. But at least you have to tell cUrl to store them while being redirected, right? – Tom Desp Feb 22 '12 at 12:51
  • Cna you provide with the example of array('Headername: HeaderValue')? what shoud go inside the array – Joon. P Mar 30 '16 at 04:46
  • Agree. It is mostly the `user-agent`. – Dharmendra Yadav Apr 22 '19 at 05:08
5

To answer your question "Is there any way they can block CURL requests?": Yes, in fact one may detect a cURL request by reading the User-Agent header.

You can change the user agent by calling curl_setopt($ch, CURLOPT_USERAGENT, 'My user agent string!');.

Vitamin
  • 1,526
  • 1
  • 13
  • 27
1

Just to elaborate a little more on this, you can use the curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0'); or something like that to fake the user agent. In this case, the server would think a Firefox browser was making the request.

ashton gleason
  • 11
  • 1
  • 1
    Welcome to Stack Overflow! Please note you are answering a very old and already answered question. Here is a guide on [How to Answer](http://stackoverflow.com/help/how-to-answer). – help-info.de Sep 14 '18 at 16:44