PHP cURL how to add the User Agent value OR overcome the Servers blocking cURL requests?

Question

I am transferring an Object Array. I have a cURL client (submitter) on own Server and listening script on other's Server, which one is not under my control. Then i think there, they are blocking the incoming cURL requests because when i test with the normal HTML <form>, it is working. But not via cURL anyway.

So i think they have done some restriction to cURL.

Then my questions here are:

Can a Server restrict/block the cURL incoming requests?
If so, can i trick/change the HTTP Header (User Agent) in my initiating cURL script?
Or is there any other possible stories?

Thanks!

score 55 · Answer 1 · edited Apr 01 '14 at 04:14

IF you are still facing the problem then do the following.

1.

$config['useragent'] = 'Mozilla/5.0 (Windows NT 6.2; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0';

curl_setopt($curl, CURLOPT_USERAGENT, $config['useragent']);
curl_setopt($curl, CURLOPT_REFERER, 'https://www.domain.com/');

2.

$dir                   = dirname(__FILE__);
$config['cookie_file'] = $dir . '/cookies/' . md5($_SERVER['REMOTE_ADDR']) . '.txt';

curl_setopt($curl, CURLOPT_COOKIEFILE, $config['cookie_file']);
curl_setopt($curl, CURLOPT_COOKIEJAR, $config['cookie_file']);

NOTE: You need a COOKIES folder in directory.

3.

curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);

If doing these don't solve the problem then Give the Sample Input/Output/Error/etc. So, that more precise solution can be provided.

score 36 · Answer 2 · answered Jul 23 '13 at 03:42

36

  $agent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)';
  $curl=curl_init();
  curl_setopt($curl, CURLOPT_USERAGENT, $agent);

answered Jul 23 '13 at 03:42

srain

8,944
6
30
42

score 7 · Accepted Answer · answered Jul 23 '13 at 03:57

7

In the server side, we can block some requests by recognize the header fields(including refer, cookie, user-agent and so on) in http request, the ip address, access frequency. And in most case, requests generated by machine usually has something different than human requests,for example, no refer & cookie, or with higher access frequency, we can write some rules to deny these requests.
According to 1, you can try your best to simulate real requests by filling the header fields, using random and slower frequency, using more ip addresses. (sounds like attack)
Generally, using lower frequency and do not make heavy load for their server, follow their access rules, they will seldom block your requests.

answered Jul 23 '13 at 03:57

TroyCheng

571
3
10

1

Yes, for most middle and small level traffic, it's usually not worth to do this. But for those site with heavy load, I saw many kinds of methods to filter the traffic. – TroyCheng Jul 23 '13 at 04:02
Ya mine is on some third-party Cloud provider. So i'm sure they will be doing this. So for me, what even thing should i ask them to check? :( – 夏期劇場 Jul 23 '13 at 06:34
Just describe your problem and ask them about the filter rules and how to avoid being denied. – TroyCheng Jul 23 '13 at 06:47

score 1 · Answer 4 · answered Jul 23 '13 at 03:52

1

Server cannot block only cURL requests because they are just HTTP requests. So changing User Agent of your cURL can solve your problem, as server will think you are connecting through browser presented in UA.

answered Jul 23 '13 at 03:52

AxelPAL

1,047
3
12
19

score 1 · Answer 5 · answered Jan 23 '21 at 21:11

Example of curl GET call in php. ftp file in a variable. The solution was on Stackoverflow... where ?!? not mine.

BTW, you need to be able to execute php code from within html modify your /etc/apache2/mods-enabled' edit '@mime.conf if you want to do so... Go to end of file and add the following line:

"AddType application/x-httpd-php .html .htm" BEFORE tag '< /ifModules >' verified and tested with 'apache 2.4.23' and 'php 5.6.17-1' under 'debian'

I choose to execute php in html file because faster development.

example code begin :

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <title></title>
  </head>
  <body>

  <?php

  $host = "https://tgftp.nws.noaa.gov/data/observations/metar/decoded/CYHU.TXT";
  $agent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)";

  $curl = curl_init();
  curl_setopt($curl, CURLOPT_URL, $host);
  curl_setopt($curl, CURLOPT_USERAGENT, $agent);
  curl_setopt ($curl, CURLOPT_RETURNTRANSFER, 1) ;
  curl_exec($curl);
  $ftp_result=curl_exec($curl);

  print_r($ftp_result);

  //and the big work commencing,
  //extracting text ...

  $zelocation="";
  $zedatetime="";
  $zewinddirection="";
  $zewindspeed="";
  $zeskyconditions="";
  $zetemp="";
  $zehumidity="";

  ?>

  </body>
</html>

score 1 · Answer 6 · answered Jan 24 '21 at 12:49

I've faced the same issue when I was trying login to a website using CURL, the server was rejecting my request until I've sent the user-agent header and the cookies returned when entering the login page, however, you can use this curl library if you don't familiar with curl.

$curl = new Curl();

$curl->setHeaders('user-agent', 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0');

// Disable SSL verification
$curl->setOpt(CURLOPT_SSL_VERIFYPEER, '0');

$curl->post($url, $data);

$response = $curl->getRawResponse();

PHP cURL how to add the User Agent value OR overcome the Servers blocking cURL requests?

6 Answers6

Linked

Related