0

I am recently working on a project on which I need to scrape some data from an external website. It is working on Localhost but stopped working on the live host. I explored on google as well on StackOverflow where people suggested that open PHP curl extension etc but everything already opened because I am doing a lot more scraping on that hosting which is working as cake.

Code is

$url = "https://pakesimdata.com/sim/search-result.php";
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_TIMEOUT, 40);
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POSTFIELDS, "cnnum=3005210911");
echo $html = curl_exec($ch);

When I echo the result with URL https://pakesimdata.com/sim/search.php or https://pakesimdata.com/ I got the result but it is not working when I do the POST request to acquire the result on URL https://pakesimdata.com/sim/search-result.php. It shows me nothing. I also do the error handling but got no luck, gave no error which put me a great headache. I can not grab what's going on and on which section I need to work to get the results.

Muhammad Usama
  • 59
  • 1
  • 7
  • Please include the code where you are POSTing data. Probably there is the problem. – Alexander Dobernig Jan 24 '21 at 07:51
  • @AlexanderDobernig I am doing POST request on ```https://pakesimdata.com/sim/search-result.php``` URL. where get empty response. But got response on ```https://pakesimdata.com/``` URL. – Muhammad Usama Jan 24 '21 at 07:59
  • @AlexanderDobernig I already posted the code. here ```curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_POSTFIELDS, "cnnum=3005210911"); ``` I send the POST request. I am newbie so sorry If I am doing mistake. – Muhammad Usama Jan 24 '21 at 08:05
  • What debugging have you done? Is `cookie.txt` writeable? Is it created? What does [`curl_getinfo()`](https://www.php.net/manual/en/function.curl-getinfo.php) show? How about [`CURL_VERBOSE/CURL_STDERR`](https://stackoverflow.com/questions/3757071/php-debugging-curl)? – Don't Panic Jan 24 '21 at 08:30
  • Try to `ping pakesimdata.com` from your live host to see if you are getting packets back. – farhodius Jan 24 '21 at 08:45
  • @farhodius It gives the response with IP. – Muhammad Usama Jan 24 '21 at 12:53
  • @Don'tPanic every debugging shows an empty response. I tried one by one. – Muhammad Usama Jan 24 '21 at 12:54

1 Answers1

0

It is working on Localhost but stopped working on the live host.

This would suggest two possibilites 1.) Your host does not like outgoing http connections which you rules out already 2.) The remote host does not like your scraping

ad 2.) Can it be they have blocked your IP address or some other mechanism is protecting the page from beeing used by your script as usually operators of services like this do not like to be scraped by bots.

LOCALHOST will probably appear as your own private IP address with probably a dynamic always changing IP address - like an ordinary user.

Your real server will have a fixed IP adress and probably the external website analzes the traffic and blocks ip adresses for abuse if a lot of requests are coming from the same IP.

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
Alexander Dobernig
  • 753
  • 6
  • 21