0

I am using file_get_contents() to fetch the contents from a page. It was working perfectly, but suddenly stopped working and started to show the error below:

"Warning: file_get_contents(https://uae.souq.com/ae-en/apple-iphone-x-with-facetime-256gb-4g-lte-silver-24051446/i/): failed to open stream: HTTP request failed! in /home/xxx/xxxx/xxx/index.php on line 6.

So I tried the same code on localserver, it was working perfectly. Then I tried on another server, and it was working perfectly there too. So I contacted the hosting provider, they said the problem is with the url that they may be preventing the access. So I tried another url (https://www.w3schools.com/) and it is getting contents without any error.

Now I am really confused what the problem is. If the problem is with the server, other urls shouldn't have worked. And if the problem is with url, it shouldn't have worked on the second server and local server.

Here is the test code:

<?php
$html= file_get_contents("https://uae.souq.com/ae-en/apple-iphone-x-with-facetime-256gb-4g-lte-silver-24051446/i/");
echo $html;
?>

What is the problem here? Even if the problem is with url or server, why was it working perfeclty earlier?

Abdulla
  • 441
  • 1
  • 9
  • 21
  • Your code is working fine, maybe they have put restrictions to your IP. – M A Nov 15 '18 at 04:22
  • You can always use curl in php, http://php.net/manual/en/book.curl.php – M A Nov 15 '18 at 04:24
  • Possible duplicate of [PHP file\_get\_contents() returns "failed to open stream: HTTP request failed!"](https://stackoverflow.com/questions/697472/php-file-get-contents-returns-failed-to-open-stream-http-request-failed) – M A Nov 15 '18 at 04:25
  • @MA, it is working fine on some servers. Do you mean restrictions to my server IP? – Abdulla Nov 15 '18 at 04:38
  • @MA, tried curl when I started to get this error, getting some issues there as I have never tried curl before – Abdulla Nov 15 '18 at 04:39
  • @MA, The question is different from the linked question as it is working perfectly on other servers. – Abdulla Nov 15 '18 at 04:42

1 Answers1

0

It sounds like that site (souq.com) has blocked your server. The block may be temporary or it may be permanent. This may have happened because you made too many requests in a short time, or did something else that looked "suspicious," which triggered a mechanism that prevents misbehaving robots from scraping the site.

You can try again after a while. Another thing you can try is setting the User-Agent request header to impersonate a browser. You can find how to do that here: PHP file_get_contents() and setting request headers

If your intention is to make a well behaved robot, you should set the User-Agent header to something that identifies the request as coming from a bot, and follow the rules the site specifies in its robots.txt.

Joni
  • 108,737
  • 14
  • 143
  • 193
  • That may be the case, because I had set up a cron job that gets the contents every minute. Does setting request headers resolve it if it is blocked permanently? – Abdulla Nov 15 '18 at 04:47
  • It may help to set the user-agent header to something a real browser uses – Joni Nov 15 '18 at 04:55
  • Can you please explain how to set the User-Agent header so that it looks the request is coming from a bot? – Abdulla Nov 15 '18 at 08:52