0

I have an ajax call that gets many, many pages of contents from a website, but it fails to open steams sometimes even though the same exact links that failed to open streams works just fine, when I ran them again. The problem is, then other links would failed to open stream. Anyone know why this is happening?

Warning: file_get_contents(http://www.merriam-webster.com/browse/dictionary/a/acoustically.htm): failed to open stream: HTTP request failed!
Vic Seedoubleyew
  • 9,888
  • 6
  • 55
  • 76
frosty
  • 2,559
  • 8
  • 37
  • 73
  • 1
    Maybe a problem with their network; the server's connection you are executing the code on; or they limit the number of requests. – chris85 Nov 26 '15 at 02:13
  • @chris85 Can I fix this by using a while loop, that keep trying to get their contents when there's not a match? – frosty Nov 26 '15 at 02:14
  • Yea, you could use a `while` to execute it until you get a response although if there server is down you will enter an infinite loop. – chris85 Nov 26 '15 at 02:15
  • I made a sample code to test this and 46 out of 500 requests to that page failed. It didn't seem to be a fixed number of requests but could have been a time filter (e.g. 2 requests a seconds per IP). – chris85 Nov 26 '15 at 02:17
  • @chris85 Haha. That explains it then. I limited my ajax callback to 100 milliseconds! – frosty Nov 26 '15 at 02:23
  • Yea, looks like that is against their TOS, might run into issues in the future too. `You may not use data mining, robots, screen scraping, or similar data gathering and extraction tools on the Services, except with our express written permission.` or this is what I see it as happening I'll leave this up to another user though that perhaps has more experience with this. – chris85 Nov 26 '15 at 02:26
  • @chris85 Well, looks like it's time to use that while loop. – frosty Nov 26 '15 at 02:29
  • Possible duplicate of [file\_get\_contents - failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found](https://stackoverflow.com/questions/9574130/file-get-contents-failed-to-open-stream-http-request-failed-http-1-1-404-not) – Amaan Iqbal Jul 02 '17 at 19:17

0 Answers0