PHP file_get_html is not working

Question

I have used simple_html_dom library but i can not get HTML content only for 1 URL but i am getting 503 error. Check my below code.

$base = 'http://www.amazon.com/gp/offer-listing/B001F0M4K8/ref=dp_olp_all_mbc/183-8463780-9861412?ie=UTF8&condition=new';

echo $html = file_get_html($base);

Error : Warning: file_get_contents(http://www.amazon.com/gp/offer-listing/B001F0M4K8/ref=dp_olp_all_mbc/183-8463780-9861412?ie=UTF8&condition=new) [function.file-get-contents]: failed to open stream: HTTP request failed! HTTP/1.1 503 Service Unavailable in D:\xampp\htdocs\webcrawler-amazon\webcrawler-amazon\simple_html_dom.php on line 76

I am stuck here so please help me.

It could be something like they only allow certain user-agents or something else. 503 to me would be a custom generated. a simple file_get_contents fails as well. So you might have to try a few options.. And of course as already suggested depending on the type of request/fetch. — Angry 84, Apr 15 '15 at 09:49
Try to load the content with CURL like in this example: http://stackoverflow.com/questions/18667441/simple-html-dom-file-get-html-not-working-is-there-any-workaround — smiggle, Apr 15 '15 at 09:50

score 2 · Answer 1 · edited Apr 15 '15 at 09:50

2

I think, server just blocks your request, you will not be able to fetch data from it, using simple HTTP requests.

You can try using curl, proxies, or both (there are ready to use solutions for this, like: AngryCurl, or RollingCurl)

edited Apr 15 '15 at 09:50

ʰᵈˑ

11,279
3
26
49

answered Apr 15 '15 at 09:45

Andriy

973
8
13

score 0 · Answer 2 · answered Apr 15 '15 at 09:51

0

I recommand you to do this with cURL : http://php.net/manual/en/book.curl.php

You can use it with PHP or in command line. There is tons of example online.

answered Apr 15 '15 at 09:51

Dr. Z

236
3
19

score 0 · Answer 3 · answered Apr 15 '15 at 09:56

It's the Amazon's anti-bots defense system.

The returned page starts with the following HTML comment:

<!--
        To discuss automated access to Amazon data please contact api-services-support@amazon.com.
        For information about migrating to our APIs refer to our Marketplace APIs at https://developer.amazonservices.com/ref=rm_5_sv, or our Product Advertising API at https://affiliate-program.amazon.com/gp/advertising/api/detail/main.html/ref=rm_5_ac for advertising use cases.
-->

You need to either mimic very well the behaviour of a real customer using a browser or ask them about an approved way to get data from their systems automatically. Using an API is better (and easier) than scrapping web pages, anyway.

score 0 · Answer 4 · answered Jan 12 '18 at 22:19

I am doing the same, they are sending the following to you. Sometimes, you can get by it.

Enter the characters you see below
Sorry, we just need to make sure you're not a robot. For best results, please make sure your browser is accepting cookies.
Type the characters you see in this image:
 

Try different image 
Continue shopping 

Conditions of Use Privacy Policy 
© 1996-2014, Amazon.com, Inc. or its affiliates

PHP file_get_html is not working

4 Answers4

Linked