3

I'm trying to retrieve the HTML for a web site. When I view the page in Chrome I see a form with dates and rate options. But when I request the site with RoboBrowser I see a short HTML response which is completely different. What I am doing wrong?

My code:

from robobrowser import RoboBrowser
browser = RoboBrowser(parser='html.parser')

browser.open('http://www.marriott.com/reservation/availabilitySearch.mi?propertyCode=ATLRZ')

print(browser.parsed)

Result:

<!DOCTYPE doctype html>

<html>
<head><script src="/common/js/marriottCommon.js" type="text/javascript"> </script>
<meta charset="utf-8">
</meta></head>
<body>
<script>
        var xhttp = new XMLHttpRequest();
        xhttp.addEventListener("load", function(a,b,c){
          window.location.reload()
        });
        xhttp.open('GET', '/reservation/availabilitySearch.mi?istl_enable=true&istl_data', true);
        xhttp.send();
      </script>
</body>
</html>
Casey
  • 2,611
  • 6
  • 34
  • 60

1 Answers1

3

Try to imitate what a javascript-running browser would do. It looks like the site is reloading the page when it loads. That implies it might not let you access it unless you have the right cookies. The xhr request might be an additional check to prevent bots. So use browser.session to keep the cookies, send the xhr a request and then try to reload the page with your new cookies.

kmaork
  • 5,722
  • 2
  • 23
  • 40
  • Thank you. I opened sent a request to the XHR link in between my other requests. That got me step further, but now I get a 'your session has timed out' page as a result. – Casey Dec 23 '16 at 12:41
  • are you using `browser.session` for all your requests? – kmaork Dec 23 '16 at 12:43
  • Yes because every instance of RoboBrowser starts with requests.Session() as shown [here](http://robobrowser.readthedocs.io/en/latest/_modules/robobrowser/browser.html#RoboBrowser) – Casey Dec 23 '16 at 12:46
  • well, open the link (http://www.marriott.com/reservation/availabilitySearch.mi?propertyCode=ATLRZ) in your browser. This isn't your code's fault, but the url isn't valid anymore. – kmaork Dec 23 '16 at 12:51
  • After loading it once I loaded it again and it worked. Try doing the same with your request. – kmaork Dec 23 '16 at 12:53
  • Try going to the url, open the network tab in chrome developer tools, switch preserve log on, then right click the refresh button and choose empty cache and hard reload. Then monitor the communication between the browser and the server and imitate the exact behavior. – kmaork Dec 23 '16 at 12:57
  • Hi @Ni., I am trying to solve the exact same problem that casey is and running into the exact same issue. However, I am using the python requests library to scrape the webpage. What do you mean by 'loading it once I loaded it again'? Does that mean you sent the same `get` request twice? Can you take a look at my question I asked with all the details about what I have tried: http://stackoverflow.com/questions/42406115/bypassing-script-response-when-scraping-website-with-requests-beautifulsoup. – Chirag Feb 23 '17 at 17:14