1

As part of a web crawler using scrapy I am trying to press a "show more" button on a webpage to reveal the rest of the page at https://hungryhouse.co.uk/takeaways/aberdeen-bridge-of-dee-ab10 . It looks like a javescript button but I am unable to see how it works.

The code on the web page is:

<div id="restsPages">
<a class="next" data-url="https://hungryhouse.co.uk/takeaways/aberdeen-bridge-of-dee-ab10">Show more</a>
<a class="back">Back to top</a>
</div>

The url within the "data-url" does not reveal more of the page if I just use that as a url.

I am using selenium and the python code i am using to make this work is (and I am new to coding in general but learning fast):

import scrapy

from hungryhouse.items import HungryhouseItem
from selenium import webdriver  

class HungryhouseSpider(scrapy.Spider):
    name = "hungryhouse"
    allowed_domains = ["hungryhouse.co.uk"]
    start_urls = ["https://hungryhouse.co.uk/takeaways/westhill-ab10",
                  ]
    def __init__(self):
        self.driver = webdriver.Chrome()

    def parse(self,response):
        self.driver.get(response.url)

        while True:
            next =self.driver.find_element_by_xpath('//*[@id="restsPages"]/a[@class="next"]')
            try:
                next.click()
            except:
                break
        self.driver.close()

The error i get is: 'chromedriver' executable needs to be in PATH

Any ideas how I can press this button from within the code?

nevster
  • 371
  • 3
  • 15
  • 1
    http://stackoverflow.com/questions/29858752/error-message-chromedriver-executable-needs-to-be-available-in-the-path – Nf4r Nov 20 '16 at 00:46
  • @Nf4r That worked in part. I am now using self.driver = webdriver.Chrome("C:/Users/andrew/Downloads/chromedriver_win32/chromedriver.exe") and I see a chrome window open up and the button is pressed to reveal the whole page . . . . but then my code only picks up the original page, it is as if it does not pick up anything that has been revealed by pressing the button . . . . I have put the scrapy code to find the items after the next.click() . . . . also, for some reason I get a popup windown telling me "There is no disk in the drive" though that does not appear to stop the button being pressed – nevster Nov 20 '16 at 04:40
  • ok, I got my Code to work - your answer was bang on the money - my code needed a little adjustment. Still have the popup window telling me "There is no disk in the drive" though I suspect that is something unrelated to all this. – nevster Nov 20 '16 at 23:23

0 Answers0