Activating a button within a Python spider program

Question

I am absolutely stuck on this one. I am scraping restaurant URLs from a webpage and there is a button at the bottom to reveal more restaurants. The website button code is below (i believe):

<div id="restsPages">
<a class="next" data-url="https://hungryhouse.co.uk/takeaways/aberdeen-bridge-of-dee-ab10">Show more</a>
<a class="back">Back to top</a>
</div>

It is the "Show more" button i am trying to activate. The url within the "data-url" does not reveal more of the page.

It all seems a bit odd on what do do to activate the button from within the python spider?

The code i am trying to use to make this work is:

import scrapy

from hungryhouse.items import HungryhouseItem
from selenium import webdriver  

class HungryhouseSpider(scrapy.Spider):
    name = "hungryhouse"
    allowed_domains = ["hungryhouse.co.uk"]
    start_urls = ["https://hungryhouse.co.uk/takeaways/westhill-ab10",
                      ]
    def __init__(self):
        self.driver = webdriver.Chrome()

    def parse(self,response):
        self.driver.get(response.url)

        while True:
            next =self.driver.find_element_by_xpath('//*[@id="restsPages"]/a[@class="next"]')
            try:
                next.click()
            except:
                break
        self.driver.close()

.... rest of the code follows

The error i get is: 'chromedriver' executable needs to be in PATH

This was resolved http://stackoverflow.com/questions/40699416/pressing-a-button-within-python-code with reference to the answer at http://stackoverflow.com/questions/29858752/error-message-chromedriver-executable-needs-to-be-available-in-the-path — nevster, Nov 22 '16 at 01:09

score 0 · Accepted Answer · edited May 23 '17 at 10:32

This was resolved at Pressing a button within python code with reference to the answer at Error message: "'chromedriver' executable needs to be available in the path"

But specifically

self.driver = webdriver.Chrome()

needed to change to

self.driver = webdriver.Chrome("C:/Users/andrew/Downloads/chromedriver_win32/chromedriver.exe")

in my case.

i.e. I needed to add the path to chromedriver.exe.

Activating a button within a Python spider program

1 Answers1