Scrapy follow javascript input button

Question

I have following input's on the page:

<input name="ct99" value="" id="ct99" class="GetData" type="submit">
<input name="ct92" value="" id="ct92" class="GetData" type="submit">
<input name="ct87" value="" id="ct87" class="GetData" type="submit">

class GetData display some click-able icon. When click on it new page is opened. Some JavaScript taking care of it. How can I follow this?

I'm already try code below just to see if scrapy follow inputs, but without success.

def parse(self, response):
    sel = Selector(response)

    links = sel.xpath("//input[@class='GetData']").extract()
    for data in links:
        yield scrapy.FormRequest.from_response(response,
            formdata={}, callback=self.after_click)


def after_click(self, response):
    url = response.url
    print '\nURL', url

you need pass form url in next request, do you have all html? — LeoCBS, Jan 21 '16 at 15:51

score 3 · Accepted Answer · edited May 23 '17 at 12:31

3

There are two common ways to approach the problem:

using browser developer tools (Network tab), inspect the requests sent when you click a particular button and then simulate this request using scrapy.Request or scrapy.FormRequest
automate a browser using selenium: locate the button and click it, then grab the .page_source and instantiate a Selector instance, see samples here:
- Scrapy with Selenium crawling but not scraping
- How to use selenium along with scrapy to automate the process?

edited May 23 '17 at 12:31

Community

1
1

answered Jan 21 '16 at 16:00

alecxe

462,703
120
1,088
1,195

Excellent answer thank you. I use selenium, but how to iterate over inputs? I have error TypeError: 'WebElement' object is not iterable links = self.driver.find_element_by_xpath("//input[@class='GetData']") – Goran Jan 21 '16 at 17:57
@Goran sure, use `find_elements_by_xpath` (watch the `s`). – alecxe Jan 21 '16 at 17:57
I'm use self.driver = webdriver.Firefox() but it close with error "StaleElementReferenceException: Message: Element not found in the cache - perhaps the page has changed since it was looked up" Any hint? – Goran Jan 21 '16 at 19:43
@Goran please consider making a separate question and provide as many details as you can. Thank you! – alecxe Jan 21 '16 at 20:32

Scrapy follow javascript input button

1 Answers1