Combining Scrapy with CasperJS

Question

When I try Splash by using Scrapy-Splash, it doesn't support navigating like a real browser. It only renders the HTML, but when it clicks a button, it doesn't render the next page it navigates to.

From my simple research, web scraping with navigation is only supported by CasperJS. But, is it possible to combine Scrapy and CasperJS

But from what I know Selenium can't simulate to navigation URL when a button is clicked. That's why I consider CasperJS — Aminah Nuraini, Mar 01 '17 at 12:19
I mean, when a button is clicked, it will trigger all of the request the page would made in the browser and then open the next page the will be opened just like in the real browser — Aminah Nuraini, Mar 01 '17 at 12:47

score 1 · Accepted Answer · edited May 23 '17 at 12:31

1

You're confusing browser with testing frameworks, Selenium does exactly that but instead of Firefox, use PhantomJS as a headless browser (which is what CasperJS uses).

Click method

# Assume the button has the ID "submit" :)
driver.find_element_by_id("submit").click()

alexce answer on scrapy with selenium

If you need the content of the requests, use something like BrowserMob as a proxy.

edited May 23 '17 at 12:31

Community

1
1

answered Mar 01 '17 at 13:01

Rafael Almeida

5,142
2
20
33

Is it possible to do it using Splash? Splash has Scrapy Splash which is an official library to combine Splash with Scrapy – Aminah Nuraini Mar 01 '17 at 13:04
1

Depends on the complexity of the problem, if you need to return elements as you navigate a web page not really. If you need to navigate to somewhere and return elements from a single page it's possible and recommended but the scripting for it is in LUA. – Rafael Almeida Mar 01 '17 at 13:08

Combining Scrapy with CasperJS

1 Answers1