Scrapy crawl site with dynamic routing

Question

How can I scrape all tools from site with dynamic routing

http://growthtools.io/social-media-automation-tools

When I was trying to

scrapy shell 'http://growthtools.io/social-media-automation-tools'

I recieved following result

2017-01-07 22:43:06 [root] DEBUG: Using default logger
2017-01-07 22:43:06 [root] DEBUG: Using default logger

In [1]: view(response)

and response object did't contain tools elements.

In [3]: In [2]: response.css('.toolsList')
Out[3]: []
In [5]: 'toolsList' in response.body
Out[5]: False

Who can describe how can I parse http://growthtools.io/social-media-automation-tools and why reponse object did't contain all page content?

The site uses JavaScript to display the page. You should use a headless browser like Splash or PhantomJS to render it first. — Richard Dowinton, Jan 07 '17 at 20:22

score 0 · Accepted Answer · edited May 23 '17 at 10:30

The page load involves JavaScript executed by the browser which Scrapy is not. You can though solve it with scrapy-splash which provides a middleware to use in your Scrapy project. The middleware uses the Splash JS rendering service which you can run through the docker.

As far as testing it in the Scrapy Shell, you can follow this example to run it from the shell.

Works for me:

$ scrapy shell 'http://localhost:8050/render.html?url=http://growthtools.io/social-media-automation-tools' 
In [1]: response.css('.toolsList')
Out[1]: 
[<Selector xpath=u"descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' toolsList ')]" data=u'<div class="col-md-10 col-xs-12 toolsLis'>,
 <Selector xpath=u"descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' toolsList ')]" data=u'<div class="col-md-10 col-xs-12 toolsLis'>,
 <Selector xpath=u"descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' toolsList ')]" data=u'<div class="col-md-10 col-xs-12 toolsLis'>,
 <Selector xpath=u"descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' toolsList ')]" data=u'<div class="col-md-10 col-xs-12 toolsLis'>,
 <Selector xpath=u"descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' toolsList ')]" data=u'<div class="col-md-10 col-xs-12 toolsLis'>,
 <Selector xpath=u"descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' toolsList ')]" data=u'<div class="col-md-10 col-xs-12 toolsLis'>,
 <Selector xpath=u"descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' toolsList ')]" data=u'<div class="col-md-10 col-xs-12 toolsLis'>,
 <Selector xpath=u"descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' toolsList ')]" data=u'<div class="col-md-10 col-xs-12 toolsLis'>,
 <Selector xpath=u"descendant-or-self::*[@class and contains(concat(' ', normalize-space(@class), ' '), ' toolsList ')]" data=u'<div class="col-md-10 col-xs-12 toolsLis'>]

Scrapy crawl site with dynamic routing

1 Answers1