How can i follow the links if i don't subclass from crawl spider in scrapy

Asked Dec 11 '12 at 21:48

Active Dec 11 '12 at 22:40

Viewed 137 times

This is my code and my spider is not following links I have sub class from InitSpider

class TestSpider(InitSpider):
    name = 'login'
    allowed_domains = ['example.com']
    start_urls = ['http://www.example.com']
    rules = (Rule(SgmlLinkExtractor(allow=('example\.com', 'PostSearch'),restrict_xpaths = "//dd[@class='nextPage']"), callback='parse_items', follow=True),)

    def parse_items(self, response):

        hxs = HtmlXPathSelector(response)

am using this example https://stackoverflow.com/a/5857202/1894766*

edited May 23 '17 at 12:29

Community

asked Dec 11 '12 at 21:48

user1894766

Are you getting any errors? – Talvalin Dec 12 '12 at 01:23
i don't get any error but it don't follow any links except the start url given – user1894766 Dec 12 '12 at 10:19
Which website are you crawling and how are you running the crawl? `scrapy crawl someurl`? – Talvalin Dec 12 '12 at 10:42
it can be any website , can u just give me simple example where where u subclass from `initSpider` and it follows link. if i subclass from crawlspider then its working fine. i am crawling amazon by scrapy crawl myspidername – user1894766 Dec 12 '12 at 14:04

How can i follow the links if i don't subclass from crawl spider in scrapy

0 Answers0