4

How to do LinkExtractor uses SplashRequest to send a request.
I run this code but send request only to home page, the links of homepage aren't extracted.

Code:

rules = { Rule(
        LinkExtractor(
            restrict_xpaths=('//ul[contains(@class, "nav-list")]'),
        ),
        callback="parse_item",
        process_request='use_splash'
    )
}

def use_splash(self, request):
    return SplashRequest(url=request.url,
                            endpoint="render.json",
                            args={
                                    "wait":1,
                                    "har": 1,
                                    "html": 1,
                            })
def start_requests(self):
    yield SplashRequest("http://www.example.com",
        callback=self.parse_item,
        endpoint="render.json",
        args={
            "wait":1,
            "har": 1,
            "html": 1,
        })

def parse_item(self):
     # Extract item

EDIT
Any comment?

Ricardo
  • 83
  • 1
  • 8
  • I think your question is too broad: the problem probably doesn't involve *both* LinkExtractor **and** ScrapySplash, maybe try to eliminate one and reask the question – Steven Almeroth Apr 18 '19 at 17:21
  • Does this answer your question? [Scrapy CrawlSpider + Splash: how to follow links through linkextractor?](https://stackoverflow.com/questions/45886068/scrapy-crawlspider-splash-how-to-follow-links-through-linkextractor) – Gallaecio Nov 20 '19 at 09:28

0 Answers0