0

If I am using the next button given on the bottom of the webpage it opens a different webpage than opening it by copying the URL in href attribute of its CSS selector on a new tab

I am making a scraper using scrapy. This is the start URL:

https://www.flipkart.com/search?q=laptops&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off.

Spider is working ok till page 18. From page 18 it is going to a different web page than it should go using the next button. TO check the problem i manually tried to open the webpage using the link in href but landed on a different webpage.

Following is the code to open crawl the next webpage-

if Spider_king.count<max_laptops:
        Spider_king.page_number+=1

        if Spider_king.page_number==2:
            Spider_king.css_selector='a._3fVaIS'
        else:
            Spider_king.css_selector='a._3fVaIS:nth-child(12)'
        next_page=response.css(Spider_king.css_selector+'::attr(href)').get()

        yield response.follow(next_page,callback=self.parse)
cronoik
  • 15,434
  • 3
  • 40
  • 78
Aviral Agarwal
  • 1
  • 2
  • Might worth reading https://stackoverflow.com/help/minimal-reproducible-example – Marcos May 27 '19 at 23:38
  • You might also want to check https://stackoverflow.com/q/8550114/939364 . If JavaScript is not the issues, cookies are the likely culprit. – Gallaecio May 28 '19 at 07:56

0 Answers0