Scraping "older" pages with scrapy, rules and link extractors

Question

I have been working on a project with scrapy. With help, from this lovely community I have managed to be able to scrape the first page of this website: http://www.rotoworld.com/playernews/nfl/football-player-news?ls=roto%3anfl%3agnav. I am trying to scrape information from the "older" pages as well. I have researched "crawlspider", rules and link extractors, and believed I had the proper code. I want the spider to perform the same loop on subsequent pages. Unfortunately at the moment when I run it, it just spits out the 1st page, and doesn't continue to the "older" pages.

I am not exactly sure what I need to change and would really appreciate some help. There are posts going all the way back to February of 2004... I am new to data mining, and not sure if it is actually a realistic goal to be able to scrape every post. If it is I would like to though. Please any help is appreciated. Thanks!

import scrapy
from scrapy.contrib.spiders import CrawlSpider,Rule
from scrapy.contrib.linkextractors import LinkExtractor



class Roto_News_Spider2(crawlspider):
    name = "RotoPlayerNews"

    start_urls = [
        'http://www.rotoworld.com/playernews/nfl/football/',
    ]

    Rules = (Rule(LinkExtractor(allow=(), restrict_xpaths=('//input[@id="cp1_ctl00_btnNavigate1"]',)), callback="parse_page", follow= True),)


    def parse(self, response):
        for item in response.xpath("//div[@class='pb']"):
            player = item.xpath(".//div[@class='player']/a/text()").extract_first()
            position= item.xpath(".//div[@class='player']/text()").extract()[0].replace("-","").strip()
            team = item.xpath(".//div[@class='player']/a/text()").extract()[1].strip()
            report = item.xpath(".//div[@class='report']/p/text()").extract_first()
            date = item.xpath(".//div[@class='date']/text()").extract_first() + " 2018"
            impact = item.xpath(".//div[@class='impact']/text()").extract_first().strip()
            source = item.xpath(".//div[@class='source']/a/text()").extract_first()
            yield {"Player": player,"Position": position, "Team": team,"Report":report,"Impact":impact,"Date":date,"Source":source}

Answer 1

My suggestion: Selenium

If you want to change of page automatically, you can use Selenium WebDriver. Selenium makes you to be able to interact with the page click on buttons, write on inputs, etc. You'll need to change your code to scrap the data an then, click on the older button. Then, it'll change the page and keep scraping.

Selenium is a very useful tool. I'm using it right now, on a personal project. You can take a look at my repo on GitHub to see how it works. In the case of the page that you're trying to scrap, you cannot go to older just changing the link to be scraped, so, you need to use Selenium to do change between pages.

Hope it helps.

Answer 2

No need to use Selenium in current case. Before scraping you need to open url in browser and press F12 to inspect code and to see packets in Network Tab. When you press next or "OLDER" in your case you can see new set of TCP packets in Network tab. It provide to you all you need. When you understand how it work you can write working spider.

import scrapy
from scrapy import FormRequest
from scrapy.contrib.spiders import CrawlSpider,Rule
from scrapy.contrib.linkextractors import LinkExtractor



class Roto_News_Spider2(CrawlSpider):
    name = "RotoPlayerNews"

    start_urls = [
        'http://www.<DOMAIN>/playernews/nfl/football/',
    ]

    Rules = (Rule(LinkExtractor(allow=(), restrict_xpaths=('//input[@id="cp1_ctl00_btnNavigate1"]',)), callback="parse", follow= True),)


    def parse(self, response):
        for item in response.xpath("//div[@class='pb']"):
            player = item.xpath(".//div[@class='player']/a/text()").extract_first()
            position= item.xpath(".//div[@class='player']/text()").extract()[0].replace("-","").strip()
            team = item.xpath(".//div[@class='player']/a/text()").extract()[1].strip()
            report = item.xpath(".//div[@class='report']/p/text()").extract_first()
            date = item.xpath(".//div[@class='date']/text()").extract_first() + " 2018"
            impact = item.xpath(".//div[@class='impact']/text()").extract_first().strip()
            source = item.xpath(".//div[@class='source']/a/text()").extract_first()
            yield {"Player": player,"Position": position, "Team": team,"Report":report,"Impact":impact,"Date":date,"Source":source}

        older = response.css('input#cp1_ctl00_btnNavigate1')
        if not older:
            return

        inputs = response.css('div.aspNetHidden input')
        inputs.extend(response.css('div.RW_pn input'))

        formdata = {}
        for input in inputs:
            name = input.css('::attr(name)').extract_first()
            value = input.css('::attr(value)').extract_first()
            formdata[name] = value or ''

        formdata['ctl00$cp1$ctl00$btnNavigate1.x'] = '42'
        formdata['ctl00$cp1$ctl00$btnNavigate1.y'] = '17'
        del formdata['ctl00$cp1$ctl00$btnFilterResults']
        del formdata['ctl00$cp1$ctl00$btnNavigate1']

        action_url = 'http://www.<DOMAIN>/playernews/nfl/football-player-news?ls=roto%3anfl%3agnav&rw=1'

        yield FormRequest(
            action_url,
            formdata=formdata,
            callback=self.parse
        )

Be carefull you need to replace all to corrent one in my code.

Answer 3

If your intention is to fetch the data traversing multiple pages, you don't need to go for scrapy. If you still want to have any solution related to scrapy then I suggest you opt for splash to handle the pagination.

I would do something like below to get the items (assuming you have already installed selenium in your machine):

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("http://www.rotoworld.com/playernews/nfl/football/")
wait = WebDriverWait(driver, 10)

while True:
    for item in wait.until(EC.presence_of_all_elements_located((By.XPATH,"//div[@class='pb']"))):
        player = item.find_element_by_xpath(".//div[@class='player']/a").text
        player = player.encode() #it should handle the encoding issue; I'm not totally sure, though
        print(player)

    try:
        idate = wait.until(EC.presence_of_element_located((By.XPATH, "//div[@class='date']"))).text
        if "Jun 9" in idate: #put here any date you wanna go back to (last limit: where the scraper will stop)
            break
        wait.until(EC.presence_of_element_located((By.XPATH, "//input[@id='cp1_ctl00_btnNavigate1']"))).click()
        wait.until(EC.staleness_of(item))
    except:break

driver.quit()