I am writing a Scrapy spider that crawls a set of URLs once per day. However, some of these websites are very big, so I cannot crawl the full site daily, nor would I want to generate the massive traffic necessary to do so.
An old question (here) asked something similar. However, the upvoted response simply points to a code snippet (here), which seems to require something of the request instance, though that is not explained in the response, nor on the page containing the code snippet.
I'm trying to make sense of this but find middleware a bit confusing. A complete example of a scraper which can be be run multiple times without rescraping URLs would be very useful, whether or not it uses the linked middleware.
I've posted code below to get the ball rolling but I don't necessarily need to use this middleware. Any scrapy spider that can crawl daily and extract new URLs will do. Obviously one solution is to just write out a dictionary of scraped URLs and then check to confirm that each new URL is/isn't in the dictionary, but that seems very slow/inefficient.
Spider
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor
from cnn_scrapy.items import NewspaperItem
class NewspaperSpider(CrawlSpider):
name = "newspaper"
allowed_domains = ["cnn.com"]
start_urls = [
"http://www.cnn.com/"
]
rules = (
Rule(LinkExtractor(), callback="parse_item", follow=True),
)
def parse_item(self, response):
self.log("Scraping: " + response.url)
item = NewspaperItem()
item["url"] = response.url
yield item
Items
import scrapy
class NewspaperItem(scrapy.Item):
url = scrapy.Field()
visit_id = scrapy.Field()
visit_status = scrapy.Field()
Middlewares (ignore.py)
from scrapy import log
from scrapy.http import Request
from scrapy.item import BaseItem
from scrapy.utils.request import request_fingerprint
from cnn_scrapy.items import NewspaperItem
class IgnoreVisitedItems(object):
"""Middleware to ignore re-visiting item pages if they were already visited
before. The requests to be filtered by have a meta['filter_visited'] flag
enabled and optionally define an id to use for identifying them, which
defaults the request fingerprint, although you'd want to use the item id,
if you already have it beforehand to make it more robust.
"""
FILTER_VISITED = 'filter_visited'
VISITED_ID = 'visited_id'
CONTEXT_KEY = 'visited_ids'
def process_spider_output(self, response, result, spider):
context = getattr(spider, 'context', {})
visited_ids = context.setdefault(self.CONTEXT_KEY, {})
ret = []
for x in result:
visited = False
if isinstance(x, Request):
if self.FILTER_VISITED in x.meta:
visit_id = self._visited_id(x)
if visit_id in visited_ids:
log.msg("Ignoring already visited: %s" % x.url,
level=log.INFO, spider=spider)
visited = True
elif isinstance(x, BaseItem):
visit_id = self._visited_id(response.request)
if visit_id:
visited_ids[visit_id] = True
x['visit_id'] = visit_id
x['visit_status'] = 'new'
if visited:
ret.append(NewspaperItem(visit_id=visit_id, visit_status='old'))
else:
ret.append(x)
return ret
def _visited_id(self, request):
return request.meta.get(self.VISITED_ID) or request_fingerprint(request)
