0

is there a way i can write test cases for my scrapy spider i written that gives ouptut eg:[ {"company_name": "A + Communications and Security", "source_url": "/company/a--communications-and-security"}, {"company_name": "A&A Technology Group", "source_url": "/company/a-a-technology-group"}]

import scrapy

class CompanySpider(scrapy.Spider):
    name = 'company'
    start_urls = ['https://www.adapt.io/directory/industry/telecommunications/A-1']
    custom_settings = {"TELNETCONSOLE_ENABLED" : False,"ROBOTSTXT_OBEY" : False}
    
    def parse(self,response):
        for company in response.xpath("//div[contains(@class,'DirectoryList_link')]"):
            yield{
                'company_name' : company.xpath("./a/text()").get(),
                'source_url' : company.xpath("./a/@href").get().split('https://www.adapt.io')[-1]
            }
grroom model
  • 31
  • 5

1 Answers1

0

You could try using Spiders Contracts or local fake responses which reflect the state of the website the moment you mapped it.

How to work with the scrapy contracts?

João Vitor Lopes
  • 1
  • 1
  • 1