0

I started project in django & scrapy. On the homepage I want to user can input model of car (I add car details to link to select right listing for scraping) and click button and the extraction process start and load data to django db.

Can you help me how can I do it, is it possible? How make this button? Thanks

polko
  • 59
  • 4

1 Answers1

0

I think that you can use the CrawlerProcess for this thing. CrawlerProcess documentation

All you have to do is to run the python function by clicking the button. I'm attaching the simple function here, hope it'll help you.

from scrapy.crawler import CrawlerProcess
from your_scraper_project.spiders.spider_file import spiderClass
# if your project name is carScraper and spider file is named car_spider and class is carSpider
# from carScraper.spiders.car_spider import carSpider

# 
def run_car_scraper(car_model):
    process = CrawlerProcess()
    process.crawl(carSpider, first=car_model) # Assuming that you are passing the argumment of the car_model to scrape specific models
    process.start() # This will start your spider

After the spider saves the data in your database or from wherever you are taking the data to display, you can fetch the data and return it from the function.

if you want to save the scraped data in the variable this answer may be useful to you. How to save the data from a scrapy crawler into a variable?

keval sakhiya
  • 21
  • 3