2

I am new to scrapy and python. I have written a spider which takes start_urls from command line and start scraping. Upto this, everything is fine.

Now, i want to submit start_urls from a web page (Designed using Flask).After clicking on a button, it should trigger the scrapy shell and start scraping websites.

Vasim
  • 257
  • 2
  • 14

1 Answers1

0

This is what scrapyd project can help with. There is a JSON API which you would use to deploy and run your spiders. This way your Web app part would not be blocked by the active web-scraping process:

Scrapyd is an application for deploying and running Scrapy spiders. It enables you to deploy (upload) your projects and control their spiders using a JSON API.

alecxe
  • 462,703
  • 120
  • 1,088
  • 1,195