11

The scrapy doc says that:

Scrapy comes with a built-in service, called “Scrapyd”, which allows you to deploy (aka. upload) your projects and control their spiders using a JSON web service.

is there some advantages in comformance use scrapyd?

gnemoug
  • 467
  • 1
  • 5
  • 21

1 Answers1

11

Scrapyd allows you to run scrapy on a different machine than the one you are using via a handy web API which means you can just use curl or even a web browser to upload new project versions and run them. Otherwise if you wanted to run Scrapy in the cloud somewhere you would have to scp copy the new spider code and then login with ssh and spawn your scrapy crawl myspider.

Scrapyd will also manage processes for you if you want to run many spiders in parallel; but if you have Scrapy on your local machine and have access to the command-line or a way to run spiders and just want to run one spider at a time, then you're better off running the spider manually.

If you are developing spiders then for sure you don't want to use scrapyd for quick compile/test iterations as it just adds a layer of complexity.

Steven Almeroth
  • 7,758
  • 2
  • 50
  • 57
  • 2
    a while ago i was actually using celery to control my spiders. What would be the pros and cons of this vs that.. – goh Apr 17 '13 at 09:13
  • 1
    How can I secure that web API such that no port scanning people get to know about my Scrapyd deployment and can not use the same API? I know I am late by 3 years, but I failed to find anything regarding my query. – harkirat1892 Jul 12 '16 at 14:04