1

I am working on scrapy, i am scheduling a spider i had wrote with the following command

curl http://localhost:6800/schedule.json -d project=example -d spider=exampledotcom

I am using this by seeing in scrapy tutorial, but when i run above command i used to get the below error

curl: (7) couldn't connect to host

Why curl is unable to connect to host, is anything wrong in the command ? and also can anyone explain me the concept of why we are using "http://localhost:6800/schedule.json", because finally my intension is to run multiple spiders in a single project.

Shiva Krishna Bavandla
  • 25,548
  • 75
  • 193
  • 313

1 Answers1

0

Scheduling a spider means telling scrapyd to run the spider when it can. Thus your have to run scrapyd, before trying to contact it:

scrapy server

or 

twistd -ny extras/scrapyd.tac
warvariuc
  • 57,116
  • 41
  • 173
  • 227
  • :Thanks i started scrapyd and deployed a project as in tutorial , actually i am creating multiple spiders in a single project and want all spiders at once, so i had seen this post "http://stackoverflow.com/questions/10801093/run-multiple-scrapy-spiders-at-once-using-scrapyd", here every thing is working fine and finally in the code i am getting the results of the "response" as ">" – Shiva Krishna Bavandla Jun 08 '12 at 11:44
  • but nothing happens just giving a response – Shiva Krishna Bavandla Jun 08 '12 at 11:45
  • but the intension of the above code in the post is to run multiple spider and execute spider, in my case spider should write all the data to json file.My spider is working fine and when i used individually its storing to json files, i am trying what if more spider files are there so i had created a custom command for running multiple spiders – Shiva Krishna Bavandla Jun 08 '12 at 11:50
  • here i am only getting response but it seems the spider in allcrawl.py(custom command) i had written is not workong – Shiva Krishna Bavandla Jun 08 '12 at 11:51
  • When you schedule a spider you send a request to scrapyd, and it just returns the result of scheduling - a json answer with job_id and similar. scrapyd starts your spider either immediately or when it finds it appropriate: http://stackoverflow.com/questions/9161724/scrapy-s-scrapyd-too-slow-with-scheduling-spiders – warvariuc Jun 08 '12 at 12:25