2

I've written a few Scrapy spiders that I want to run inside a django project when the user makes a request. As I was not able to get the spiders to work when they were inside of one project, I separated them into their own projects. I have been able to run a spider from a bash script successfully and thought that I could run that script inside a celery task.

I've seen posts like this but those don't use bash scripts to run the spiders and I also can't tell if they are running multiple spiders (not to mention that the code is probably deprecated at this point). This approach is of course just an idea, so I just want to find out what the proper way is to do this with the current versions of scrapy, django, and celery. Does it make sense to run them individually in a bash script, or is there another way to run multiple spiders within a celery task? I'm not really sure where to start here so any help would be appreciated, thanks.

Community
  • 1
  • 1
loremIpsum1771
  • 2,497
  • 5
  • 40
  • 87
  • why not just fix your imports so that your spider code can be imported and run from the celery tasks? – scytale Jul 20 '15 at 14:58
  • @scytale Thanks, seems like a plausible solution. I found [this SO post](http://stackoverflow.com/questions/11528739/running-scrapy-spiders-in-a-celery-task) which addresses the issue, but the code seems to be deprecated (not to mention the fact that it was a workaround in the first place). Do you think that solution will still work? – loremIpsum1771 Jul 20 '15 at 21:19
  • just try it. some things might have changed - you will need to read the code used in the workrounds in order to understand how it fixes the problem - this will enable you to adapt it to your needs. – scytale Aug 02 '15 at 09:40
  • @scytale I took you suggestion and did try it last week. I posted a question about a problem I had with doing it [here](http://stackoverflow.com/questions/31689545/how-to-tell-if-django-celery-task-is-properly-running-scrapy-spider), thanks. – loremIpsum1771 Aug 02 '15 at 20:01

0 Answers0