6

I'm having trouble calling a scrapy spider in my django view. How can I do this? I tried to follow this tutorial http://tryolabs.com/Blog/2011/09/27/calling-scrapy-python-script/ but did not work in the import settings.

Vinnicyus Gracindo
  • 162
  • 1
  • 2
  • 6
  • 1
    Please post the full error message. Thank you. – mechanical_meat Feb 13 '13 at 18:03
  • Use [django dynamic scraper](https://github.com/holgerd77/django-dynamic-scraper). – Nick Feb 13 '13 at 19:10
  • 2
    That script is somewhat out-of-date and scrapy has changed a lot since then. Try out the answer to this question: http://stackoverflow.com/questions/14777910/scrapy-crawl-from-script-always-blocks-script-execution-after-scraping – Talvalin Feb 13 '13 at 21:42

2 Answers2

2

If the error is coming from 

from scrapy.conf import settings

it's likely because scrapy simply can't find the settings file it's expecting. Since it's customary in Django to use settings for django's own configuration, it would be clearest if you don't use that term.

You can specify your scrapy settings within your Django settings:

"""settings.py"""
# stuff

SCRAPY_SETTINGS = {
    ... # put your usual scrapy keys and values here
}

# more stuff

Then, instead of importing scrapy.conf.settings, you can instead use:

from django.conf import settings

and where you reference scrapy settings in your script, you should change the argument to CrawlerProcess to settings.SCRAPY_SETTINGS

self.crawler = CrawlerProcess(settings.SCRAPY_SETTINGS)

If you have any further trouble, please post the full error you're getting and the code for your view.

mgojohn
  • 881
  • 9
  • 15
0

Or, alternatively, you can create the settings object like so:

from scrapy.settings import Settings
settings = Settings()
settings.setmodule('path.to.scrapy.settings', priority='project')
Bobby
  • 6,840
  • 1
  • 22
  • 25