3

Here is my folder layout:

  • my_django_project
    • project
      • project
        • __init__.py
        • settings.py
        • urls.py
        • wsgi.py
      • people
        • management
          • __init__.py
          • commands
            • __init__.py
            • scrapy.py
        • migrations
        • __init__.py
        • admin.py
        • models.py
        • tests.py
        • views.py
      • scrapy_project
        • scrapy_project
          • spiders
            • __init__.py
            • my_scraper.py
          • __init__.py
          • items.py
          • pipelines.py
          • settings.py

And what I have been doing is following this tutorial: Access django models inside of Scrapy

And I followed everything and I still can't do much about it. Every time I run python manage.py scrapy crawl my_scraper it gives me the same error:

ImportError: No module named scrapy_project.settings

Now I do import the settings in Django's settings.py:

import os

os.environ['SCRAPY_SETTINGS_MODULE'] = 'scrapy_project.settings'

I have tried different approaches. Now I have been able to connect scrapy with Django models, but I want to be able to run the scrapy project from Django.

Here is my scrapy.py in the commands folder:

from __future__ import absolute_import
from django.core.management.base import BaseCommand

class Command(BaseCommand):
    def run_from_argv(self, argv):
        self._argv = argv
        self.execute()

    def handle(self, *args, **options):
        from scrapy.cmdline import execute
        execute(self._argv[1:])
Community
  • 1
  • 1
Nazariy
  • 717
  • 6
  • 23

1 Answers1

4

As far as I understand, you need to move the inner scrapy_project package one level up.

alecxe
  • 462,703
  • 120
  • 1,088
  • 1,195
  • 1
    Basically I moved what was in the **first scrapy_project folder** to the top so that it is under the **second project** folder. Thank You! – Nazariy Jan 23 '15 at 03:22