Here is my folder layout:
- my_django_project
- project
- project
- __init__.py
- settings.py
- urls.py
- wsgi.py
- people
- management
- __init__.py
- commands
- __init__.py
- scrapy.py
- migrations
- __init__.py
- admin.py
- models.py
- tests.py
- views.py
- management
- scrapy_project
- scrapy_project
- spiders
- __init__.py
- my_scraper.py
- __init__.py
- items.py
- pipelines.py
- settings.py
- spiders
- scrapy_project
- project
- project
And what I have been doing is following this tutorial: Access django models inside of Scrapy
And I followed everything and I still can't do much about it.
Every time I run python manage.py scrapy crawl my_scraper
it gives me the same error:
ImportError: No module named scrapy_project.settings
Now I do import the settings in Django's settings.py
:
import os
os.environ['SCRAPY_SETTINGS_MODULE'] = 'scrapy_project.settings'
I have tried different approaches. Now I have been able to connect scrapy with Django models, but I want to be able to run the scrapy project from Django.
Here is my scrapy.py
in the commands
folder:
from __future__ import absolute_import
from django.core.management.base import BaseCommand
class Command(BaseCommand):
def run_from_argv(self, argv):
self._argv = argv
self.execute()
def handle(self, *args, **options):
from scrapy.cmdline import execute
execute(self._argv[1:])