Scrapy project can't find django.core.management

Question

I'm trying to follow the method here to 'Scrapy' data from the web and simultaneously save that data directly to my Django database using Scrapy's item pipeline.

However, when I try to run scrapy crawl spidername, I'm getting the error:

ImportError: No module named django.core.management

At first I thought it was because my Scrapy project was outside of my Django project folder, but even after I moved the whole project into my Django project folder I kept getting the same error. If I open a python shell inside the Scrapy project folder in its new location (inside my Django project folder), import django.core.management works fine. So what's going on?

EDIT: Some additional info: I'm doing this on a Webfaction server, and the path to my Django project is /home/gchorn/webapps/django_app/django_project. I'm using Django version 1.4.1, Scrapy 0.16 and Python2.7. The layout of the Django+Scrapy project is as follows:

django_project/
    __init__.py
    manage.py
    settings.py
    urls.py
    myproject #folder containing wsgi.py
    app1
    app2
    app3
    templates
    ScrapyProject/
        scrapy.cfg
        ScrapyProject/
            __init__.py
            items.py
            pipelines.py
            settings.py
            spiders/
                __init__.py
                my_spider.py

Just to confirm, are you opening a simple Python interpreter or running "python manage.py shell"? — Talvalin, Feb 04 '13 at 15:18
When you open the shell, is that using the same version of Python (2.7)? — SaiyanGirl, Feb 05 '13 at 22:11
@Dana, yes I'm just typing in `Python2.7` to open the shell. — GChorn, Feb 06 '13 at 07:10
@Talvalin, I get `/usr/local/bin/python` when typing `which python` in the top-level ScrapyProject directory (just inside my top-level Django project directory). — GChorn, Feb 07 '13 at 01:20

score 0 · Answer 1 · answered Sep 19 '15 at 17:16

Try setting this in your Spider's settings.py:

import os
import sys

sys.path.append('/home/gchorn/webapps/django_app')

os.environ['DJANGO_SETTINGS_MODULE'] = 'django_project.settings'

Then you can import your model classes like:

from django_project.app1.models import some_model

Scrapy project can't find django.core.management

1 Answers1

Linked