1

I have two packages (diretories ) in my Python project

src
/textmining
mining.py...def mining():#...
__init.py....__all__ = ["mining"]
/crawler
crawler.py 
 in crawler.py I use the mining class
 mining=mining()
main.py 
__init__.py

my main.py is as follow:

scrapy_command = 'scrapy runspider {spider_name} -a crawling_level="{param_1}"'.format(spider_name='crawler/crawler.py',
                                                                                   param_1=crawling_level)

   process = subprocess.Popen(scrapy_command, shell=True)

when I run crawler, it prompts runspider: error: Unable to load 'Crawler.py': cannot import name mining

Mehdi
  • 133
  • 1
  • 12
  • Which version of Python are you using? Python 2 and 3 have very different ways of handling package structure. Also, make sure your filenames are correctly spelled. `__init__.py` you've written `__initi.py` in this code. If it's a copy-paste, correct that mistake, if it's not, edit your question to not have that typo. – Gemtastic Feb 19 '17 at 10:20
  • http://stackoverflow.com/q/13437402/3763850 Maybe this question and its answers can help you in regards to scrapy. – Gemtastic Feb 19 '17 at 11:42

1 Answers1

0

You need an __init__.py in every folder that's a part of the same package module.

src
    __init__.py
    /textmining
        __init__.py
        mining.py
    /crawler
        __init__.py
        crawler.py

For simplicity, you should add a main.py in the src folder and call the function you want to start your program with from there as it's fairly difficult to import modules from sibling directories if you start your script in a non-root directory.

main.py

from crawler import crawler

crawler.start_function()

crawler.py

from src.textmining import mining
miner = mining()

Without turning everything into a python module you'd have to import a folder into the current script or __init__.py module by adding to path:

# In crawler.py 
import sys
import os
sys.path.append(os.path.abspath('../textmining'))
import mining

However, messing around with the path requires you to keep in mind what you've done and may not be a thing you desire.

Gemtastic
  • 6,253
  • 6
  • 34
  • 53
  • I have a main.py in my src folder and added a the file __init__.py following your codes but getting the same error – Mehdi Feb 19 '17 at 10:49
  • I updated my answer with examples of what the imports looks like. You need to provide more code in your question because it leads on to assume that you're trying to use normal imports. – Gemtastic Feb 19 '17 at 11:06
  • I get..Unable to load 'crawler/crawler.py': No module named src.textmining – Mehdi Feb 19 '17 at 11:10
  • 1
    Did you remove the `scrapy_command` part in main and make sure you're calling the init files `__init__.py`? – Gemtastic Feb 19 '17 at 11:12
  • I have to call sctapy command as the crawler is a scrapy module..how to call init files as well..I am new to pyhton. – Mehdi Feb 19 '17 at 11:23
  • You don't call the `__init__.py` files, it's an automatic process in python which is why it's important to spell the filenames correctly. If your question is actually about scrapy I recommend you make a new question about that, this question appears to be about imports. I don't know how scrapy imports scrapy files or if it's python 2.7 compatible. I recommend that you leave the scrapy bit out just to see if you can get the imports to work, because to me it looks like scrapy imports out of context. – Gemtastic Feb 19 '17 at 11:39