Python config ImportError

Question

I am working on a python project and whose directory structure looks like this,

SEC-Edgar
├── SECEdgar
│   ├── __init__.py
│   ├── companylist.txt
│   ├── crawler.py
│   ├── crawler.pyc
│   ├── data.txt
│   └── test.py
├── config.py
├── requirements.txt
└── setup.py

I am trying to use config module inside crawler.py but it is giving an ImportError.

Traceback (most recent call last):
  File "SECEdgar/test.py", line 3, in <module>
    from crawler import SecCrawler
  File "/Users/rahul/Code/SEC-Edgar/SECEdgar/crawler.py", line 9, in <module>
    from config import DEFAULT_DATA_PATH
ImportError: No module named config

The import statement in crawler.py is

 from config import DEFAULT_DATA_PATH

I am not able to understand how import works in python, particularly when it comes to different directories like importing from root to base directory.

Do I need to add __init__.py on root directory also, so that it becomes a package and then use . to import it?

Is there a better way to handle import or am I missing some fundamentals here?

See if this works http://stackoverflow.com/questions/36996391/importing-another-project-as-modules-in-python/36996510#comment61546863_36996510 — Mukund Gandlur, May 03 '16 at 08:52
This may help https://stackoverflow.com/questions/714063/importing-modules-from-parent-folder — Windsooon, May 03 '16 at 09:16
I think you would have to rename your `SEC-Edgar` directory to e.g. `SEC_Edgar` because module names have to follow the Python naming restrictions, which do only allow alphanumeric characters plus the underscore. This and using relative imports. — Byte Commander, May 03 '16 at 09:25
I have added `__init__.py` to SEC-Edgar and also renamed it to `secedgar` to comply with python modules name but still I am getting ImportError. — Rahul, May 03 '16 at 10:07

praba230890 · Answer 1 · 2016-05-05T06:28:12.327

First change the from config import DEFAULT_DATA_PATH to from ..config import DEFAULT_DATA_PATH then add an empty __init__.py in SEC-Edgar directory.

Then, try executing your module as python -m SEC-Edgar.SECEdgar.test or like that for any module you are running, from the location(directory) where SEC-Edgar exists.

There are three things to note here.

your config.py and crawler.py modules doesn't share the same parent directory. So, when you try from config import DEFAULT_DATA_PATH from crawler.py Python can't import it for you.
We can import modules(.py files) from other directories in python only if that directory is a package. It can be done by just adding an empty __init__.py file in that directory.
We can now use relative imports. But, During relative imports Python uses the __name__ property of the module, As per PEP 328:

Relative imports use a module's name attribute to determine that module's position in the package hierarchy. If the module's name does not contain any package information (e.g. it is set to 'main') then relative imports are resolved as if the module were a top level module, regardless of where the module is actually located on the file system.

So, we running it with -m option to supply the package informations for Python to perform relative imports.

[In response to comment] Here is what working for me,

SEC-Edgar
├── SECEdgar
│   ├── __init__.py
│   ├── crawler.py
│   └── test.py
├── __init__.py
└── config.py

I'm outside SEC-Edgar directory,

C:\Users\username\Documents> ls
SEC-EDGAR  OTHERS

C:\Users\username\Documents> python -m SEC-Edgar.SECEdgar.test
Lets assume I'm an SecCrawler

config.py

DEFAULT_DATA_PATH = "some/path"

crawler.py

from ..config import DEFAULT_DATA_PATH

SecCrawler = "Lets assume I'm an SecCrawler"

test.py

from crawler import SecCrawler

print(SecCrawler)

Changing from `config` to `..config` doesn't works. The only solution worked was adding project path to `PYTHONPATH`. — Rahul, May 04 '16 at 07:46
Yes, adding project path to PYTHONPATH will work but you have add the path when you move the project and it's not the right way. http://stackoverflow.com/a/19669773/887007 — praba230890, May 05 '16 at 06:17

score 0 · Answer 2 · edited May 23 '17 at 12:09

0

use from ..config import DEFAULT_DATA_PATH

Refer: Intra-package References

If you get error : Attempted relative import in non-package

Refer : this question

edited May 23 '17 at 12:09

Community

1
1

answered May 03 '16 at 09:24

Ani Menon

27,209
16
105
126

I get value error ```Traceback (most recent call last): File "crawler.py", line 9, in from ..config import DEFAULT_DATA_PATH ValueError: Attempted relative import in non-package``` – Rahul May 03 '16 at 10:05
Create a file`__init__.py` in the parent folder. – Ani Menon May 03 '16 at 10:09
I have created it but still I am getting the error. – Rahul May 03 '16 at 10:24
Check : [packaging for relative import in python](http://stackoverflow.com/questions/4348452/python-packaging-for-relative-imports) – Ani Menon May 03 '16 at 10:26

score 0 · Answer 3 · edited May 23 '17 at 10:31

Your code is fine, the issue is caused by Python path.

try to answer this question: how/where can python find the module called 'config'?

So, I suggest you find these steps to understand and solve the issue:

reduce the scope in your crawler.py, use this main:

if __name__ == '__main__':
    import sys
    print(sys.path)

    from config import DEFAULT_DATA_PATH
    print 'DEFAULT_DATA_PATH=%s' % DEFAULT_DATA_PATH

print(sys.path) will show you the python path. you probally cannot find your project root in the path.
if you're clear that you need to manage the python path, you may config it in your IDE. e.g. I'm using pycharm, I'll select the source root in settings/Project Structure and check 'Add source roots to PYTHONPATH' in Run/Debug Configurations.
after you configured it, re-run your crawler.py, the output will be samilar as this: ['C:\\dev\\projects\\SEC-Edgar\\SECEdgar', ... 'C:\\dev\\projects\\SEC-Edgar'.] DEFAULT_DATA_PATH=TEST TEST will be the value in SEC-Edgar\config.py

this is only for dev, for packaing/deployment, you'll need to config your setup.py properly. you may refer to: https://docs.python.org/2/distutils/setupscript.html

Python config ImportError

3 Answers3