4

Ok, let's give you an example (original question below).

I have made a package called "myproject" which shows the problem I'm facing.

  1. Download the package here: https://dl.dropbox.com/u/11013311/myproject.zip
  2. Install the package (eg.: sudo python setup.py develop)
  3. open an Ipython session and type: "import myproject"
  4. if you start typing "myproject." and hit tab twice to get the available methods/values..., you'll see "myproject.myproject" which himself shows the same problem.

Is one of you guys able to explain me what I'm doing wrong? The answer from Andrew Alcock do not help me to fix it, or at least I don't see where could be the problem.

Of course, with such a simple example, there is no need for the package __init__.py file to be that complex, but in my real project I do need COOL instance to be accessible via myproject.COOL.

Thanks for your answers!

EDIT: I've awarded utapyngo the bounty because his solution is effective and I've learnt something more (relative imports with deep submodules). But I'd like to thanks Andrew and nehz for their answers (nehz also provided a solution that fixed my problem, but very subjectively I find it 'less beautiful' ; and Andrew provided useful advices). Too bad I can't share the bounty.

**

ORIGINAL QUESTION:

**

I'm not sure I've phrased correctly the question. I have created a large code with a number of sub-packages, and for simplicity let's call it 'CODE'.

The problem is: 'CODE' appears in the namespace so I can have CODE.CODE or CODE.CODE.CODE, etc... an infinite number of times which looks strange to me and is probably (I guess) a hint that something is wrong (although the code works perfectly without warning).

I guess the problem has something to do with my __init__.py and the structure of my code, so I give some more informations here.

Simplified code structure:

CODE
  | __init__.py
  | tools
     | __init__.py
     | mytools.py
  | other
     | __init__.py
     | init.py
  | sub
     | __init__.py
     | module.py

File: __init__.py (the first one, at the root of CODE)

import CODE.tools.mytools as MyTools
import CODE.other.init

OBJ = CODE.other.init.function()
...

The file mytools.py do not import OBJ from CODE, or any other module that may import OBJ. init.py may import modules such as mytools.py. Finally, modules like module.py may import either mytools.py or OBJ (from CODE). Usually, all import are made with absolute import, such as in: from CODE.sub.module import func.

Does anyone have an explanation for such a behavior? I did not find any related question on SO, but it may be due to the wrong phrasing of mine.

mhavel
  • 735
  • 1
  • 7
  • 19
  • factor the shared part out is the best answer... – Joran Beasley Jul 16 '12 at 23:57
  • I downloaded the code and ran the python setup.py develop. However, I could not reproduce the problem in my environment, specifically from the command line and from within PyCharm. Specifically, Python fails when the statement "import myproject.myproject" is issued. I don't have any experience in lpython, but could the problem be lpython itself? – Andrew Alcock Aug 31 '12 at 00:35
  • @Andrew, I reproduced it both in IPython and in regular python: >>> import myproject; myproject.myproject – utapyngo Aug 31 '12 at 11:54
  • @Andrew, strange... as utapyngo, I do have the problem in both regular python interpreter, and IPython. Though I've never tested the problem inside a script. – mhavel Aug 31 '12 at 19:11
  • @Andrew, yep, I've just checked within a script, and I do get an error when importing `myproject.myproject`. Maybe it is only related to python interpreter only, but why? – mhavel Aug 31 '12 at 19:16
  • I am really confused now. My environment is a virtual environment (virtualenv) using Python 2.7 on Mac. I have tried reproducing this by starting Python in various project directories, modifying PYTHONPATH and *still* not been able to reproduce it. Can you try starting the Python environment in, say, your home directory, then in the project directory? Have you set $PYTHONPATH? Can you also send the value of sys.path that Python reports (import sys \n print sys.path \n)? – Andrew Alcock Sep 01 '12 at 02:40
  • @Andrew First I have to tell you that I'm also using a Mac with Python 2.7 (but the problem is the same on my linux machine). I'm not using a virtual environment though. I tried to start IPython both in my home dir. (as usual), and from the project dir., and the problem is present in both cases. I do not set $PYTHONPATH. And here is my sys.path: https://dl.dropbox.com/u/11013311/sys.path.txt – mhavel Sep 01 '12 at 03:01
  • The path looks fine. On a wildly unlikely off-chance: What is located at /Users/mhavel/Documents/working/SET/trunk? Do you have a myproject there as well? – Andrew Alcock Sep 01 '12 at 03:11
  • @Andrew, SET is actually the code I'm developing, in which I have this problem. But I've created the package `myproject` as an example of the problem, without any link between those twos package (they share nothing, not even a file name). – mhavel Sep 01 '12 at 04:01

3 Answers3

3

The problem is that the package you have imported (basically the top level __init__.py file) contains a reference to itself (the module) in and amongst the other imports. Since the module now has a namespace with itself in it, you can now accesses it as CODE.CODE. ... .CODE.MyTools.

I would suggest:

1) Have an __init__.py in each of your subpackages (tools, other)

2) Within CODE's modules, do not 'import CODE' or any subpackage. Instead make direct imports of the modules (files) you are interested in.

For example, in CODE.sub.module.py:

Do not:

import CODE.other     # "other" is a package (a directory)

Do:

import CODE.other.init    # "init" is a module (a file)

This way sanity lies.

EDIT: Reframing your specific example

The file mytools.py do not import OBJ from CODE, or any other module that may import OBJ. init.py may import modules such as mytools.py.

OK

Finally, modules like module.py may import either mytools.py or OBJ (from CODE).

This is your problem. Don't import CODE here. If you need to simplify the rather long "CODE.other.init.function", you can do it with the from .. import statement:

> from CODE.other.init import function as OBJ 

But be aware that many Pythonites dislike this as it leads to confusion.

Usually, all import are made with absolute import, such as in: from CODE.sub.module import func.

OK

Andrew Alcock
  • 19,401
  • 4
  • 42
  • 60
  • Thanks for your answer. First, I do have `__init__.py`files in each sub-package (I just did not put them in the simplified structure I given you, my bad). Then I think I do not have any import of the main package itself (CODE) or any sub-package. As I said, I use absolute import, but I'm gonna definitely check twice since you're saying this is likely the cause of my problems :) And finally, I did not understand your second edited point. `OBJ` is here the results of the function `function()`, not a renaming. Thanks a lot for your input! – mhavel Jul 17 '12 at 02:08
  • I think we're on to something. All names declared in \_\_init\_\_.py are visible to the importer of that package - so OBJ is visible to anyone who imports CODE. This is often used by frameworks to place key global objects into the scope of the caller. See http://stackoverflow.com/questions/1383239/can-i-you-use-init-py-to-define-global-variables – Andrew Alcock Jul 17 '12 at 02:23
1

Just delete the reference after you use an absolute import:

import atexit

import myproject.mymodule_one.suby as SubY
obj = SubY.myclass()
import myproject.mymodule_two.initialization

COOL = myproject.mymodule_two.initialization.create_cool()
del myproject

EDIT: FYI deleting the reference won't unload the module, and the instance in COOL is still valid.

nehz
  • 2,172
  • 2
  • 23
  • 36
  • Thanks a lot for your answer. It is working well, but is it really pythonic to delete the reference like that? – mhavel Aug 31 '12 at 18:45
  • 1
    yes because you can always reference it again by import. The module is still loaded as you can see in sys.modules[] so there is no performance hit. – nehz Sep 01 '12 at 01:55
1

In your main __init__.py you don't have to and even should not reference your module name (who knows, you would probably want to rename it in future).

Just replace

import myproject.mymodule_...

with

import mymodule_...

Now there is no myproject.myproject:

>>> dir(myproject)
['COOL', 'SubY', '__builtins__', '__doc__', '__file__', '__name__', '__package__',
 '__path__', '__version__', 'atexit', 'exit_report', 'mymodule_one', 'mymodule_two',
 'obj', 'toto']
utapyngo
  • 6,946
  • 3
  • 44
  • 65
  • Thanks a lot, it is working great. You're saying that referencing to the module name is not a good idea, which seems fine. But doing so, how do I use the COOL object elsewhere in the package (let's say `mymodule_one` has a file called `math.py`which makes use of `myproject.COOL`. How would you do that whithout naming `myproject`? – mhavel Aug 31 '12 at 18:51
  • 1
    @mhavel, you may use [relative import](http://docs.python.org/tutorial/modules.html#intra-package-references) there: from .. import COOL; COOL.cool() – utapyngo Sep 01 '12 at 01:03
  • sorry I've not been precise enough. I knew I could use relative import for such a low-depth module (relative to the root of the package), but how about `myproject.mymodule_one.mysubmodule.mysubsubmodule.file.py`? I did not see any example in the documentation I've read about relative import... – mhavel Sep 01 '12 at 02:55
  • @mhavel: from ..mymodule_one.mysubmodule.mysubsubmodule.file import OBJECT – utapyngo Sep 02 '12 at 05:34