6

Let me start by saying I've done extensive research over the course of the past week and have not yet found actual answers to these questions - just some fuzzy answers that don't really explain what is going on. If that's just cause I missed what I was looking for, I'm sorry - please just point me in the correct direction.

My directory structure is:

TestProject/
    runtest*
    testpackage/
        __init__.py
        testmod.py
        testmod2.py
        testsubs/
            testsubmod.py

A couple notes:

  • I'm using python2.7 on Ubuntu
  • I'm testing with bpython
  • I'm running bpython from specific directories to test the way import behaves
  • I'm attempting to follow best practices.
  • This package is not installed, it's in a random dev directory
  • This directory is not in the pythonpath
  • I have a single init.py in the package directory
  • There are no init.py files in the nested directories
  • The init.py file is empty
  • testpackage/testmod.py contains TestModClass
  • testpackage/testsubs/testsubmod.py contains TestSubModClass

Things I've observed:

  • When I run bpython from TestProject/ import testpackage works
    • This does not import testpackage.testmod
    • I cannot access testpackage.testmod at all
  • When I run bpython from TestProject/ import testpackage.testmod fails
  • When I run bpython from TestProject/ from testpackage import testmod works
  • I can add code to init.py to explicitly import testmod.py, but not testsubs/testmod.py
    • I don't think this is the correct way to do this, what if the user doesn't want to import that module?
  • from testmod.py i can import testmod2, but not testpackage.testmod2
    • This would be nice to do so that I can name my own modules with overlapping names with STL or twisted (such as testpackage.logging) without causing errors (it sucks to have to name my own modules crap like customerlogging, instead of just mypackage.logging)

And the questions:

  1. Does python deal differently with imports on packages & modules that exist in the pythonpath than when you are trying to import from your current directory?
  2. Why doesn't import testpackage give me access to testpackage.testmod? When I import os, I can then access os.path (etc).
  3. With a package, should I stick to using a single init.py in the base directory, or should I be nesting them in subsequent directories?
  4. How can I import a module specifying the package name? I.E. from testmod.py, I would like to import testpackage.testmod2 rather than just testmod2.
  5. What is the proper way to import submodules from the subsubs directory?
    1. The only solution I see is to append that directory to the pythonpath from init.py, but I don't know if that's the correct way.

Thanks in advance.

ThomasH
  • 22,276
  • 13
  • 61
  • 62
nfarrar
  • 2,291
  • 4
  • 25
  • 33
  • 3. testsubs also needs an `__init__.py`, it's a subpackage. 2. because os's `__init__.py` says so. 4. The package needs to be in your path. 4. there are no submodules, only subpackages. – agf Aug 23 '11 at 15:37

2 Answers2

9

First off, you will find all the information you need in section 6 of The Python Tutorial.

(1) Does python deal differently with imports on packages & modules that exist in the pythonpath than when you are trying to import from your current directory?

No, it doesn't. Actually, Python always searches sys.path when importing modules. Modules in the current directory are only found since sys.path contains an entry with the empty string, meaning the current directory.

(2) Why doesn't import testpackage give me access to testpackage.testmod? When I import os, I can then access os.path (etc).

For efficiency, import testpackage only loads testpackage/__init__.py. If you need testpackage.testmod, you have to explicitely import it:

import testpackage   # Just imports testpackage, not testpackage.testmod!
import testpackage.testmod   # Import *both* testpackage and testpackage.testmod!

If you always want to export testmod, import it within __init__.py, this is what os (os/__init__.py) does. This way, testpackage.testmod is always available implicitely if you import testpackage.

Since Python is cross-platform, there is actually no way to consistently and automatically load modules in a directory, because some filesystems are case-insensitive (Windows!). Python would not know whether to load os/path.py as os.path or os.Path, etc.

(3) With a package, should I stick to using a single __init__.py in the base directory, or should I be nesting them in subsequent directories?

You always need an __init__.py for each subpackage. There were discussions on dropping this requirement, but it was decided to keep it as it is.

(4) How can I import a module specifying the package name? I.E. from testmod.py, I would like to import testpackage.testmod2 rather than just testmod2.

This should work. Just ensure that you run the code from the top-level directory. If the current directory is testpackage, testmod does not know that it's in a package.

The preferred way is to use relative intra-package imports, though:

from . import testmod2

This prevents name clashes if there is a global module named testmod2 and enables you to use the names of well-known modules within your package without problems.

(5) What is the proper way to import submodules from the subsubs directory? The only solution I see is to append that directory to the pythonpath from __init__.py, but I don't know if that's the correct way.

No, don't do that! Never, ever put a directory to sys.path when one of it's parent directories already is in sys.path! This could cause your modules to be loaded twice, which is a bad thing!

Usually you should be able to load modules from subpackages using absolute or relative imports:

import testpackage.testsubs.testsubmod
from testpackage.testsubs import testsubmod
from .testsubs import testsubmod

Just make sure to create a __init__.py within testsubs/!

Ferdinand Beyer
  • 64,979
  • 15
  • 154
  • 145
  • Ahah! That answers a lot of stuff, but most importantly - whether or not you can import a module from a package depends on whether the package's __init__.py file imports the modules! I modified the root __init__.py to import testmod.py but not testmod2.py - now, from the TestProject directory I can import testpackage.testmod but not testpackage.testmod2.py. Interesting! – nfarrar Aug 23 '11 at 15:53
  • @nathanfarrar - **No**, you misunderstood that. If there is a `package/__init__.py` and a `package/foo.py`, you can always do `import package.foo`, no matter if `__init__.py` imports `foo` or not! *But*, you have to `import package.foo`, `import package` alone will not import `foo`! – Ferdinand Beyer Aug 23 '11 at 15:56
  • Ah yes, I see that now. Thank you! – nfarrar Aug 23 '11 at 16:19
  • 1
    I cannot get the relative import to work - from . import testmod2 results in ValueError: Attempted relative import in non-package, however the directory contains a __init__.py file. – nfarrar Aug 24 '11 at 02:42
  • Actually, if I import the package it does work, however if I attempt to execute testmod.py directly which contains from . import submod2, then it fails. – nfarrar Aug 24 '11 at 02:48
  • What is the current working directory when you attempt this? As I said, you have to run Python from the top-level directory. If you want to *run* a package module, use `python -m testpackage.testmod`, *not* `python testpackage/testmod.py`. – Ferdinand Beyer Aug 24 '11 at 06:14
0
  1. No. The current directory is just added to PYTHONPATH.
  2. First, you need a __init__.py in there. Second, os.path comes because os is importing os.path.
  3. You need a __init__.py in every directory, also in the one containing the testpackage directory.
  4. In the package's __init__.py, import the modules you wish to provide.
  5. from sub1.sub2 import submodule
    1. No, if the top-most directory is on PYTHONPATH, and this and the subdirectories have their __init__.py, this is all you need to import from any of the subdirectories.
ThomasH
  • 22,276
  • 13
  • 61
  • 62
  • The test project directory is the base directory which contains the executable, documentation, the package, etc. The testpackage/ directory has an __init__.py in it, it's just empty. So it sounds like I should be explicitly importing the other modules from __init__.py. Also, I should place an __init__.py in the testsubs directory which imports testsubsmod.py and I should import testsubs from testpackage/__init__.py – nfarrar Aug 23 '11 at 15:32
  • @nathanfarra Re. \__init.py__ files: If the ``TestProject`` path is in your PYTHONPATH then the top-most init file needs to be on the directory level of ``testpackage`` (This one you have). Then you can write e.g. ``from testpackage import testmod``. – ThomasH Aug 24 '11 at 17:13