Best way to keep Python modules organised

Question

I am using Python to work in Maya and over the months I have started to build a lot of different modules that shape various tools.

Initially I was simply stacking everything inside one single root package, composed of huge modules containing a bit of functions, classes etc depending on the situation, with a separate "core" module containing all the generic stuff.

root (package)
    animation.py  (module)
    audio.py      (module)
    modelling.py  (module)
    rigging.py    (module)
    library.py    (module)
    core.py       (module)

Recently I have seen some examples of a different approach, which consist in building packages for each macro area and splitting the code inside them into various smaller modules. Something similar to this.

root (package)
    animation (package)
        __init__.py  (module)
        core.py      (module)
        utils.py     (module)
        ui.py        (module
        bar.py       (module)

    audio (package)
        __init__.py  (module)
        core.py      (module)
        utils.py     (module)
        ui.py        (module
        foo.py       (module)

   rigging (package)
        __init__.py  (module)
        core.py      (module)
        utils.py     (module)
        ui.py        (module
        foo.py       (module)

   etc

Inside the __init__.py modules the other modules are imported this way.

from core import *
from utils import *
from ui import Window
from foo import Foo

So that when the whole package is imported the result is quite similar to just having everything inside one huge module that contains everything. At the end, to access the functionalities of each module/package this code should work for both the structures.

import animation
animation.Foo()

My question is:

Which of these methods is more logic to organise code? On one way the "package" structure looks better organised, but it's giving me more cyclic imports and reload problems (that may just be a sign of my bad import habits). On the other, the classic modules are starting to become quite hard to navigate because of the amount of stuff I put in them.

It's all just preference, people could argue for a millennia about the positives and negatives of any style. — tgikal, Aug 29 '18 at 16:27
I understand. Is it normal to stumble into a lot of cycling imports while using the second method? I am liking it more, but I am struggling especially when declaring classes in different modules and trying to call them in other packages. I get cycles in the imports and i am forced to place the import inside functions or classes which does not seem a good practice. — TheArcadeFire, Aug 29 '18 at 16:35
There's nothing cyclic about what you have. You may want to read up on absolute vs relative imports, as you may not be importing the module you think you are in some case. — chepner, Aug 29 '18 at 16:51
I think I didn't express properly what I mean. I have two modules A and B, the module A needs some functions from B and B has a definition of a class which inherits from a class in A. When I import A, it imports B as well, which tries to Import A and inherit from one of its classes before the definition of A is even finished. This is happening to me a lot when working with the second structure. — TheArcadeFire, Aug 29 '18 at 16:57

score 1 · Accepted Answer · answered Aug 29 '18 at 16:46

First of all, welcome to StackOverflow! Second, it is definitely a matter of preference, but you should avoid the syntax from package import * because it pollutes the namespace a lot, especially if you have many functions.

About your question, if you take a look at many standard library packages in Python, they consist of a root folder with roughly 5-6 py files. Now, this may not be the prettiest thing to do, but it is not disorganized, and it allows for easy imports.

For your situation, as you have many files to consider, you should first think, "Are my Python files too small?" Coming from a Java background, I understand the drive to make lots of files each containing a small bit. Try to put like things with like things in a way that makes sense, without having too many files.

If all of these modules are connected by some sort of main application, what you should try to do is separate the modules as much as possible, so you need to import as few things as possible from each module. If you are writing a library, you should 100% try to make the modules as stand alone as possible, unless you plan on shipping it as one big library. When shipping as one library, you can basically organize it however you want as long as it makes sense and works.

Basically:

Make a sub module for each and you get prettiness and organization, but import hell
Have 5-6 large files and imports are easy, but the files take time to navigate and find what you need to use/modify

Could you please expand on the reason why the syntax `from package import *` is not great? Is there any situation where it can be useful? Should I import each module with his own name, or at max, using `import module as mod`? — TheArcadeFire, Aug 29 '18 at 17:00
https://stackoverflow.com/questions/2386714/why-is-import-bad The main reason is all the variables, functions and classes pollute your file's namespace. If you get in the habit of doing this, eventually there will be a collision of names. However, there are libraries that are separated into submodules, meaning `from package import *` just imports the names of the submodules and nothing else. — Steampunkery, Aug 30 '18 at 04:39

Best way to keep Python modules organised

1 Answers1