What is the proper way to split a python module into submodules?

Question

I am working in a gitlab API library in python. It started as a pet project to understand more about modules so I didn't think about it at the start and started writing all the code in the __init__.py inside the module dir.

Of course, now that is has grown, I can see that the organization is quite poor, there is way to much method for one class, testing has become difficult, checking the code is quite confusing as there is a lot in there.

So I have thougth of splitting it in several submodules, you know the tipical from X import Y so the code is more readable, testeable, smaller but I have found that I have absolutely no idea how to implement it as I got plenty of shared variables and such all over the class with all methods using class variables....

So, is there any good documentation on creating python modules with "submodules"? How to share objects variables between classes? Any clear modules that I should be checking that have the login on one submodule and the functions in other, for example?

Cheers!

Give this a read? http://stackoverflow.com/questions/2098088/should-i-create-each-class-in-its-own-py-file — brandonscript, Dec 11 '13 at 23:38
I don't think this question is answerable in the present form. As a general hint, start by rewriting your functions so that they don't have external dependencies (shared variables). What would be more helpful for you is to extract the most problematic parts of your code and ask for a [code review](http://codereview.stackexchange.com/). — georg, Dec 11 '13 at 23:42
@thg435 But what if they depend completely from others? Let's say, you have a package which has a module login and a module dosomething. What if the dosomething module needs the login module to suceed so the user has generated a token which will be used in the dosomething.getsomething()? — Itxaka, Dec 11 '13 at 23:47
Itxaka then you have not successfully split the code. dosomething just takes a token. It does not care where it came from. It just has to "be a token". This means it has token.valid() for instance. Anything that honors the interface works. This way, you can test things that want a token without first needing to really authenticate. — Sean Perry, Dec 11 '13 at 23:50
@Itxaka It is OK for a module to depend on other modules for specific *functionality* that they implement. That is why you are modularizing the code in the first place: To divide responsibilities in a sensible manner, and to be able to reuse functionality in different places without having to implement it multiple times. However, you should not modify the same variable(s) from different functions. — itsjeyd, Dec 11 '13 at 23:53
If it's a "pet project" as you have described it, then you have created the perfect situation! You can learn way more by correcting mistakes then by doing it (unintentional) right in the first place! ;-) "Law of Demeter", "Separation of Concern", "Composition over Inheritance" should be good starting buzz-words! ;-) — Don Question, Dec 11 '13 at 23:55

score 2 · Answer 1 · edited Dec 09 '20 at 16:44

This question has been asked a lot and the problem with this kind of question is it will be rather subjective (at best). Sure there are lots of Best Practices such as:

Logical grouping of functionality into modules
Keeping your modules terse and not too complex
Designing a sensible API

etc...

If you find it confusing, difficult to use, chances are others will too. Test your API and design on your co-workers and friends. Get their view of how it feels ot use your library.

Some useful references:

Thanks for all the links, I will have a look at them – Itxaka Dec 17 '13 at 11:41 — Itxaka, Dec 17 '13 at 11:41

score 1 · Accepted Answer · answered Dec 11 '13 at 23:48

In general this is referred to as 'refactoring'. Lots of good Python and general computer science literature out there if you look for it.

Up front based on what you have described it sounds like you need to stop using the data stored in your classes directly and start using member functions for access.

class A(object):
    def __init__(self, value):
        self.value = value

a = A(1)
print a.value
a.value = 5
print a.value

See how I modified value directly? This can lead to all kinds of assumptions in code. You are better off with using methods to retrieve values and make changes. Note, mindless getFoo/setFoo is also bad. Try to have your methods expose behavior not knobs to twist.

class A(object):
    def __init__(self, value):
        self.value = value

    def is_safe(self):
        if value > 10:
            return False
        return True

    def increase_value(self, increment):
        if increment <= 0:
            raise ValueError("only positive increments are allowed")
        self.value += increment

See how the logic is self contained? We never want an instance of A to get smaller so we only allow increments and name the function increase to indicate this. setValue does not tell what or why.

As for modules and submodules, group matching things together. "matching" is based on your own code's needs. All of the user functions? All of the DB functions? Putting things in modules allows for the possibility of plugging in a different module with the same exposed API. Think authentication from a SQL DB or from a local text file.

What is the proper way to split a python module into submodules?

2 Answers2