managing a python package that depends on other packages

Question

I can’t imagine this is an original question, but I can’t seem to find an answer. I must be using the wrong search terms.

Our team is developing a Python utility package. We’ll call it “tools”. Various classes in “tools” require other packages and modules.

“tools” has several classes, with names like:

• Parser
• Logger
• FooTester
• BarTester

There are 2 other packages. One is called “foo”, and the other is “bar”. Some teams only use the “foo” package, some teams only use “bar” package. It is a requirement that teams that only use the “foo” package don’t install the “bar” package.

The “FooTester” class requires the “foo” package, as it has a “import foo” in it. The “BarTester” class requires the “bar” package, as it has a “import bar” in it.

Both teams want to put the following at the top of their script: “import tools”, then use their respective Tester class.

As we have it right now, you can’t do that unless you have the “bar” and the “foo” package installed.

What is the standard way of doing this? Is there?

Answer 1

I would think that FooTester would be part of the foo package and BarTester part of the bar package. (Or perhaps each in its own package/module.)

Given the requirement that teams that use the foo package don't install the bar package, it seems weird to put the testers for both in a tools package that both use.

Answer 2

"tools" can define its classes conditionally:

fooed = barred = False

try:
    import foo
    fooed = True
    class FooTester(object):
        pass # your class here...
except ImportError:
    pass

try:
    import bar
    barred = True
    class BarTester(object):
        pass # your class here...
except ImportError:
    pass

if not fooed or barred:
    raise RuntimeError("You are not fooed or barred")

Answer 3

There are few solutions

Option All: Install everything, ignore your special requirement

This does not answer your question, but is probably the most effective solution. Unless you have some packages, which are for any reason difficult to install, it is not worth the effort to separate their installation for different teams. If you count the time needed to implement that, you would find, it is not feasible. If time needed for installation of packages is your concern, see related SO answer for pip configuration speeding up your install down to a second or two even for package requiring compilation.

Option Multi: Organize the solution into multiple multilevel packages

You may split the solution into packages like

• tool.common
• tool.teamA
• tool.teamB
• tool.teamB

Tool is in this case so called namespace package and shall be empty.

Any of packages might require installation of other packages, typically the tool.common could be part of required packages in all others.

Option Pip: Use pip with requirements.txt files

There is other common pattern, using pip and requirements.txt files.

pip allows installation of files declared in some text file, typically called reuirements.txt, but any other name would work.

Instead of using:

$ pip install tool.teamA

people would somehow get the code to work with, e.g. by:

$ git clone repo4tools

In that repository, there would be files

• requirements-teamA.txt
• requirements-teamB.txt
• requirements-teamC.txt

and whatever else would be needed.

Each requirement file would contain only the packages needed for given team.

People would then simply call:

$ cd repo4tools
$ pip install -r requirements-teamA.txt

For developers, last option is becoming quite popular, especially in combination with virtualenv. Sometime there are even multiple requirements files for each team, one for real use, another for developing and testing etc.