Running unittest with modules that must import other modules

Question

Our Python 3.10 unit tests are breaking when the modules being tested need to import other modules. When we use the packaging techniques recommended by other posts and articles, either the unit tests fail to import modules, or the direct calls to run the app fail to import modules. The other posts and articles we have read do not show how to validate that both the application itself and the unit tests can each import modules when called separately. So we created a bare bones example below and are asking how to structure the packaging correctly.

What specific changes must be made to the syntax below in order for the two Python commands given below to successfully run on the bare bones example app given below?

Problem description

A Python 3.10 app must import modules when called either directly as an app or indirectly through unit tests.

Packages must be used to organize the code.

Calls to unit tests are breaking because modules cannot be found.

The two test commands that must run without errors to validate solution of this problem are:

C:\path\to\dir>python repoName\app\first.py

C:\path\to\dir>python -m unittest repoName.unitTests.test_example

We have reviewed many articles and posts on this topic, but the other sources failed to address our use case, so we have created a more explicit example below to test the two types of commands that must succeed in order to meet the needs of this more explicit use case.

App structure

The very simple structure of the app that is failing to import packages during unit tests is:

repoName
  app
    __init__.py
    first.py
    second.py
    third.py
  unitTests
    __init__.py
    test_example.py
  __init__.py

Simple code to reproduce problem

The code for a stripped down example to reproduce the problem is as follows:

The contents of repoName\app\__init__.py are:

print('inside app __init__.py')
__all__ = ['first', 'second', 'third']

The contents of first.py are:

import second as second
from third import third
import sys

inputArgs=sys.argv

def runCommands():
  trd = third() 
  if second.something == 'platform':
    if second.another == 'on':
      trd.doThree()
  if second.something != 'unittest' :
    sys.exit(0)

second.processInputArgs(inputArgs)
runCommands()

The contents of second.py are:

something = ''
another = ''
inputVars = {}

def processInputArgs(inputArgs):
    global something
    global another
    global inputVars
    if ('unittest' in inputArgs[0]):
      something = 'unittest'
    elif ('unittest' not in inputArgs[0]):
      something = 'platform'
      another = 'on'
    jonesy = 'go'
    inputVars =  { 'jonesy': jonesy }

The contents of third.py are:

print('inside third.py')
import second as second

class third:

  def __init__(self):  
    pass

  #@public
  def doThree(self):
    print("jonesy is: ", second.inputVars.get('jonesy'))

The contents of repoName\unitTests\__init__.py are:

print('inside unit-tests __init__.py')
__all__ = ['test_example']

The contents of test_example.py are:

import unittest

class test_third(unittest.TestCase):

  def test_doThree(self):
    from repoName.app.third import third
    num3 = third() 
    num3.doThree()
    self.assertTrue(True)

if __name__ == '__main__':
    unittest.main()

The contents of repoName\__init__.py are:

print('inside repoName __init__.py')
__all__ = ['app', 'unitTests']

Error resulting from running commands

The command line response to the two commands are given below. You can see that the call to the app succeeds, while the call to the unit test fails.

C:\path\to\dir>python repoName\app\first.py
inside third.py
jonesy is:  go

C:\path\to\dir>python -m unittest repoName.unitTests.test_example
inside repoName __init__.py
inside unit-tests __init__.py
inside app __init__.py
inside third.py
E
======================================================================
ERROR: test_doThree (repoName.unitTests.test_example.test_third)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\path\to\dir\repoName\unitTests\test_example.py", line 15, in test_doThree
    from repoName.app.third import third
  File "C:\path\to\dir\repoName\app\third.py", line 3, in <module>
    import second as second
ModuleNotFoundError: No module named 'second'

----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (errors=1)

What specific changes must be made to the code above in order for all the modules to be imported correctly when either of the given commands are run?

Answer 1

Creating an "alias" for modules

Update the contents of repoName\app\__init__.py to:

print('inside app __init__.py')
__all__ = ['first', 'second', 'third']

import sys

import repoName.app.second as second
sys.modules['second'] = second

import repoName.app.third as third
sys.modules['third'] = third

import repoName.app.first as first
sys.modules['first'] = first

How to ensure first.py gets run even when imported

So when the test fixture imports repoName.app.third, Python will recursively import the parent packages so that:

import repoName.app.third is equivalent to

import repoName
# inside repoName __init__.py
import app
#inside app __init__.py
import third
#inside third.py

So running from repoName.app.third import third inside test_doThree, executes repoName\app\__init__.py. In __init__.py, import repoName.app.first as first is called. Importing first will execute the following lines at the bottom of first.py

second.processInputArgs(inputArgs)
runCommands()

In second.processInputArgs, jonesy = 'go' is executed setting the variable to be printed out when the rest of the test is ran.

Answer 2

Here is how I have gone about trying to solve this.

I exported the PYTHONPATH to the repo folder repoName (I am using linux)

cd repoName
export PYTHONPATH=`pwd`

then in test_example.py

import unittest

class test_third(unittest.TestCase):

  def test_doThree(self):
    from app.third import third # changed here
    num3 = third()
    num3.doThree()
    self.assertTrue(True)

if __name__ == '__main__':
    unittest.main()

Then in third.py

print('inside third.py')
import app.second as second # changed here

class third:

  def __init__(self):  
    pass

  #@public
  def doThree(self):
    print("jonesy is: ", second.inputVars.get('jonesy'))

Also it is worth noting that I did not create any __init__.py files

Answer 3

The code in the question relies on first.py being imported so it calls a function in second.py to set a global that is used by third.py. As the Zen Of Python says:

Explicit is better than implicit

The current structure will be difficult to maintain, test, and debug as your project grows. I have redone the example in the question removing globals and code being executed on import.

first.py

import sys

from app import second
from app.third import Third


def run_commands(input_args):
    trd = Third()
    if input_args.another == "on":
        trd.do_three(input_args)


def main():
    input_args = second.process_input_args(sys.argv)
    run_commands(input_args)


if __name__ == "__main__":
    main()

second.py

from dataclasses import dataclass


@dataclass
class InputArgs:
    something: str
    another: str
    jonesy: str


def process_input_args(input_args):
    something = "platform"
    another = "on"
    jonesy = "go"
    return InputArgs(something, another, jonesy)

third.py

import sys

print("inside third.py")


class Third:
    def __init__(self):
        pass

    # @public
    def do_three(self, input_args):
        print("jonesy is: ", input_args.jonesy)

test_example.py

import io
import unittest
from unittest import mock

from app.second import InputArgs
from app.third import Third


class ThirdTests(unittest.TestCase):
    def test_doThree(self):
        input_args = InputArgs(something="platform",
                               another="on",
                               jonesy="go")

        num3 = Third()
        with unittest.mock.patch('sys.stdout', new=io.StringIO()) as fake_out:
            num3.do_three(input_args)
            self.assertEqual("jonesy is:  go\n", fake_out.getvalue())


if __name__ == "__main__":
    unittest.main()

For Python development I would always recommend having a Python Virtual Environment (venv) so that each repo's development is isolated.

In the repoName directory do (for Linux):

python3.10 -m venv venv

Or like the following for windows:

c:\>c:\Python310\python -m venv venv

You will then need to activate the venv.

Linux: . venv/bin/activate

Windows: .\venv\scripts\activate.ps1

I would suggest packaging the app as your module then all your imports will be of the style:

from app.third import third
trd = third()

or

from app import third
trd = third.third()

To package app create a setup.py file in the repoName directory. The file will look something like this:

from setuptools import setup

setup(
    name='My App',
    version='1.0.0',
    url='https://github.com/mypackage.git',
    author='Author Name',
    author_email='author@gmail.com',
    description='Description of my package',
    packages=['app'],
    install_requires=[],
    entry_points={
        'console_scripts': ['my-app=app.first:main'],
    },
)

I would also rename the unitTests directory to something like tests so that the unittest module can find it automatically as it looks for files and directories starting with test.

So a structure something like this:

repoName/
├── app
│   ├── __init__.py
│   ├── first.py
│   ├── second.py
│   └── third.py
├── setup.py
├── tests
│   ├── __init__.py
│   └── test_example.py
└── venv

You can now do pip install to install from a local src tree in development mode. The great thing about this is that you don't have to mess with the python path or sys.path.

(venv) repoName $ pip install -e .
Obtaining file:///home/user/projects/repoName
  Preparing metadata (setup.py) ... done
Installing collected packages: My-App
  Running setup.py develop for My-App
Successfully installed My-App-1.0.0

With the install done the app can be launched:

(venv) repoName $ python app/first.py
inside app __init__.py
inside third.py
jonesy is:  go

In the setup file we told python that my-app was an entry point so we can use this to launch the same thing:

(venv) repoName $ my-app 
inside app __init__.py
inside third.py
jonesy is:  go

For the unittests we can use the following command and it will discover all the tests because we have used test to start directory and file names.

(venv) repoName $ python -m unittest 
inside app __init__.py
inside unit-tests __init__.py
inside third.py
jonesy is:  go
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

Now we have this setup it is easy to package up app for distribution. Either directly to users or via a Package Index like https://pypi.org/

Install the build module:

(venv) repoName $ pip install --upgrade build

To build the Python wheel:

(venv) repoName $ python build

There should now be a dist folder with a wheel in it which you can send to users. They can install this with pip:

pip install My_App-1.0.0-py3-none-any.whl