2

I am working on a Jenkins CI/CD pipeline for my Python package. My project file hierarchy is as follows:

project/
- package_name
  - file1.py
  - file2.py
  - etc...
- tests
  - unit
    - __main__.py
    - __init__.py
    - test1.py
    - test2.py

All unit tests (I am using unittest) are run using a single command

python -m tests.unit

via adding __init__.py of the following content:

contents

import os
os.chdir(os.path.dirname(os.path.abspath(__file__)))

and __main__.py which looks like this

contents

import unittest
import sys

sys.path.append('../..')

loader = unittest.TestLoader()
start_dir = '.'
suite = loader.discover(start_dir)
runner = unittest.TextTestRunner(verbosity=2).run(suite)

First, the path is changed to the ./tests/unit. After that, the top directory is added to the import path so that the package can be imported in tests. This works as intended (i.e., all rests are executed by running python -m test.unit at the top of the project directory) on my personal laptop (Python 3.6.4).

However, when I use the same trick on a remote Jenkins server (Python 3.6.4 as well), I am getting the following error:

no module named test.unit.__main__; 'test.unit' is a package and cannot be directly executed

I have researched the issue but none of the proposed solutions seems to be working in my case.

How can I modify my code to create a test suite in unittest that will run without any issues locally and remotely?

EDIT I tried modifying the PYTHONPATH variable, but no success

slava-kohut
  • 4,203
  • 1
  • 7
  • 24
  • 1
    If you are using unittest you can just run `python -m unittest discover`. You don't need the `__main__.py`. See the [docs](https://docs.python.org/3/library/unittest.html#command-line-interface). You can probably just delete the contents of `__init__.py` as well. – adamkgray Jul 05 '20 at 12:53
  • @mark_s it does have the init file, the package is a legit Python package – slava-kohut Jul 09 '20 at 23:48
  • I'm not yet familiar with Jenkins, and I feel a little goofy asking this, but the remote server isn't defaulting to Python 2.7 when running your code is it? Some of the Google responses for your error message associate it with Python version 2.x .. – mark_s Jul 10 '20 at 00:28
  • @mark_s the same version of Python in both systems (3.6.4, local and remote) – slava-kohut Jul 10 '20 at 12:17
  • 1
    Are you sure that the issue is not caused by the `python -m test.unit` vs `python -m tests.unit` typo (note the single letter difference)? At least the text of your question suggests that that may be case. If the typo is in your question, please edit it, and I will remove my comment. – Leon Jul 10 '20 at 20:42

2 Answers2

1

I can run the tests using this command:

python -m unittest discover

You should never need to run os.chdir or append to sys.path just to run your tests.

I replicated your project structure, like this:

.
├── package
│   └── module.py
└── tests
    ├── __init__.py
    └── unit
        ├── __init__.py
        └── test_module.py

Notice how each directory under tests/ has an __init__.py file, which is actually empty. It only needs to exist. Its existence makes that directory a module. For more use cases read this answer.

You don't need __main__.py. That is for when you want to run python3 -m package from the command line (instead of python3 package/module.py or something like that). See this answer.

In this case unittest just needs the __init__.py files in the test dirs to work properly.

It is also important that the name of each test file begins with test_, as this is a naming convention that unittest looks for.

For reference, the contents of module.py looks like this:

def method():
    return True

And the contents of test_module.py looks like this:

from package.module import method
from unittest import TestCase

class TestMethod(TestCase):
    def test_method(self):
        self.assertTrue(method())

Making these modifications to your code will allow you to run your test suite in unittest without any issues locally and remotely.

adamkgray
  • 1,717
  • 1
  • 10
  • 26
  • thanks for your input. I am more interested to learn why importing tests as a module works in one place, but not the other. – slava-kohut Jul 06 '20 at 12:36
1

1. Why it does not work?

1.1. About python -m and __main__.py

When you run python -m tests.unit, what code will be ran by the python interpreter, is, in this order

tests.__init__.py
tests.unit.__init__.py
tests.unit.__main__.py

1.2. Reproducing the error

Now, if you remove the __main__.py, you will get the following error:

No module named tests.unit.__main__; 'tests.unit' is a package and cannot be directly executed

This is almost the same message as you did get. If you have in your sys.path a folder that contain folder called test with structure like this (Note: the test-folder is not in prular, and there is no __main__.py!)

test
├── __init__.py
└── unit
    └── __init__.py

and run command

python -m test.unit

what python interpreter tries to run, in this order is

test.__init__.py
test.unit.__init__.py
test.unit.__main__.py <-- missing!

Since test.unit.__main__.py is missing, you will get error message

No module named test.unit.__main__; 'test.unit' is a package and cannot be directly executed

which is the error message you are getting. So, the reason for your error message most probably, that you have somewhere in your sys.path directories a directory called test with the structure shown above, and you are trying to call python -m test.unit instead of python -m tests.unit.

2. How to make it work?

  • Remove your os.chdir and sys.path.append hacks that you are using in your __init__.py and __main__.py. At least, they are not needed for python unittest to work.
  • Create unittests using the pattern shown in the documentation (by subclassing unittest.TestCase)
  • Run your unit tests by
python -m unittest
Niko Föhr
  • 28,336
  • 10
  • 93
  • 96