py.test gives Coverage.py warning: Module sample.py was never imported

Question

I ran a sample code from this thread. How to properly use coverage.py in Python?

However, when I executed this command py.test test.py --cov=sample.py it gave me a warning, therefore, no report was created.

platform linux2 -- Python 2.7.12, pytest-3.2.3, py-1.4.34, pluggy-0.4.0
rootdir: /media/sf_Virtual_Drive/ASU/CSE565_testand 
validation/Assignments/temp, inifile:
plugins: cov-2.5.1
collected 3 items                                                                                                                                                                                                                      

test.py ...Coverage.py warning: Module sample.py was never imported. (module-not-imported)
Coverage.py warning: No data was collected. (no-data-collected)

Anyone has an idea why coverage.py does not work?

hence, if I run coverage run -m py.test test.pyseparately, it does not show any warning.

Answer 1

Short answer: you need to run with the module name, not the file name: pytest --cov sample test.py

Long answer:

One comment in the answer you linked (How to properly use coverage.py in Python?) explains that this doesn't seem to work if the file you are trying to get the coverage of is a module imported by the test. I was able to reproduce that:

./sample.py

def add(*args):
    return sum(args)

./test.py

from sample import add

def test_add():
    assert add(1, 2) == 3

And I get the same error:

$ pytest --cov sample.py test.py
========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 3.7.2, pytest-4.3.1, py-1.8.0, pluggy-0.9.0
rootdir: /path/to/directory, inifile:
plugins: cov-2.6.1
collected 1 item

test.py .                                                                                                                                                                                          [100%]Coverage.py warning: Module sample.py was never imported. (module-not-imported)
Coverage.py warning: No data was collected. (no-data-collected)
/path/to/directory/.venv/lib/python3.7/site-packages/pytest_cov/plugin.py:229: PytestWarning: Failed to generate report: No data to report.

  self.cov_controller.finish()
WARNING: Failed to generate report: No data to report.

However, when using the module name instead:

pytest --cov sample test.py
========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 3.7.2, pytest-4.3.1, py-1.8.0, pluggy-0.9.0
rootdir: /path/to/directory, inifile:
plugins: cov-2.6.1
collected 1 item

test.py .                                                                                                                                                                                          [100%]

---------- coverage: platform darwin, python 3.7.2-final-0 -----------
Name        Stmts   Miss  Cover
-------------------------------
sample.py       2      0   100%

The pytest-cov documentation seems to indicate you can use a PATH, but it might not be working in all cases...

Answer 2

tl;dr Use coverage to generate the statistics file .coverage and then create a report that scopes to your specific file only.

coverage run -m pytest .\test\test_named_prng.py
coverage html --include=named_prng.py 

Situation

Let's suppose you have some python files in your package, and you also have test cases within a single test file (test/test_named_prng.py). You want to measure the code coverage of your test file on one specific file within your package (named_prng.py).

\namedPrng
│   examples.py
│   named_prng.py
│   README.md
│   timeit_meas.py
│   __init__.py
│
└───test
        test_named_prng.py
        __init__.py

Here namedPrng/__init__.py imports examples.py and named_prng.py, where the other init file is empty. An example with files is available on my GitHub.

Problem

Your problem is that with pytest or with coverage you cannot scope the report to your specific file (named_prng.py), because every other file imported from your package is also included in the report.

root cause

If you have an __init__.py in the level where the module you want to import is located, then __init__.py may import more files than necessary as the __init__.py will be executed. There are options to tell pytest and coverage to restrict which modules you want to investigate, but if they involve further modules from your package, they will be analysed too.

symptom with pytest

The option --cov of the package pytest-cov, which is used if you issue pytest with the option --cov, doesn't work if the (sub)module you want to create the coverage test on was imported from __init__.py.

If you run pytest (from namedPrng) with

pytest .\test\test_named_prng.py --cov --cov-report=html

you will get a report every .py file except the timeit_meas.py, because it is never imported: nor the test, nor its init, nor the imported named_prng.py, nor its init.

If you run pytest with

pytest .\test\test_named_prng.py --cov=./ --cov-report=html

then you explicitly tell coverage (invoked with pytest) to include everything in your level, therefore every .py file will be included in the report.

You'd like to tell coverage to create the report only on the source code of named_prng.py, but if you specify your module to --cov with

pytest .\test\test_named_prng.py --cov=named_prng --cov-report=html

or with --cov=named_prng.py you will get a warning:

Coverage.py warning: Module named_prng.py was never imported. (module-not-imported)
Coverage.py warning: No data was collected. (no-data-collected)
WARNING: Failed to generate report: No data to report.

symptom with coverage

One can run the coverage and report separately and hope that more detailed options can be passed to coverage.

By issuing

coverage run -m pytest .\test\test_named_prng.py
coverage html

you get the same report on the 5 .py files. If you try to tell coverage to use only named_prng.py by

coverage run --source=named_prng -m pytest .\test\test_named_prng.py

or with --source=named_prng.py, you will get a warning

Coverage.py warning: Module named_prng.py was never imported. (module-not-imported)
Coverage.py warning: No data was collected. (no-data-collected)

and no report will be created.

Solution

You need to use the --include switch for coverage which unfortunately cannot be passed to pytest in a CLI.

Use coverage CLI

You can restrict the scope of investigation during code coverage calculation time:

coverage run --include=named_prng.py -m pytest .\test\test_named_prng.py
coverage html

or at reporting time.

coverage run -m pytest .\test\test_named_prng.py
coverage html --include=named_prng.py 

Use pytest + settings file

One can call pytest with detailed configuration via a config file. Where you issue pytest, set up a .coveragerc file with the content

[run]
include = named_prng.py

Check coverage's description on the possible options and patterns.

Answer 3

This can be solved by running coverage first on your test file then generate the report as follows:

coverage run test.py

coverage report -m