70

I've just started using Coverage.py module and so decided to make a simple test to check how it works.

Sample.py

def sum(num1, num2):
    return num1 + num2


def sum_only_positive(num1, num2):
    if num1 > 0 and num2 > 0:
        return num1 + num2
    else:
        return None

test.py 

from sample import sum, sum_only_positive

def test_sum():
    assert sum(5, 5) == 10

def test_sum_positive_ok():
    assert sum_only_positive(2, 2) == 4

def test_sum_positive_fail():
    assert sum_only_positive(-1, 2) is None

As you see, all my code is covered with tests and py.test says all of them pass. I expect Coverage.py to show 100% coverage. Well, no.

Coverage.py results

Well, Coverage.py may not see test.py file, so I copied test functions to sample.py file and ran Coverage again:
enter image description here

Then I added this block of code:

if __name__ == "__main__":
    print(sum(2, 4))
    print(sum_only_positive(2, 4))
    print(sum_only_positive(-1, 3))

and removed all test functions. After that, Coverage.py shows 100%:

enter image description here

Why is it so? Shouldn't Coverage.py show code test coverage, not just execution coverage? I've read an official F.A.Q. for Coverage.py, but can't find the solution.
Since many SO users are familiar with code testing and code coverage, I hope you can tell me, where am I mistaken.

I have just one thought here: Coverage.py may simply watch which lines of code aren't executed so I should write tests for those lines. But there're lines which are executed already but aren't covered with tests so Coverage.py will fail here.

Groosha
  • 2,897
  • 2
  • 22
  • 38
  • How do you invoke coverage / pytest? – Łukasz Rogalski Apr 09 '16 at 13:21
  • @Rogalski pytest: `python -m py.test test.py` and coverage: `python -m coverage run sample.py` (on Windows) – Groosha Apr 09 '16 at 13:25
  • it doesn't show 100%, it shows the same lines as not covered...it doesn't work for me still. i copied pasted your code and make sure i have py.test and coverage.py with pip. i am seeing same in command line and in intellij, please LMK. – Nirmal Apr 05 '19 at 17:15

4 Answers4

42

Coverage looks for a .coverage file to read and generate that report for you. Py.test on its own does not create one. You need py.test plugin for coverage:

pip install pytest-cov

If you already have it, then you can run both at once like this:

py.test test.py --cov=sample.py

Which means run test module test.py and record/display coverage report on sample.py.

If you need to have multiple test runs and accumulate their recorded coverage and then display a final report, you can run it like this:

py.test test.py --cov=sample.py --cov-report=
py.test test.py --cov=sample2.py --cov-report=
py.test test.py --cov=sample3.py --cov-report=

Which means run test module test.py and record (only) coverage on sample.py - don't display a report.

Now you can run coverage command separately for a complete report:

coverage report -m

The command above simply displays a formatted coverage report based on the accumulated .coverage data file from previous test runs. -m means show lines missed i.e. lines not covered by tests:

Name        Stmts   Miss  Cover   Missing
-----------------------------------------
sample.py       6      0   100%

Coverage supports more switches like --include and --omit to include/exclude files using path patterns. For more info check out their docs: https://coverage.readthedocs.io/en/6.0.2/source.html?highlight=reporting#reporting

Daniel Andrei Mincă
  • 4,446
  • 2
  • 19
  • 30
fips
  • 4,319
  • 5
  • 26
  • 42
  • 3
    Doesn't work if the file to examine is a submodule in a package that is imported in the test. Whatever I specify at `--cov=<>` - just the file name, relative or absolute path - I get "ERROR: Failed to generate report: No data to report." – ivan_pozdeev Nov 02 '17 at 14:08
  • "asterisk/mydir/asterisk" works for omit. I would write * for asterisk but it turns into bold. – Squirrel Jan 31 '18 at 14:11
  • 1
    Use `--branch` to also add branch coverage. – Melroy van den Berg Sep 14 '18 at 13:47
  • 2
    You could also do something like `coverage run --source= -m pytest ` instead of `py.test test.py --cov=sample.py` if the additional dependency isn't worth it. – karuhanga Oct 16 '18 at 11:53
  • `ERROR: usage: py.test [options] [file_or_dir] [file_or_dir] [...] py.test: error: unrecognized arguments: --cov=test.py` – cryanbhu Dec 19 '20 at 11:15
  • You need to install the [pytest-cov](https://pypi.org/project/pytest-cov/) which adds support for the `--cov` option, or alternatively the approach in the comment above should also work. – fips Dec 19 '20 at 12:21
  • @Squirrel You can use backticks (\`) for raw formatting. This \`\*/mydir/\*\` ~~> `*/mydir/*` – Mark Moretto Mar 10 '23 at 13:42
26

It's a little hard to parse through your experiments, and you haven't included the command lines you used with each experiment. But: if you run the tests with:

python -m py.test test.py

then you can run them under coverage.py with:

coverage run -m py.test test.py
Ned Batchelder
  • 364,293
  • 75
  • 561
  • 662
  • 1
    By the way, I've included commands used to run tests and coverage just under my post in second comment: pytest: `python -m py.test test.py` and coverage: `python -m coverage run sample.py` (on Windows). I see your second command differs from mine, will check this, thank you! – Groosha Apr 10 '16 at 07:50
  • 3
    Maybe the problem here is a basic misunderstanding: you aren't supposed to run the tests, and then coverage. You are supposed to use coverage to run your tests (the way I recommend), or enable coverage during the running of tests (for example with a test runner's coverage plugin). – Ned Batchelder Apr 11 '16 at 01:19
  • Doesn't work for me :( `coverage -m` -> `No such option -m`. `coverage run py.test test.py` -> `Unknown command py.test` (py.test is installed) – Groosha Apr 11 '16 at 12:55
  • @NedBatchelder This method includes test.py in the generated report. Is that considered a best practice when using a tool like coverage? By that I mean is the intent to always measure both the coverage on the module to be tested and the test itself? If not how would I exclude the test.py coverage from the report? – Grr Mar 16 '17 at 20:06
  • 2
    I see no reason to exclude tests from measurement, and only advantages. You have code in your tests, you want to know that it is being executed. Coverage.py can tell you that. – Ned Batchelder Mar 16 '17 at 23:48
22

The below command worked for me:

coverage run --source=sample -m pytest test.py

coverage report -m
Anand
  • 823
  • 1
  • 10
  • 19
4

putting __init__.py in every test folder resolves the original question and shows correct coverage.

user5194542
  • 41
  • 3