66

I am new to coverage and ran into a strange problem. My coverage is taking my virtual environment site packages into account. Here is the output of the coverage run:

coverage run test.py
....................
----------------------------------------------------------------------
Ran 20 tests in 0.060s

OK
(atcatalog)- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -jmfrank63@fullstack-audio-text-catalog:~/workspace (git master)       [19:58:45]
$ coverage report
Name                                                                              Stmts   Miss  Cover
-----------------------------------------------------------------------------------------------------
/home/ubuntu/Envs/atcatalog/lib/python2.7/site-packages/flask/__init__               18      0   100%
/home/ubuntu/Envs/atcatalog/lib/python2.7/site-packages/flask/_compat                38     20    47%
/home/ubuntu/Envs/atcatalog/lib/python2.7/site-packages/flask/app                   528    255    52%
/home/ubuntu/Envs/atcatalog/lib/python2.7/site-packages/flask/blueprints            156    118    24%
                             .
                             .
                             .
/home/ubuntu/Envs/atcatalog/lib/python2.7/site-packages/werkzeug/urls               412    215    48%
/home/ubuntu/Envs/atcatalog/lib/python2.7/site-packages/werkzeug/utils              242    175    28%
/home/ubuntu/Envs/atcatalog/lib/python2.7/site-packages/werkzeug/wrappers           568    298    48%
/home/ubuntu/Envs/atcatalog/lib/python2.7/site-packages/werkzeug/wsgi               448    352    21%
atcatalog/__init__                                                                    7      0   100%
atcatalog/views/__init__                                                              0      0   100%
atcatalog/views/publang                                                               7      0   100%
atcatalog/views/pubtext                                                               1      0   100%
atcatalog/views/userlang                                                             13      0   100%
atcatalog/views/users                                                                 5      0   100%
atcatalog/views/usertext                                                             14      0   100%
test                                                                                120      0   100%
-----------------------------------------------------------------------------------------------------
TOTAL                                                                             12530   8044    36%
(atcatalog)- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -jmfrank63@fullstack-audio-text-catalog:~/workspace (git master)       [19:58:55]

Here is the structure of my project directory which resides under home:

workspace/
├── README.md
├── atcatalog
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── static
│   ├── templates
│   └── views
│       ├── __init__.py
│       ├── __init__.pyc
│       ├── publang.py
│       ├── publang.pyc
│       ├── pubtext.py
│       ├── pubtext.pyc
│       ├── userlang.py
│       ├── userlang.pyc
│       ├── users.py
│       ├── users.pyc
│       ├── usertext.py
│       └── usertext.pyc
├── requirements.txt
├── run.py
└── test.py

I had the virtual environment at first inside the project directory and now moved it out to ~/Envs with virtualenvwrapper, but the problem persisted. run.py and test.py are not special in any way, they both import app from atcatalog. I also tried to find ways to omit the virtual environment directory, but google gave no answer (surprisingly). I don't think it is the purpose of coverage to test already well tested site-packages. So I would exclude them from the run.

How can I accomplish to avoid coverage having testing my site-packages?

oz123
  • 27,559
  • 27
  • 125
  • 187
Johannes Maria Frank
  • 2,747
  • 1
  • 29
  • 38
  • I would try using py.test with coverage. – tknickman Aug 22 '15 at 20:38
  • Of course it is an option to switch to a different testing framework. But that doesn't explain what is going wrong here. None of the tutorials showed this behavior. And there are no records of other people having this problem. I am probably doing something stupid wrong. – Johannes Maria Frank Aug 22 '15 at 20:45
  • When you moved the virtual env, did you delete the old one? What is the result of ls -a in workspace/ – tknickman Aug 22 '15 at 20:47
  • Yes the old is deleted. I also got rid of the .coverage file. Also the output shows coverage is using my new virtual environment which is clearly outside my project directory. – Johannes Maria Frank Aug 22 '15 at 20:50
  • Have you given this a read? http://coverage.readthedocs.org/en/coverage-4.0b1/source.html#source – tknickman Aug 22 '15 at 20:53
  • Yes, that what puzzles me. From the docs: When running your code, the coverage run command will by default measure all code, unless it is part of the Python standard library. – Johannes Maria Frank Aug 22 '15 at 20:57
  • Strange, flask and werkzeug are definitely in the psl. Did you install with pip? – tknickman Aug 22 '15 at 21:01
  • Ok, your comments made me solve the problem. Flask and Werkzeug are not part of the standard library. So I tried coverage run --source /home/ubuntu/workspace test.py and the problem is gone. It might be ok for professionals, but I think anything under site-packages should not be tested on a standard basis. – Johannes Maria Frank Aug 22 '15 at 21:08
  • Ah, that's really interesting. Yea I would agree with you there. Glad it's fixed! – tknickman Aug 22 '15 at 21:09

4 Answers4

83

Thanks to tknickman I figured it out: Use either

coverage run --source <path to project dir> test.py

or create a configuration file .coveragerc which resides in the directory you run coverage from, with the following content:

[run]
source =
    <path to project dir>

This provides you do not have your virtual environment installed under the project directory. If you have the virtual environment installed under the project dir you can use

coverage run --source <project path> --omit <pattern> test.py

Note that omit wants a file pattern like

~/projectdir/venv/*

instead of a path.

The corresponding .coveragerc would look like this:

[run]
source=
    <path to project dir>
omit=
    <path to project dir>/<name of virtual env>/*

I still think that like packages of the standard library any packages installed under site-packages should not be covered by default.

Community
  • 1
  • 1
Johannes Maria Frank
  • 2,747
  • 1
  • 29
  • 38
  • Hello @JohannesMariaFrank, I have my folder with my python files and inside is venv - was trying to run it, but I do not know what to put as or cause I am running it inside that folder where are the files - so nothing works for me, and if I create the .coveragerc file - still I would not know what to put as - should I put "" - like empty string or? thank you – Radek Sep 02 '18 at 19:44
  • Hi @Radek An absolute path should be perfectly fine, in fact `~/projectdir/venv` is an absolute path as ~ resolves to `/home/radek` or `C:\Users\radek`. – Johannes Maria Frank Sep 05 '18 at 08:57
  • I have did @JohannesMariaFrank as you said and got this error: Can't find '_ _main_ _' module in '/[path-to-my-directory]/venv/include' – Radek Sep 05 '18 at 20:04
  • Hi @Radek sounds like https://stackoverflow.com/questions/24723547/received-cant-find-main-module-in-packagename-with-python-package – Johannes Maria Frank Sep 07 '18 at 22:38
  • btw, if your `.coveragerc` file is in the same location, then do `source=.` `omit=./venv/*` in your `.coveragerc` file – cryanbhu Dec 19 '20 at 12:00
8

Try using py.test and then specifiying your test options in a setup.cfg file. You will need to pip install pytest first.

For example:

[pytest]
norecursedirs = build docs/_build *.egg .tox *.venv
python_files = tests/functional* tests/integration*
addopts =
    #--verbose
    --tb short
    # Turn on --capture to have brief, less noisy output
    # You will only see output if the test fails
    # Use --capture no if you want to see it all or have problems debugging
    --capture fd
    # --capture no
    # show extra test summary info as specified by chars (f)ailed, (E)error,      (s)skipped, (x)failed, (X)passed.
    - rfEsxX
    --junitxml junit.xml
    --cov workspace --cov-report xml --cov-report term-missing

You can read more about configuring py.test here: https://pytest.org/latest/customize.html

tknickman
  • 4,285
  • 3
  • 34
  • 47
  • 5
    This answer doesn't have anything to do with the original question which was about coverage.py and not py.test. – Jupiter Jun 04 '19 at 15:45
8

In your setup.cfg file include:

[coverage:run]
omit=*/site-packages/*,*/tests/*,*/.eggs/*

Or any other folders that show up in your results that you want to hide from coverage.

Marc Maxmeister
  • 4,191
  • 4
  • 40
  • 54
1

If using pytest, you can specify exclusive paths or files to test in setup.cfg (see docs):

[pytest]
# a directory
testpaths = tests

# exact file(s)
python_files = tests/test1.py tests/test2.py

It looks like if you include the python_files and testpaths parameters, then the python_files will only be used.

Evan Siroky
  • 9,040
  • 6
  • 54
  • 73