coverage.py does not cover script if py.test executes it from another directory

Question

I got a python script which takes command line arguments, working with some files. I'm writing succeeding tests with py.test putting this script through its paces, executing it with subprocess.call.

Now I want to analyze code coverage with coverage.py. Coverage, when used via the pytest-cov plugin (which has subprocess-handling built-in), does not see/cover my script when it is called from a temporary testing directory created with py.test's tmpdir fixture. Coverage does see my script when it's called in the directory it resides in (and the filename argument points to a remote path).

In both situations, my tests pass! Coverage 3.6, pytest-2.3.5, pytest-cov 1.6, all from PyPi.

Question: How can I get coverage to recognize my script even if it's executed in another directory? Is this a bug in coverage, or something which is just not possible to do? Would be surprised if the latter, after all, tmpdir is a stock mechanism of py.test...

Minimal example:

I got a script my_script.py which just echoes the contents of a file arg_file.txt supplied via command-line argument. In two different tests, this is once called in a tmpdir, and once in the script's location. Both tests pass, but the in the tmpdir test, I get no coverage information!

Test run:

~/pytest_experiment$ py.test -s
=================================== test session starts ====================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 2 items 

tests/test_in_scriptdir.py 
set_up: In directory /tmp/pytest-52/test_10
Running in directory /home/cbuchner/pytest_experiment
Command: ./my_script.py /tmp/pytest-52/test_10/arg_file.txt
--Contents of arg_file.txt--

.
tests/test_in_tmpdir.py 
set_up: In directory /tmp/pytest-52/test_11
Running in directory /tmp/pytest-52/test_11
Command: /home/cbuchner/pytest_experiment/my_script.py arg_file.txt
--Contents of arg_file.txt--

.

================================= 2 passed in 0.06 seconds =================================

Coverage:

~/pytest_experiment$ py.test --cov=my_script.py tests/test_in_scriptdir.py=================================== test session starts ====================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 1 items 

tests/test_in_scriptdir.py .
--------------------- coverage: platform linux2, python 2.7.4-final-0 ----------------------
Name        Stmts   Miss  Cover
-------------------------------
my_script       3      0   100%

================================= 1 passed in 0.09 seconds =================================
~/pytest_experiment$ py.test --cov=my_script.py tests/test_in_tmpdir.py=================================== test session starts ====================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 1 items 

tests/test_in_tmpdir.py .Coverage.py warning: No data was collected.

--------------------- coverage: platform linux2, python 2.7.4-final-0 ----------------------
Name    Stmts   Miss  Cover
---------------------------

================================= 1 passed in 0.09 seconds =================================

The files are here: https://gist.github.com/bilderbuchi/6412754

Edit: Interstingly, when running the coverage tests with -s, too, there's more curious output - coverage warns that No data was collected, when obviously it was collected, and in the tmpdir test warns that Module my_script.py was never imported.??

~/pytest_experiment$ py.test -s --cov=my_script.py tests/test_in_scriptdir.py
=================================== test session starts ====================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 1 items 

tests/test_in_scriptdir.py 
set_up: In directory /tmp/pytest-63/test_10
Running in directory /home/cbuchner/pytest_experiment
Command: ./my_script.py /tmp/pytest-63/test_10/arg_file.txt
--Contents of arg_file.txt--

Coverage.py warning: No data was collected.
.
--------------------- coverage: platform linux2, python 2.7.4-final-0 ----------------------
Name        Stmts   Miss  Cover
-------------------------------
my_script       3      0   100%

================================= 1 passed in 0.09 seconds =================================
~/pytest_experiment$ py.test -s --cov=my_script.py tests/test_in_tmpdir.py=================================== test session starts ====================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 1 items 

tests/test_in_tmpdir.py 
set_up: In directory /tmp/pytest-64/test_10
Running in directory /tmp/pytest-64/test_10
Command: /home/cbuchner/pytest_experiment/my_script.py arg_file.txt
--Contents of arg_file.txt--

Coverage.py warning: Module my_script.py was never imported.
Coverage.py warning: No data was collected.
Coverage.py warning: Module my_script.py was never imported.
Coverage.py warning: No data was collected.
.Coverage.py warning: No data was collected.

--------------------- coverage: platform linux2, python 2.7.4-final-0 ----------------------
Name    Stmts   Miss  Cover
---------------------------

================================= 1 passed in 0.09 seconds =================================

Sorry for non-answer. But it sounds to me that you will have to look inside py.test to see speciality of tmpdir fixture. If it in some sort of context manager, coverage is getting called after it exits (or something like that). — Senthil Kumaran, Sep 07 '13 at 16:57
Thanks. I'm already in contact with the coverage developer, I'll post a summary once we've gotten to the bottom of this. — Christoph, Sep 07 '13 at 19:51
@Christoph if you are still interested in getting this to work with pytest-cov feel free to open an issue on https://github.com/schlamar/pytest-cov. I'm the new maintainer of pytest-cov and might be able to help. — schlamar, Apr 24 '14 at 14:12
@schlamar thanks, but I have already worked around that issue on my side (see my answer below). While I normally would (properly) open a bug report about this, I currently don't have time to care about this and test it again and post a repro procedure, sorry. — Christoph, Apr 25 '14 at 08:59
@schlamar I just checked, and I can still repro using the procedure above, but I'm not even sure anymore if this is actually a slight user error when using coverage to cover stuff called by `subprocess` — Christoph, Apr 25 '14 at 09:24

score 17 · Answer 1 · answered Jan 10 '16 at 14:51

17

I encountered the same issue when calling "py.test --cov ..." from tox. I found a hint on this page: http://blog.ionelmc.ro/2014/05/25/python-packaging/ even though it does not mention this explicitly. Using "--develop" for tox will make sure that coverage data gathering is called from the same directory as coverage analysis. This section in tox.ini made it work for me to have a test environment for coverage:

[tox]
envlist = ...,py34,cov

[testenv:cov]
# necessary to make cov find the .coverage file
# see http://blog.ionelmc.ro/2014/05/25/python-packaging/
usedevelop = true
commands = py.test --cov=<MODULE_NAME>
deps = pytest pytest-cov

answered Jan 10 '16 at 14:51

okurz

271
2
2

This is so far the best answer which just helped me save another hour. Thumbs up for you. – Andrew Dunai Nov 01 '16 at 12:15
2

This makes the testing a bit less thorough, since you are no testing against the dist generated from your setup.py. – augurar Dec 29 '16 at 10:46
This is because your library name doesn't match the module name. – Natim Feb 01 '17 at 10:02

score 5 · Accepted Answer · answered Sep 16 '13 at 09:59

5

This turned out to be a problem of relative paths confusing coverage when the measured script is run from another directory. Coverage result files ended up in that directory, instead of the root directory of the project.

To solve this, I stopped using pytest-cov, and used pure coverage instead. I used full paths instead of relative paths wherever relevant.

So, e.g. define the environment variable necessary to enable subprocess coverage via export COVERAGE_PROCESS_START=/full/path/to/.coveragerc. In the .coveragerc, the coverage result file is specified via

     [run]
     data_file = /full/path/to/.coverage

and any --source and --include options should use full paths, too. Then it was possible to get correct coverage measurement.

answered Sep 16 '13 at 09:59

Christoph

5,480
6
36
61

28

I am afraid that this is not a generally acceptable answer as this cannot work if you have continuous integration, we need a proper solution. – sorin Jan 13 '15 at 11:41
what about CI precludes you from using full paths? – Christoph Jan 14 '15 at 09:58
9

@Christoph: You often don't control (nor should you have to control) the exact location where your sources are built/extracted (whatever build step is relevant). Hence the need to rely on relative paths. – ereOn Apr 09 '15 at 15:22
1

Point taken. Although, you can get at least partly around that by working with environment variables holding full paths that you generate on the fly, via `os.path.abspath()`. In any case, I don't know if this is even still a problem, my old code I used this in doesn't have full paths in `.covragerc`. – Christoph Apr 10 '15 at 10:37

Midnighter · Answer 3 · 2018-01-09T17:16:53.913

0

Another option with tox is to set the PYTHONPATH in tox.ini:

[testenv] setenv = PYTHONPATH = {toxinidir} commands = pytest --cov=<your package> - codecov

edited Jan 09 '18 at 17:16

answered Jan 09 '18 at 16:03

Midnighter

3,771
2
29
43

score 0 · Answer 4 · answered Aug 10 '19 at 02:16

0

according to this blog: https://thomas-cokelaer.info/blog/2017/01/pytest-cov-collects-no-data-on-travis/

You should add all the __init__.py files in tests folder! This solution works for me.

answered Aug 10 '19 at 02:16

Jingpeng Wu

452
4
6

coverage.py does not cover script if py.test executes it from another directory

4 Answers4

Linked