29

I use coveralls in combination with coverage.py to track python code coverage of my testing scripts. I use the following commands:

coverage run --parallel-mode --source=mysource --omit=*/stuff/idont/need.py ./mysource/tests/run_all_tests.py
coverage combine
coveralls --verbose

This works quite nicely with the exception of multiprocessing. Code executed by worker pools or child processes is not tracked.

Is there a possibility to also track multiprocessing code? Any particular option I am missing? Maybe adding wrappers to the multiprocessing library to start coverage every time a new process is spawned?

EDIT:

I (and jonrsharpe, also :-) found a monkey-patch for multiprocessing.

However, this does not work for me, my Tracis-CI build is killed almost right after the start. I checked the problem on my local machine and apparently adding the patch to multiprocessing busts my memory. Tests that take much less than 1GB of memory need more than 16GB with this fix.

EDIT2:

The monkey-patch does work after a small modification: Removing the config_file parsing (config_file=os.environ['COVERAGE_PROCESS_START']) did the trick. This solved the issue of the bloated memory. Accordingly, the corresponding line simply becomes:

cov = coverage(data_suffix=True)
SmCaterpillar
  • 6,683
  • 7
  • 42
  • 70
  • Do you not test the code for those child processes directly? – jonrsharpe Feb 03 '15 at 11:14
  • Well, yes most of it I do. But there are certain parts that are only useful and are only executed in case multiprocessing is used (like wrapping data base access with locks or a multiprocessing queue to enforce serial data storage). And I know myself that this code is working due to the successful tests. It just would be nice if this would also show up on coveralls :-) – SmCaterpillar Feb 03 '15 at 11:19
  • 1
    See https://bitbucket.org/ned/coveragepy/issue/117/enable-coverage-measurement-of-code-run-by, via http://nedbatchelder.com/code/coverage/trouble.html – jonrsharpe Feb 03 '15 at 11:51
  • Thanks, I stumbled across this as well. However, the monkey-patch does not work for me. Adding this to my script kills my Travis-CI built almost instantly. I checked this on my local machine as well. Apparently, the monkey patch busts my memory. Coverage allocates more than 16GB of memory for tests that usually need much less than 1GB. – SmCaterpillar Feb 03 '15 at 12:54
  • 1
    @SmCaterpillar I'm very interested to hear about your experiences. The idea of removing the config file parsing seems very odd: I can't imagine how parsing the config file would radically change the memory footprint. What is the value of COVERAGE_PROCESS_START for you? Do you have a .coveragerc file? Drop me an email if you want to dig into it. – Ned Batchelder Feb 04 '15 at 22:11
  • Hey, nope I am not using a coveragerc file. So I rechecked and, accordingly, the ``config_file=os.environ['COVERAGE_PROCESS_START']`` throws a key error. However, the main program continues to run and spawns new processes. Maybe this was the reason why the memory was garbled? – SmCaterpillar Feb 05 '15 at 15:10

3 Answers3

30

Coverage 4.0 includes a command-line option --concurrency=multiprocessing to deal with this. You must use coverage combine afterward. For instance, if your tests are in regression_tests.py, then you would simply do this at the command line:

coverage run --concurrency=multiprocessing regression_tests.py
coverage combine
M.javid
  • 6,387
  • 3
  • 41
  • 56
David Ketcheson
  • 4,035
  • 2
  • 30
  • 25
  • 2
    Thanks for pointing out the requirement to use `coverage combine` afterward. I had been spinning my wheels for a while trying to figure out why the `concurrency=multiprocessing` that I had in my `.coveragerc` file wasn't working. – Mike Jarvis Sep 19 '16 at 17:14
  • Could you add a simple basic example to this answer? It would help me out a lot. – Luca Cappelletti Dec 03 '18 at 07:12
  • If you're using the [`pytest-cov`](https://pypi.org/project/pytest-cov/) plugin you'll need specify `concurrency = multiprocessing` in one of the config location (`pyproject.toml`, `.converagerc`, etc) in the [`[run]`](https://coverage.readthedocs.io/en/latest/config.html#run-concurrency) section. – Paul P Feb 16 '23 at 18:04
2

I've had spent some time trying to make sure coverage works with multiprocessing.Pool, but it never worked.

I have finally made a fix that makes it work - would be happy if someone directed me if I am doing something wrong.

https://gist.github.com/andreycizov/ee59806a3ac6955c127e511c5e84d2b6

Andrey Cizov
  • 695
  • 10
  • 20
  • 1
    I believe this link here might help you: http://pytest-cov.readthedocs.io/en/latest/mp.html#ungraceful-process-shutdown - it seems you need to call join in the threads after to avoid a few issues. – Davis Z. Cabral Jun 18 '18 at 01:55
1

One of the possible causes of missing coverage data from forked processes, even with concurrency=multiprocessing, is the way of multiprocessing.Pool shutdown. For example, with statement leads to terminate() call (see __exit__ here). As a consequence, pool workers have no time to save coverage data. I had to use close(), timed join() (in a thread), terminate sequence instead of with to get coverage results saved.

Andrey Kuznetsov
  • 29
  • 5
  • Yep, I had the same issue with using `with`. I had a blocking operation (`starmap`) so I just did a `close` then `join` inside the `with` block. – C S Jan 19 '23 at 20:23