20

I've got a python program which is tested by running it several times with different inputs, and comparing the outputs against reference results.

I'd like to get code coverage of all the tests combined, so I can see if there are extra sets of inputs I should be using to get complete coverage. I've looked at the coverage module but can't work out how I can make it do this.

Any clues?

oz123
  • 27,559
  • 27
  • 125
  • 187
xorsyst
  • 7,897
  • 5
  • 39
  • 58

2 Answers2

30

If running on the same machine, run it with the -a option, which accumulates coverage data across multiple calls.

Example:

coverage erase

coverage run -a <command> [arguments, ...]

coverage run -a <command> ... # repeat as many times as needed.

coverage report

coverage html

doc: http://coverage.readthedocs.org/en/latest/cmd.html#data-file

Hope this helps.

Rafael Toledo
  • 974
  • 13
  • 19
ahaywood
  • 416
  • 4
  • 3
4

Ned Batchelder's coverage.py has a feature to combine the results of multiple runs, which seems to be exactly what you are looking for.

Sven Marnach
  • 574,206
  • 118
  • 941
  • 841
  • Yes, this is exactly right. You can use this to run across multiple data sets, across multiple versions of Python, or multiple dependency choices (e.g. different versions of middleware or support modules). You can "orchestrate" multiple test runs in a shell script, but you can do that one better: The [tox](https://tox.readthedocs.org/en/latest/) test runner will auto-create pristine virtual environments and run tests in those; highly recommended for anything beyond the simplest testing setup. – Jonathan Eunice Jan 08 '16 at 21:18
  • @opa The OP selected this answer because it apparently helped them. The other answer was added two years later. I'm a bit puzzled why this got you so angry – I'm happy to improve my answers based on friendly suggestions, but when being shouted at I don't really feel like volunteering more time to help people. – Sven Marnach Feb 13 '19 at 21:32
  • @opa I did not complain about your issue with this answer. You do have a point, and I never questioned that. I was rather complaining about the tone of your comment, which is unnecessarily rude. It would have been sufficient to say "You should include the gist of the linked page in this answer, since otherwise it will become useless if that page goes away" or similar. Instead you chose to say that the answer is "godawful", that it is unfortunate that the OP selected it etc, which sounds rather hostile. I will improve the answer when I find some time to look into this. – Sven Marnach Feb 14 '19 at 20:50
  • @SvenMarnach You're right, my hostility wasn't productive. I get particularly frustrated when I see experienced users have made the same mistakes that lead to new users producing poor quality content. – Krupip Mar 19 '19 at 21:59