1

Use Case: I'm trying to create a diagnostic test for my unit testing framework that runs each test case and then returns a list of public methods which have not yet been covered by the test case. The idea is to get increased test coverage by making sure that all public methods of each are tested by at least 1 unit test.

Problem: I'm having trouble detecting if a method from a class has been called. I've tried using the debug_backtrace() function, but it only returns calls from the test framework, not the class method being tested. The only way I can get the method name being run is to actually place the debug_backtrace() function into the method of the tested class, which is an unacceptable option because I would have to have it in every public method of every class.

Is there an alternate way of doing the backtrace to get the tested class method? Or, in a slightly different approach, is there a way to start and stop a listener that will record all methods that are being called (I can parse out anything that doesn't belong)?

Pseudo-code:

runTest($testName){

   //run the test
   //get all recent classes and methods used
   //parse out test framework info
   //compare used public methods to public method list for the tested class

}

Or:

runTest($testName){

   //start listener
   //run the test
   //close listener, get all recent classes and methods used
   //parse out test framework info
   //compare used public methods to public method list for the tested class

}
VirtuosiMedia
  • 52,016
  • 21
  • 93
  • 140
  • 1
    "The idea is to get increased test coverage by making sure that all public methods of each are tested by at least 1 unit test." --- I don't think this metric can represent *anything useful*. Why don't you use xdebug like other (phpunit) testing frameworks do? – zerkms Sep 11 '11 at 06:55
  • @zerkms - Can you explain why you don't think the metric is useful? The point isn't to imply that a method is completely covered by having at least 1 unit test, rather, it's meant to point out which public methods don't have *any* coverage at all. As for xdebug, I'd prefer not to force users to install an additional extension if I don't have to. – VirtuosiMedia Sep 11 '11 at 07:02
  • "any call" is not useful metric for me. Method can be 100 lines of length but covered on 1% (just first line with `if ($bla) return;`). – zerkms Sep 11 '11 at 07:21

2 Answers2

2

See Code Coverage tools for PHP, Xdebug 2: Code Coverage Analysis and Sebastian Bergmann's PHP_CodeCoverage for some common ways to address your issue.

What you want is usually done watching php from the outside. Using debug_backtrace() is both slow and, as you said yourself, can't be fully relied on since you really want a watcher that does not modify any behavior of your php script (for example including your test methods in the backtrace).

Community
  • 1
  • 1
chelmertz
  • 20,399
  • 5
  • 40
  • 46
  • Thanks for the answer, chelmertz. Do you know of any way to do it without using Xdebug? (It looks like all of those solutions require it). – VirtuosiMedia Sep 11 '11 at 07:12
  • @VirtuosiMedia: No I don't, sorry. I would rely on xdebug though, it is really solid and used by the mainstream. Since it's an "extra" metric, I would be fine with the dependency of xdebug, but I'm not you :) – chelmertz Sep 11 '11 at 07:19
1

See our PHP Test Coverage Tool. It collects test coverage data as your program runs, without use of xdebug. It does so by inserting instrumentation into your code temporarily, e.g., for the duration of the testing process.

You get back test coverage data, displayable superimposed on the code in a nice UI, so you can see what is covered (another OP pointed out the a method might be executed, but not usefully covered). You also get back a report indicating what did not covered, that matches the hierarchy of your code. The XML version of that contains the information you want: such-and-such a method was/was not-covered and to what degree.

Ira Baxter
  • 93,541
  • 22
  • 172
  • 341
  • That tool looks nice but it's an even harder dependency to solve if the purpose of the code is to be released in the wild. Since you took that approach, did you also feel that there is no easy way to do this inside php? – chelmertz Sep 12 '11 at 19:48
  • @chlemertz: Yes and no. We build our tools this way generally out of self-defense; if we don't have to depend on arcane language debugging infrastructure, we don't get stuck when what we need isn't there. In the case of PHP, the presence of Xdebug is iffy; people with code on servers often don't have it and the server manager won't install it. Our solution works whether XDebug is present or not. I'm not sure I understand your objection to "dependency ... if code is to be released to the wild". Our instrumentation is temporary; no need to keep it or ship it. Ship your original code. – Ira Baxter Sep 12 '11 at 20:06
  • your tool looks really good but (in lack of better words) seems to be harder to get than xdebug because it seems to be Windows only and it got a price tag on it. And with the same reasoning, "xdebug works wether PHP Test Coverage Tool is present or not". No offense, +1 – chelmertz Sep 13 '11 at 06:13
  • True. Hard to compete with free (although quality might make a difference). But even Zend makes a living by charging people for stuff. – Ira Baxter Sep 13 '11 at 07:29