6

I have a PHPUnit test suite that is currently causing a fatal error due to a class definition that's not being found. This, ultimately, is a failure of the testing code itself and a failure by the developer to vindicate the test itself before committing code.

However, things like this do happen from time to time, and it would be wonderful if, when a fatal error occurs (regardless of who's ultimately responsible), the test simply be marked as a failure, and the remainder of the test suite still be executed.

I have read about the --process-isolation switch and as far as I can tell, it should take care of this. Since each test runs in a separate process, if the child dies due to a fatal error, the parent can still continue to run. In fact, this is stated explicitly in this answer to a similar question: https://stackoverflow.com/a/5340151/84762 which shows the exact type of output I would like to see myself.

However, I seem to get the exact same output regardless of whether or not I use the --process-isolation flag:

WITHOUT process isolation

[kogi@phagocyte ~]$ /usr/bin/phpunit --colors --verbose --coverage-html "target/coverage" ~/app/zend/tests/application/

PHP Fatal error:  Class 'Rmd_Database_OldObject' not found in /home/kogi/app/zend/private/models/translate/Poll.php on line 9
PHP Stack trace:
PHP   1. {main}() /usr/bin/phpunit:0
PHP   2. PHPUnit_TextUI_Command::main() /usr/bin/phpunit:46
PHP   3. PHPUnit_TextUI_Command->run() /usr/share/pear/PHPUnit/TextUI/Command.php:130
PHP   4. PHPUnit_Runner_BaseTestRunner->getTest() /usr/share/pear/PHPUnit/TextUI/Command.php:150
PHP   5. PHPUnit_Framework_TestSuite->addTestFiles() /usr/share/pear/PHPUnit/Runner/BaseTestRunner.php:96
PHP   6. PHPUnit_Framework_TestSuite->addTestFile() /usr/share/pear/PHPUnit/Framework/TestSuite.php:419
PHP   7. PHPUnit_Util_Fileloader::checkAndLoad() /usr/share/pear/PHPUnit/Framework/TestSuite.php:358
PHP   8. PHPUnit_Util_Fileloader::load() /usr/share/pear/PHPUnit/Util/Fileloader.php:79
PHP   9. include_once() /usr/share/pear/PHPUnit/Util/Fileloader.php:95
PHP  10. require_once() /home/kogi/app/zend/tests/application/translate/PollTest.php:11

Fatal error: Class 'Rmd_Database_OldObject' not found in /home/kogi/app/zend/private/models/translate/Poll.php on line 9

Call Stack:
    0.0003      91584   1. {main}() /usr/bin/phpunit:0
    0.0076     612672   2. PHPUnit_TextUI_Command::main() /usr/bin/phpunit:46
    0.0076     613744   3. PHPUnit_TextUI_Command->run() /usr/share/pear/PHPUnit/TextUI/Command.php:130
    0.0246    1249464   4. PHPUnit_Runner_BaseTestRunner->getTest() /usr/share/pear/PHPUnit/TextUI/Command.php:150
    0.0706    1626680   5. PHPUnit_Framework_TestSuite->addTestFiles() /usr/share/pear/PHPUnit/Runner/BaseTestRunner.php:96
    0.1691    8053584   6. PHPUnit_Framework_TestSuite->addTestFile() /usr/share/pear/PHPUnit/Framework/TestSuite.php:419
    0.1693    8057320   7. PHPUnit_Util_Fileloader::checkAndLoad() /usr/share/pear/PHPUnit/Framework/TestSuite.php:358
    0.1694    8057664   8. PHPUnit_Util_Fileloader::load() /usr/share/pear/PHPUnit/Util/Fileloader.php:79
    0.1711    8240600   9. include_once('/home/kogi/app/zend/tests/application/translate/PollTest.php') /usr/share/pear/PHPUnit/Util/Fileloader.php:95
    0.1805    9187768  10. require_once('/home/kogi/app/zend/private/models/translate/Poll.php') /home/kogi/app/zend/tests/application/translate/PollTest.php:11

WITH process isolation

[kogi@phagocyte ~]$ /usr/bin/phpunit --colors --verbose --coverage-html "target/coverage" --process-isolation ~/app/zend/tests/application/
PHP Fatal error:  Class 'Rmd_Database_OldObject' not found in /home/kogi/app/zend/private/models/translate/Poll.php on line 9
PHP Stack trace:
PHP   1. {main}() /usr/bin/phpunit:0
PHP   2. PHPUnit_TextUI_Command::main() /usr/bin/phpunit:46
PHP   3. PHPUnit_TextUI_Command->run() /usr/share/pear/PHPUnit/TextUI/Command.php:130
PHP   4. PHPUnit_Runner_BaseTestRunner->getTest() /usr/share/pear/PHPUnit/TextUI/Command.php:150
PHP   5. PHPUnit_Framework_TestSuite->addTestFiles() /usr/share/pear/PHPUnit/Runner/BaseTestRunner.php:96
PHP   6. PHPUnit_Framework_TestSuite->addTestFile() /usr/share/pear/PHPUnit/Framework/TestSuite.php:419
PHP   7. PHPUnit_Util_Fileloader::checkAndLoad() /usr/share/pear/PHPUnit/Framework/TestSuite.php:358
PHP   8. PHPUnit_Util_Fileloader::load() /usr/share/pear/PHPUnit/Util/Fileloader.php:79
PHP   9. include_once() /usr/share/pear/PHPUnit/Util/Fileloader.php:95
PHP  10. require_once() /home/kogi/app/zend/tests/application/translate/PollTest.php:11

Fatal error: Class 'Rmd_Database_OldObject' not found in /home/kogi/app/zend/private/models/translate/Poll.php on line 9

Call Stack:
    0.0003      91752   1. {main}() /usr/bin/phpunit:0
    0.0076     612824   2. PHPUnit_TextUI_Command::main() /usr/bin/phpunit:46
    0.0076     613896   3. PHPUnit_TextUI_Command->run() /usr/share/pear/PHPUnit/TextUI/Command.php:130
    0.0246    1250360   4. PHPUnit_Runner_BaseTestRunner->getTest() /usr/share/pear/PHPUnit/TextUI/Command.php:150
    0.0708    1627528   5. PHPUnit_Framework_TestSuite->addTestFiles() /usr/share/pear/PHPUnit/Runner/BaseTestRunner.php:96
    0.1688    8054296   6. PHPUnit_Framework_TestSuite->addTestFile() /usr/share/pear/PHPUnit/Framework/TestSuite.php:419
    0.1690    8057992   7. PHPUnit_Util_Fileloader::checkAndLoad() /usr/share/pear/PHPUnit/Framework/TestSuite.php:358
    0.1691    8058336   8. PHPUnit_Util_Fileloader::load() /usr/share/pear/PHPUnit/Util/Fileloader.php:79
    0.1707    8241296   9. include_once('/home/kogi/app/zend/tests/application/translate/PollTest.php') /usr/share/pear/PHPUnit/Util/Fileloader.php:95
    0.1801    9188464  10. require_once('/home/kogi/app/zend/private/models/translate/Poll.php') /home/kogi/app/zend/tests/application/translate/PollTest.php:11

For those of us that can't effectively diff in our heads, the two outputs are literally identical (other than the execution time and memory usage which is negligibly different).

In both cases, the fatal error kills the entire test suite. In this particular case, this happens in the 3rd test and the remaining 150 tests (in several other files/suites) are never executed.

What am I doing wrong here? Is there some other way to survive a fatal error (marking the test as failed) in one test and still execute remaining tests?

EDIT

I am using PHPUnit 3.6.10

EDIT

Comments on the answer to this question have inspired a new ticket on PHPUnit's GitHub page: https://github.com/sebastianbergmann/phpunit/issues/545

Community
  • 1
  • 1
KOGI
  • 3,959
  • 2
  • 24
  • 36
  • You'll find your answer here: http://stackoverflow.com/questions/277224/how-do-i-catch-a-php-fatal-error – Nir Alfasi Apr 09 '12 at 17:57
  • 1
    I'm not sure if you're referring to the part that says they can't be caught or the part that says they shouldn't be caught... but both statements are false. In this case, when testing the functionality of a system, it is very important to catch fatal errors without killing the rest of the testing process. If I am misunderstanding your answer, I apologize, please let me know. – KOGI Apr 09 '12 at 18:05
  • I was referring to the part that says it can't be caught. It's true you can use register_shutdown_function in order to print stuff to log and do so "clean ups" but since the frame was already destroyed (on the stack) I don't believe you can "resume" your program. Since you know exactly on which line the fatal error is triggered - why not fix it ? – Nir Alfasi Apr 09 '12 at 20:44
  • Thanks for the clarification. The point is not to fix it. That's trivial in this case. The point is to prevent the failure to run the remaining tests due to a single developer's mistake in committing. If someone commits a bad test, it shouldn't disable the whole system. I understand that fatal errors are... fatal. You can't recover from them. But as I understand it, PHPUnit's --process-isolation function forks each individual test function into a separate child process. If that child has a fatal error, it cannot recover, but the parent can still mark it as a failed test and proceed to the next – KOGI Apr 09 '12 at 20:53
  • Now I understand :) according to this: http://stackoverflow.com/questions/3841190/phpunit-fatal-error-handling you should start **each test suite** in a new process hope it helps. – Nir Alfasi Apr 09 '12 at 21:53
  • What that post is saying is that using `--process-isolation` *causes* phpUnit to start each test suite in a separate process. The only thing I am supposed to do is invoke phpUnit once from the CLI and it does the rest of the magic. – KOGI Apr 09 '12 at 22:37
  • have a loot at this: http://matthewturland.com/2010/08/19/process-isolation-in-phpunit/ – Nir Alfasi Apr 09 '12 at 23:11
  • Thank you for your help. Believe it or not, I *have* Googled this. I think you're starting to grasp at straws. – KOGI Apr 09 '12 at 23:37
  • didn't mean to disrespect - only tried to help. good luck with your search - please update the thread if you find an answer - I'm really curious about it by now :) – Nir Alfasi Apr 10 '12 at 04:59

1 Answers1

6

PHPUnit loads each test file that will be run before running any tests. This causes PHP to parse these files and execute their top-level code. If any class is loaded that, for example, extends a class that doesn't exist, you'll get a fatal error.

I don't see any way around this without enhancing PHPUnit to parse the files without executing their code during the scanning process.

David Harkness
  • 35,992
  • 10
  • 112
  • 134
  • 1
    Is this still the case when using `--process-isolation`? – KOGI Apr 10 '12 at 21:55
  • Yes, I ran a test by changing a working test to extend a misnamed class, and it stops even with isolation. – David Harkness Apr 10 '12 at 23:41
  • Well damn! Back to square one! Thank you. – KOGI Apr 11 '12 at 03:43
  • This is mostly a design consequence of PHP. In this language certain errors are deemed "fatal" that are merely "exceptional" in others. A `ClassNotFoundException` in Java can be caught, logged, and *bypassed* whereas this error is irrecoverable in PHP. We've had to override some of PHPUnit's behavior to allow testing to continue in the face of otherwise fatal errors, e.g. `@covers` annotations that reference invalid classes/methods. – David Harkness Apr 11 '12 at 04:25
  • Thank you for the info. I wonder if the author would be willing to have PHPUnit do the loading of each test file in the child process rather than in the parent... The parent could find all files to use, then spawn child processes (if using `--process-isolation`) to both load the file (parse/exec top-level code) and run the tests. This would allow all tests to run even if there are fatal errors within some of them. – KOGI Apr 11 '12 at 19:18
  • For those interested, I've opened a new ticket on PHPUnit's GitHub page: https://github.com/sebastianbergmann/phpunit/issues/545 – KOGI Apr 11 '12 at 19:24
  • The initial loading of files does at least three things: ensures the file contains a test case, counts the test methods, and calls the `@dataProvider` methods. This is all needed to determine the total number of tests as well as what data to pass to those that need it. – David Harkness Apr 11 '12 at 19:47
  • Even if phpUnit spawns a child process for this purpose, and then spawns child processes for the actual testing, that would solve the problem. May not be worth the performance overhead, though. Although, process isolation is expected to be a performance hit anyway... – KOGI Apr 11 '12 at 21:17