phpunit testsuite independence

Question

I am running a series of tests in phpunit which exist in separate testsuites, the list of which is controlled by a phpunit configuration file. When the tests are run individually (i.e. not through the configuration file and hence a since testsuite at a time) they pass but, when run together, I get a failure.

On close examination the issue is that each of these testsuite is loading in a framework (via a require_once) and that framework does some internal configuration based on settings at the time of the require_once. It would appear that, between running the testsuites separately listed in the phpunit configuration file, various things persist. In this particular case the framework is already viewed as loaded.

So - is there a way of getting phpunit to execute a sequence of testsuites independently, i.e. equivalently to running phpunit on the testsuites one at a time? (phpunit is being triggered by cruisecontrol on an autotest machine and locally by developers before submissions.) I've tried options such as '--process-isolation' and '--no-globals-backup' without success.

A quick example which illustrates the problem would be a 'constant.php' file:

<?php
if (defined('XYZZY')) define('TEST', 1);
else define('TEST', 2);

a testsuite 'TestOne.php':

<?php
define('XYZZY', "");
require_once('constant.php');
class TestOne extends PHPUnit_Framework_TestCase
{
   public function testOne()
   {
      $this->assertEquals(TEST, 1);
   }
}

a similar testsuite 'TestTwo.php':

<?php
require_once('constant.php');
class TestTwo extends PHPUnit_Framework_TestCase
{
   public function testTwo()
   {
      $this->assertEquals(TEST, 2);
   }
}

and a phpunit configuration file:

<phpunit>
  <testsuites>
    <testsuite name="First">
      <file>./TestOne.php</file>
    </testsuite>

    <testsuite name="Second">
      <file>./TestTwo.php</file>
    </testsuite>
  </testsuites>
</phpunit>

Answer 1

I ran into a similar issue and came across a different way to do it that may not have been available at the time this question was originally asked.

In my case I had to test some legacy code that checked for a global constant and performed a certain action if it was set. That was easy enough, but after defineing the constant in one test, it persisted through the remaining tests. To change this behavior PHPUnit provides @runInSeparateProcess and @preserveGlobalState annotations.

/**
 * @runInSeparateProcess
 * @preserveGlobalState disabled
 */
public function test_presence_of_constant() {
  define('SOME_OVERRIDE', 'true');
  $result = $this->target->legacyMethod();
  $this->assertEquals('thereWasSomeOverride', $result);
}

This works well if you have only a few tests that need to run in isolation.

Answer 2

i'm trying to provide you with some working examples of --process-isolation and no backup global.

Hopefully you are able to make your Tests working with those. If not just leave me a comment and i'll see to get back to you.

I've tested most of this with 3.4.15, and all against HEAD (3.5.x is expected to work too)

The simplest way to do it is:

<?php
class Test extends PHPUnit_Framework_TestCase
{
    public function testOne()
    {
        define('FOO', 'BAR');
        $this->assertEquals('BAR', FOO);
    }

    public function testTwo()
    {
        define('FOO', 'BAZ');
        $this->assertEquals('BAZ', FOO);
    }
}

and produces:

phpunit --process-isolation processTest.php

PHPUnit @package_version@ by Sebastian Bergmann.

..

Time: 0 seconds, Memory: 1.25Mb

OK (2 tests, 2 assertions)

So building from that if you group your test in testsuites or you can run the setup for each testcase (depending on how expensive it is) you could go with something simple like:

<?php
class Test extends PHPUnit_Framework_TestCase
{
    public function setUp() 
    {
        // your bootstrap for testsuite X
        // Maybe put this in a baseclass for that suite ?
    }

    public function testOne()
    {
        $this->assertEquals('BAR', FOO);
    }

In short

Make sure you run your setup code AFTER you entered the Test (and the process isolation)

If everything else fails and you/we can't solve this:

With the runkit extension (not safe for anything even NEAR production, use as a last resort) you can call functions like runkit-constant-remove()

---

Hope this helps a litte. If not: let me know