Anatomy of a Horde test suite

Anatomy of a Horde test suite

Introduction

The Horde Project has always had high standards when it comes to code quality. Of course these standards evolved with time and also with the progress the PHP community made. The code from IMP-1.0.0 (1998) didn't come with unit tests. And somehow it lacked classes. And there is an awful lot of code mixed with HTML. Somehow this looks horribly like PHP3.

Oh, it was PHP3.

Of course PHP development changed over time and so did the Horde project. Nowadays each and every commit into our repository leads to the automatic execution of thousands of unit tests written by the Horde developers and they check our code for errors. Night and day our continuous integration server broadcasts the current test status to us in particular but also to anyone else interested.

With the release of Horde 4 the test suite structure of the Horde components available via our PEAR server shows a common pattern. There are certain Do's and Don'ts and a lot of playground in between. Often the Horde_Test component is involved. So it makes sense to associate the overview on the anatomy of Horde test suites with this particular module.

Filesystem structure of a test suite

The following tree shows a hypothetical arrangment of all element types that you might find in one of the Horde component test suites:

horde/framework/Xyz/
`-- test
    `-- Horde
        `-- Xyz
            |-- AllTests.php
            |-- Autoload.php
            |-- fixtures
            |   `-- data.xml
            |-- Integration
            |   `-- IntegrationTest.php
            |-- phpunit.xml
            |-- Server
            |   |-- conf.php
            |   |-- conf.php.dist
            |   `-- ServerTest.php
            |-- SomeTest.php
            |-- Stub
            |   `-- Object.php
            |-- TestCase.php
            `-- Unit
                `-- UnitTest.php

We will go through the different elements below but lets first look at the hierarchy from the top level directory to the level of the AllTests.php file.

The top level directory horde represents the horde-git repository that everyone should be familiar with.

The majority of the test suites can be found in the framework area but by now a good number of applications have their own test suites as well. Whether a framework package or an application: In both cases the test directory in the top-level directory of the component will contain the tests.

Below the test directory the code hierarchy of the component will usually be mirrored.

For a hypothetical framework package with name Horde_Xyz this means that where will be a directory Horde with a subdirectory Xyz. The latter one contains the tests. This matches the hierarchy in the lib directory where you will also find a Horde directory that usually contains a Xyz subdirectory. If not there will at least be an Xyz.php file.

For an application named xyz the test directory will only contain a directory Xyz.

This directory structure below test is just a convention and no functional requirement for the tests. At least at the moment. At some point the idea was that you would be able to unpack all PEAR package archives into one location without the file paths in the components conflicting with each other. But when we rely on PEAR for the installation now then the different test suites of package Horde_One and Horde_Two end up in tests/Horde_One and tests/Horde_Two respectively. So there is no chance for a conflict anyway. This might change at some point though. So we stick to the convention and ask anyone writing new test suites to do it as well.

AllTests.php

This file is the only mandatory requirement for a Horde test suite. Everything else is optional but there has to be an AllTests.php file which serves as an entry point into the test suite. And there are specific requirements for this file.

The functionality expected from AllTests.php is as follows:

It must collect all tests of the test suite.
It must allow to retrieve all tests of the suite via Horde_Xyz_AllTests::suite().
It must allow running the test suite via phpunit AllTests.php.
It must allow running the test suite via php AllTests.php.

The Horde_Test package already delivers a boilerplate AllTests.php class in framework/Test/lib/Horde/Test/AllTests.php and deriving an AllTests.php for a standard test suite becomes rather simple. The full code for this is presented [#alltests below] and you can also look at an example from our repository.

Collecting the tests of a test suite

The following extract from the suite() method of the Horde_Test_AllTests class is responsible for collecting the tests of the suite:

    $basedir = dirname(self::$_file);

    $suite = new PHPUnit_Framework_TestSuite('Horde Framework - ' . self::$_package);
    $baseregexp = preg_quote($basedir . DIRECTORY_SEPARATOR, '/');

        foreach (new RecursiveIteratorIterator(new RecursiveDirectoryIterator($basedir)) as $file) {
            if ($file->isFile() && preg_match('/Test.php$/', $file->getFilename())) {
                $pathname = $file->getPathname();
                if (include $pathname) {
                    $class = str_replace(DIRECTORY_SEPARATOR, '_',
                                         preg_replace(\"/^$baseregexp(.*)\\.php/\", '\\\\1', $pathname));
                    try {
                        $suite->addTestSuite(self::$_package . '_' . $class);
                    } catch (InvalidArgumentException $e) {
                        throw new Horde_Test_Exception(
                            sprintf(
                                'Failed adding test suite \"%s\" from file \"%s\": %s',
                                self::$_package . '_' . $class,
                                $pathname,
                                $e->getMessage()
                            )
                        );
                    }
                }
            }
    }

    return $suite;

This section collects all files ending with *Test.php in the directory that contains self::$_file and all subdirectories thereof. For each such file the path hierarchy is mapped to a class name (e.g. test/Horde/Xyz/Unit/UnitTest.php to Horde_Xyz_Unit_UnitTest). This requires the package prefix in self::$_package (Horde_Xyz) and the path location relative to the directory that contains self::$_file (Unit/UnitTest.php). The .php suffix of that path name will be removed and each / replaced by _ to generate a class name in combination with the prefix. The file will be loaded and assumed to provide a test case with the generated class name. This will be added to the test suite currently being aggregated.

The two required parameters self::$_file and self::$_package have to be set using the init($package, $file) method of the Horde_Test_AllTests class.

Based on the Horde_Test_AllTests class available as boiler plate code nearly all Horde test suite AllTests.php files contain a section similar to this:

require_once 'Horde/Test/AllTests.php';

class Horde_Xyz_AllTests extends Horde_Test_AllTests
{
}

Horde_Xyz_AllTests::init('Horde_Xyz', __FILE__);

This section is already sufficient to support requirements (1), (2) and (3) from above.

Running the test suite

The default method for running the test suite of a Horde component is to run php AllTests.php on the command line. For that to work we need a few additional lines in the AllTests.php file of the Horde_Xyz component.

if (!defined('PHPUnit_MAIN_METHOD')) {
    define('PHPUnit_MAIN_METHOD', 'Horde_Xyz_AllTests::main');
}

require_once 'Horde/Test/AllTests.php';

class Horde_Xyz_AllTests extends Horde_Test_AllTests
{
}

Horde_Xyz_AllTests::init('Horde_Xyz', __FILE__);

if (PHPUnit_MAIN_METHOD == 'Horde_Xyz_AllTests::main') {
    Horde_Xyz_AllTests::main();
}

Horde_Xyz_AllTests::main(); is the critical line that will run the test suite when executing php AllTests.php. The additional define() magic is required to also allow using phpunit AllTests.php.

And finally you can run horde/framework/bin/test_framework which collects all AllTests.php files and aggregates them together in one huge test suite by using the suite() method of each component test suite.

Autoload.php

This file is not required in a test suite but it is strongly recommended that you use it. It's purpose is to setup PHP autoloading so that all tests in the test suite automatically have access to all the classes required for executing the tests. The reason why it is not mandatory is that the the basic Horde_Test_AllTests already loads a basic autoloading definition that works for most framework components.

This means that running php AllTests.php usually does not hit any autoloading problems. Running a single test case (e.g. phpunit Horde/Xyz/Unit/UnitTest.php is a different matter though.

The *Test.php files do not extend Horde_Test_AllTests and thus there is nothing that would magically setup autoloading if you try to run such a test suite in isolation. And running single test cases can be quite convenient if the whole test suite would take a long time to execute. Using an Autoload.php file alongside the AllTests.php file is the recommended way to provide a single test case with autoloading and thus enable commands such as phpunit Horde/Xyz/Unit/UnitTest.php.

Once you created an Autoload.php file for your test suite it will also be heeded by Horde/Test/AllTests.php. The latter will avoid the basic autoloading setup if it detects the presence of an Autoload.php file for the current test suite. That one will be loaded and is assumed to contain the required autoloading setup.

The content of `Autoload.php`

You should at least require the Autoload.php from Horde_Test in this file. This is what Horde_Test_AllTests also does for the test suite wrapper. The code provided by Horde/Test/Autoload.php has its own section further below.

require_once 'Horde/Test/Autoload.php';

It also makes sense to adapt the error reporting level to the same standards as required by the AllTests.php wrapper:

error_reporting(E_ALL | E_STRICT);

If you derive your test cases from a central test case definition you should load this one in Autoload.php as well:

/** Load the basic test definition */
require_once dirname(__FILE__) . '/TestCase.php';

Sometimes it makes sense to pull in the definition of test helpers that may be used throughout the test suite. They are usually not available via autoloading and need to be pulled in explicitely:

/** Load stub definitions */
require_once dirname(__FILE__) . '/Stub/ListQuery.php';
require_once dirname(__FILE__) . '/Stub/DataQuery.php';

Real world examples for Autoload.php helpers can be found in the Horde_Date and the Kolab_Storage components.

Within the test cases you only need to load the Autoload.php file which usually looks like this (and obviously depends on the position of the test case in the directory hierarchy of the test suite):

require_once dirname(__FILE__) . '/../Autoload.php';

Autoload mappings for applications

In case you test a horde application that may have special class name mappings when autoloading you need to define those before loading the Autoload.php helper from Horde_Test. The special $mappings variable accepts a class name prefix (e.g. IMP) which you can link to a specific path.

$mappings = array('IMP' => dirname(__FILE__) . '/../../lib/');
require_once 'Horde/Test/Autoload.php';

The class IMP_Imap would be loaded from dirname(FILE) . '/../../lib/Imap.php' in the example above.

The code provided by `Autoload.php` in `Horde_Test`

Horde uses autoloading according to PSR-0. Usually the Horde_Autoloader component provides the required functionality. For pure testing purposes we didn't want to add another component as a dependency to the Horde test suites. The required autoloading setup is rather trivial in most cases though. So the short hack in framework/Test/lib/Horde/Test/Autoload.php is all that is required to provide functionality similar to Horde_Autoloader during test runs.

The code in that file checks if autoloading has already been setup first. Certain autoloaders that fail to load the Horde classes are ignored:

$autoloaders = spl_autoload_functions();
if (!empty($autoloaders)) {
    /**
     * Ignore autoloaders which are incapable of loading Horde
     * classes (e.g. PHPUnit >= 3.5.0)
     */
    $autoloaders = array_diff($autoloaders, array('phpunit_autoload'));
}
if (empty($autoloaders)) {
   ...
}

If there is no suitable autoloader yet the code will setup the minimal autoloading variant required for most Horde code to work. It looks like this:

spl_autoload_register(
        create_function(
            '$class',
            '$filename = str_replace(array(\\'::\\', \\'_\\'), \\'/\\', $class);'
            . '$err_mask = error_reporting() & ~E_WARNING;'
            . '$oldErrorReporting = error_reporting($err_mask);'
            . 'include \"$filename.php\";'
            . 'error_reporting($oldErrorReporting);'
        )
    );

Which basically works by replacing "_" in class names to the path hierarchy delimiter "/" and append the ".php" suffix. If that is an existing file name the function will load it.

There is a little bit more code that deals with the application class mappings but it would be overhead to describe it here.

Download this page as: Plain Text, HTML, Latex, reStructuredText

Table of Contents