Test Runner
===========

Simple Usage
------------

The test runner consists of an importable module.  The test runner is
used by providing scripts that import and invoke the `run` method from
the module.  The `testrunner` module is controlled via command-line
options.  Test scripts supply base and default options by supplying a
list of default command-line options that are processed before the
user-supplied command-line options are provided.

Typically, a test script does 2 things:

- Adds the directory containing the zope package to the Python
  path.

- Calls the test runner with default arguments and arguments supplied
  to the script.

  Normally, it just passes default/setup arguments.  The test runner
  uses `sys.argv` to get the user's input.

This testrunner_ex subdirectory contains a number of sample packages
with tests.  Let's run the tests found here. First though, we'll set
up our default options:

    >>> import os.path
    >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex')
    >>> defaults = [
    ...     '--path', directory_with_tests,
    ...     '--tests-pattern', '^sampletestsf?$',
    ...     ]

The default options are used by a script to customize the test runner
for a particular application.  In this case, we use two options:

path
  Set the path where the test runner should look for tests.  This path
  is also added to the Python path.

tests-pattern
  Tell the test runner how to recognize modules or packages containing
  tests.

Now, if we run the tests, without any other options:

    >>> from zope.testing import testrunner
    >>> import sys
    >>> sys.argv = ['test']
    >>> testrunner.run(defaults)
    Running unit tests:
      Ran 192 tests with 0 failures and 0 errors in N.NNN seconds.
    Running samplelayers.Layer1 tests:
      Set up samplelayers.Layer1 in N.NNN seconds.
      Ran 9 tests with 0 failures and 0 errors in N.NNN seconds.
    Running samplelayers.Layer11 tests:
      Set up samplelayers.Layer11 in N.NNN seconds.
      Ran 34 tests with 0 failures and 0 errors in N.NNN seconds.
    Running samplelayers.Layer111 tests:
      Set up samplelayers.Layerx in N.NNN seconds.
      Set up samplelayers.Layer111 in N.NNN seconds.
      Ran 34 tests with 0 failures and 0 errors in N.NNN seconds.
    Running samplelayers.Layer112 tests:
      Tear down samplelayers.Layer111 in N.NNN seconds.
      Set up samplelayers.Layer112 in N.NNN seconds.
      Ran 34 tests with 0 failures and 0 errors in N.NNN seconds.
    Running samplelayers.Layer12 tests:
      Tear down samplelayers.Layer112 in N.NNN seconds.
      Tear down samplelayers.Layerx in N.NNN seconds.
      Tear down samplelayers.Layer11 in N.NNN seconds.
      Set up samplelayers.Layer12 in N.NNN seconds.
      Ran 34 tests with 0 failures and 0 errors in N.NNN seconds.
    Running samplelayers.Layer121 tests:
      Set up samplelayers.Layer121 in N.NNN seconds.
      Ran 34 tests with 0 failures and 0 errors in N.NNN seconds.
    Running samplelayers.Layer122 tests:
      Tear down samplelayers.Layer121 in N.NNN seconds.
      Set up samplelayers.Layer122 in N.NNN seconds.
      Ran 34 tests with 0 failures and 0 errors in N.NNN seconds.
    Tearing down left over layers:
      Tear down samplelayers.Layer122 in N.NNN seconds.
      Tear down samplelayers.Layer12 in N.NNN seconds.
      Tear down samplelayers.Layer1 in N.NNN seconds.
    Total: 405 tests, 0 failures, 0 errors
    False

we see the normal testrunner output, which summarizes the tests run for
each layer.  For each layer, we see what layers had to be torn down or
set up to run the layer and we see the number of tests run, with
results.

The test runner returns a boolean indicating whether there were
errors.  In this example, there were no errors, so it returned False.

(Of course, the times shown in these examples are just examples.
Times will vary depending on system speed.)
