=================
 IPython testing
=================

Namespaces
==========

At the beginning of each test, compute the globals by hand (copy find from
doctest) and put them into user_ns.  The tests MUST run in user_ns itself, not
somewhere else.  Make a new user_ns for each test to minimize side effects.


Requirements
============

Testing IPython in full requires modifications to the default behavior of nose
and doctest, because the IPython prompt is not recognized to determine Python
input, and because IPython admits user input that is not valid Python (things
like ``%magics`` and ``!system commands``.

We basically need to be able to test the following types of code:

1. Pure Python files containing normal tests.  These are not a problem, since
   Nose will pick them up as long as they conform to the (flexible) conventions
   used by nose to recognize tests.

2. Python files containing doctests.  Here, we have two possibilities:
   - The prompts are the usual ``>>>`` and the input is pure Python.
   - The prompts are of the form ``In [1]:`` and the input can contain extended
     IPython expressions.

   In the first case, Nose will recognize the doctests as long as it is called
   with the ``--with-doctest`` flag.  But the second case will likely require
   modifications or the writing of a new doctest plugin for Nose that is
   IPython-aware.

3. ReStructuredText files that contain code blocks.  For this type of file, we
   have three distinct possibilities for the code blocks:
   - They use ``>>>`` prompts.
   - They use ``In [1]:`` prompts.
   - They are standalone blocks of pure Python code without any prompts.

   The first two cases are similar to the situation #2 above, except that in
   this case the doctests must be extracted from input code blocks using
   docutils instead of from the Python docstrings.

   In the third case, we must have a convention for distinguishing code blocks
   that are meant for execution from others that may be snippets of shell code
   or other examples not meant to be run.  One possibility is to assume that
   all indented code blocks are meant for execution, but to have a special
   docutils directive for input that should not be executed.

   For those code blocks that we will execute, the convention used will simply
   be that they get called and are considered successful if they run to
   completion without raising errors.  This is similar to what Nose does for
   standalone test functions, and by putting asserts or other forms of
   exception-raising statements it becomes possible to have literate examples
   that double as lightweight tests.

4. Extension modules with doctests in function and method docstrings.
   Currently Nose simply can't find these docstrings correctly, because the
   underlying doctest DocTestFinder object fails there.  Similarly to #2 above,
   the docstrings could have either pure python or IPython prompts.


Plan
====

To address all of these, we'll need to:

* Write a proper DocTestFinder object that can correctly identify doctests in
  extension modules and run blocks of code in reST files.

* Then, subclass that one to have one that can correctly load doctests with
  python or ipython prompts.

* Finally, put in the necessary high-level tools for users to run the tests and
  developers to quickly re-test small portions.

Status
======

* Only 3-c (reST with standalone code blocks) is missing.

* We also need an easy way to disable tests: certain files (like those in
  external) should be altogether skipped.  Add --exclude and --exclude-from
  options.

* Add a standalone python script to run all tests (iptest)

* Add a way to use nose.SkipTest in doctests.  We need both a decorator
  skiptest and one skipdoctest.  They do different things.

* Tests are being run twice!  For example::

  nosetests -vs --with-ipdoctest --doctest-tests --doctest-extension=txt
  test_refs.py

runs test_refs and test_refs.test_refs.  I don't know why.
  --> The double counting comes from analyzing the docstrings AND finding the
  test functions as well.  We should probably leave it as is: test functions
  shouldn't have doctests themselves, unless they are ONLY a doctest and
  nothing else.  In that case, just name them something *not* using the word
  'test' and it works fine.

* It's not picking up variables in the execution namespace.  Why??



Other requirements
==================

* Support annotations like #random to mark entire doctests that should be run
  but whose output should be ignored (considered random).

* Support easily a way to mark starting line markers which indicate a pattern
  for lines to skip altogether.  This will allow us to avoid for example
  running any line with calls to ``plt.`` that can't be properly doctested
  since they rely on side effects (figures).  MMh, maybe just a decorator for
  injecting a dummy object might be cleaner.

* Make sure we run tests in .txt files easily.



Nose questions
==============

- Easy way to *add* plugins in a main() call?  If I say plugins=[...], those
  are the only plugins used.  I want to use the defaults and simply add extra
  ones from the main() call, similarly to what --with-foo does at the command
  line.