<?xml version='1.0' ?>
<!DOCTYPE sect1 PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
"http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" [
]>
<sect1 id="requirements">
  <title>Requirements</title>
  <authorblurb>
    <para>by <ulink url="mailto:joel@aufrecht.org">Joel Aufrecht</ulink></para>
  </authorblurb>

  <sect2 id="requirements-introduction">
    <title>Introduction</title>
    <para>Automated Testing provides a framework for executing tests of all varieties and for storing and viewing the results.</para>
  </sect2>

  <sect2 id="gatekeeper-functional-requirements">
      <title>Functional Requirements</title>

      <informaltable>
        <tgroup cols="4">
          <thead>
            <row>
              <entry><emphasis role="strong">Req #</emphasis></entry>
              <entry><emphasis role="strong">Status in 5.0</emphasis></entry>
              <entry><emphasis role="strong">Priority for 5.1 (A=required, B=optional)</emphasis></entry>
              <entry><emphasis role="strong">Description</emphasis></entry>
            </row>
          </thead>
          <tbody>
            <row>
              <entry>1</entry>
              <entry>Done</entry>
              <entry>Done</entry>
              <entry><emphasis role="strong">Execute TCL tests</emphasis>.  Execute a sequence of TCL code is executed and determine the correctness of the results.</entry>
            </row>
            <row>
              <entry>1.1</entry>
              <entry>partial</entry>
              <entry>Done</entry>
              <entry><emphasis role="strong">Execute HTTP tests</emphasis>.  Execute tests that can interact with a the webserver via the external, HTTP interface, including retrieving pages, following links, and submitting forms.  (This is partially done in the sense that we can make http calls from tcl api, but there is no framework for doing anything complicated.)</entry>
            </row>
            <row>
              <entry>1.1.1</entry>
              <entry></entry>
              <entry>Done</entry>
              <entry><emphasis role="strong">Execute tclwebtest scripts</emphasis>.  A test can contain tclwebtest commands.  If tclwebtest is not installed, those commands fail gracefully.</entry>
            </row>
            <row>
              <entry>1.1.1.1</entry>
              <entry>partial</entry>
              <entry>A</entry>
              <entry><emphasis role="strong">tclwebtest is easy to install</emphasis>.  Tclwebtest installation is fully documented and can be installed with less than five steps.  (Install is documented in 5.0, but there's a can't-find-config error; also, some new work in tclwebtest HEAD needs to packaged in a new tarball release.)</entry>
            </row>

            <row>
              <entry>2</entry>
              <entry>Done</entry>
              <entry>Done</entry>
              <entry><emphasis role="strong">Tests have categories</emphasis>.  Individual tests can be marked as belonging to zero, one, or many of these categories.  The UI provides for running only tests in selected categories, and for viewing only results of tests in selected categories.</entry>
          </row>
            

            <row>
              <entry>2.1</entry>
              <entry></entry>
              <entry>A</entry>
              <entry>Each test can be associated with a single OpenACS.org bug (ie, store bug id as in integer, or store full url so that this can point to other bugs)</entry>
            </row>

            <row>
              <entry>3</entry>
              <entry></entry>
              <entry>B</entry>
              <entry><emphasis role="strong">Tests can be ordered lists of other tests</emphasis>. minimal: verify that a test proc can call other test procs.  Better: A test can be created within the GUI by selecting other tests.  This test is stored in the database and can be exported.  (This is related to a bigger issue of storing test scripts in some format other than tcl procs.)</entry>
            </row>

            <row>
              <entry>4</entry>
              <entry></entry>
              <entry>C</entry>
              <entry><emphasis role="strong">Test scripts can be imported and exported</emphasis>.  It should be possible to import a test into the database from a file, and to export it to a file.  These files should be sharable by different OpenACS installations.  It should be possible to import/export directly between running OpenACS sites.  (We should look at what did and didn't work in acs-lang catalog files and work from there.)</entry>
            </row>

            <row>
              <entry>5</entry>
              <entry></entry>
              <entry>B</entry>
              <entry><emphasis role="strong">Macro Recording</emphasis>.  End users can create and run tests from the web interface without writing code.
                  <para>1) UI to turn on macro mode.</para>
  		      	  <para>2) basic recording: when you fill out a form while macro mode is on, the submit is caught and displayed as tclwebtest code, and then executed.</para>
		      	  <para>3) UI for creating aa_true tests automatically, based on the content of the page.  (For example, a form that says "the returned page must contain [ type regexp here] that spits out aa_true "test X" [string regexp blah blah]</para>
                        </entry>
            </row>
            <row>
              <entry>6</entry>
              <entry></entry>
              <entry>A</entry>
              <entry>Notification subscriptions are available for "email me whenever this test fails" and "notify me whenever a test in this category fails"</entry>
            </row>
            <row>
              <entry>7</entry>
              <entry></entry>
              <entry>A</entry>
              <entry>The results of an automated test are optionally written to an xml file.</entry>
          </row>
          </tbody>
        </tgroup>
      </informaltable>

      <para>Because the current test package uses in-memory variables instead of database objects to track its tests, it is incompatible with the standard category package.  It uses an internal, single-dimension category field.  Should this eventually get extended, a more complete list of categories to implement could be:</para>
<programlisting>Testing Mode
  Regression
  Smoke
  Stress
  Default-Only (for tests, such as front page UI tests, that will break 
                once the default site is modified and can be ignored on 
                non-default sites)
  production-safe
  security_risk
Layer
  Web Page
  TCL Page Contract
  TCL API
  SQL  
Severity (derives from ns_log values)
  Notice (use this for informational items that do not imply a problem)
  Warning (use this for submitted tests w/o fixes; hygiene tests such as deprecated function sweeps)
  Error (default severity)
  Test Validity Bug (use this for problems that suggest the test itself in invalid or broken)
Test Data
  Self-contained  Requires no test data, leaves no test data.
  Populate  Generates and leaves test data, for other tests or for end users.
Package Version
  5.0.0
  etc
</programlisting>

    </sect2>


	<sect2>
	    <title>References</title>
	    	<itemizedlist>
	    	    	<listitem>
	    	<para>Forum Posting: <ulink url="http://openacs.org/forums/message-view?message_id=150581">tclwebtest with openacs-4/etc/install tests -- help getting started</ulink></para></listitem>
			<listitem><para>Forum Posting: <ulink url="http://openacs.org/forums/message-view?message_id=153265">Berlin bug bash proposal</ulink></para>	    </listitem>
	    	</itemizedlist>
</sect2>      
    <sect2 id="revisions-history">
      <title>Revision History</title>
      <informaltable>
        <tgroup cols="4">
          <thead>
            <row>
              <entry><emphasis role="strong">Document Revision #</emphasis></entry>
              <entry><emphasis role="strong">Action Taken, Notes</emphasis></entry>
              <entry><emphasis role="strong">When?</emphasis></entry>
              <entry><emphasis role="strong">By Whom?</emphasis></entry>
            </row>
          </thead>
          <tbody>
            <row>
              <entry>1</entry>
              <entry>Creation</entry>
              <entry>17 Jan 2004</entry>
              <entry>Joel Aufrecht</entry>
            </row>
            <row>
              <entry>2</entry>
              <entry>Updated with notes from chat meeting</entry>
              <entry>21 Jan 2004</entry>
              <entry>Joel Aufrecht</entry>
            </row>
          </tbody>
        </tgroup>
      </informaltable>
    </sect2>
  </sect1>