








                         TTeessttiinngg??  WWhhaatt tteessttiinngg??

                               _P_e_t_e_r _M_i_l_l_e_r
                           Platypus Technology


                                 _A_B_S_T_R_A_C_T


            This  paper  presents  a  simplistic  yet powerful
            model of what a test is.  When you intend to  test
            your software, you have to design your software to
            be test_a_b_l_e.  This paper will  examine  attributes
            of  software implied by this model.  Some examples
            of automated testing will be given.





       11..  WWhhaatt iiss aa tteesstt??



       The core thesis of this paper  is  the  idea1  that  a  test
       consists  of  three  things:  a system in a defined state, a
       defined transaction, and  a  confirmation  that  the  system
       arrives in a defined state.
       
                          initial-------destination
                           statetransactionstate



       This   is   an  overly  simplistic  statement,  but  remains
       remarkable useful.  The  "system"  under  test  could  be  a
       simple object, a collection of interrelated objects, a whole
       application,  or  a  distributed  multi-layer  client-server
       system.   Equally, the transaction could be a single byte of
       input, a single edge of a state  transition  diagram,  or  a
       series  of  transactions  lumped  together as a single event
       being considered.

       Confirming that the system  under  test  has  arrived  in  a
       particular  state  can be done in may ways.  Some states are
       clearly  visable,  sometimes  they  are  available  but  not
       useful,   and   some   internal  states  are  not  for  user

       ____________________

       1. There is a growing body of knowledge called  "Transaction
          Based    testing"   or   sometimes   "Transaction   Based
          Verification".

       Testing?  What testing? Peter Miller                  Page 1





                                   - 2 -



       consumption and are much  harder  to  access  and  therefore
       harder to confirm.
       
                          initial      destination
                           state          state



                                          _o_o_p_s



       Please  note that this is a _s_i_m_p_l_i_s_t_i_c definition of a test.
       It does not cover all forms of testing  (such  as  tests  of
       usability,  maintainability,  portability, robustness and so
       on  which  make  up  the   other   zillion   software   sub-
       characteristics  listed in ISO 9126) and it is no substitute
       for a well thought out test plan.  It does, however, provide
       some language for talking about functional testing.


       22..  MMaannuuaall tteessttiinngg iiss nnoo tteessttiinngg



       Humans  are really bad at boring, repetitive tasks.  If your
       test plan  is  based  on  the  idea  that  your  staff  will
       faithfully  execute  a long list of printed instructions, at
       least once per release, then your testing  is  probably  not
       effective.

       For  example,  many manual test plans contain long sequences
       of things  the  operator  is  required  to  do,  often  with
       information  on the screen to be confirmed as correct.  This
       is all very well for successful tests, but what happens when
       one  fails?  Usually, these test scripts cover large numbers
       of behaviors.  There is thus a motivation  to  complete  the
       rest  of  the  script,  rather than stop, and have to do the
       start of the script again when the software has been  fixed.
       
                                          +
                                           +
                                           +
                                           +



       There  are  two  themes  here:  (a)  testers  have  to  look
       "productive" or they might not get paid, and (b) redoing the
       first bit again and again is boring.

       Let's  look  at  that  definition again, rephrasing what our
       manual test scripts are doing.  "Usually, these test scripts
       start  from  a defined state, and define a transaction and a

       
       Testing?  What testing? Peter Miller                  Page 2





                                   - 3 -



       confirmations  of  the  destination  state,  then  the  next
       transaction   and  confirmation,  _a_d  _n_a_u_s_e_u_m."   Now,  what
       happens when one of those  confirmations  fails?   Well,  we
       know  it's  in  _t_h_e  _w_r_o_n_g _s_t_a_t_e, so going on to execute the
       rest of the script, we are no longer fulfilling the  initial
       portion  of  our  three-part  definition:  we  aren't in the
       defined state that the transaction  is  to  be  applied  to.
       After  the  first  failure,  the  rest of the results are _n_o
       _i_n_f_o_r_m_a_t_i_o_n.
       
                                     _o_o_p_s










       For effective testing, then, you need something that is very
       good  at  accurately repeating the same script over and over
       again, and  reporting  very  promptly  when  something  goes
       wrong.   Computers  are  very  good  at  boring, repetitious
       tasks.  They don't complain when you ask  them  to  run  the
       same stupid scripts tens or even thousands of times.  And if
       the script breaks, they stop.  For effective testing,  then,
       you need automated testing.  Let the humans _w_r_i_t_e the tests,
       and let the computers _r_u_n the tests.


       33..  SSooffttwwaarree AAttttrriibbuutteess



       Automated testing requires the ability to automatically  get
       the  system  under test into a defined state, the ability to
       automatically apply one or more transaction, and the ability
       to automatically confirm the current state (either read-and-
       compare, or write-and-diff, usually).

       Some things are easy to test, e.g.
        cat > test.in
        cat > test.sed
        cat > expected-output
        sed-clone -f test.sed test.in \
            > test.out
        diff expected-output test.out

       But some things require some specific  changes  to  get  the
       three  properties.   _E_._g_.  a virtual machine simulator needs
       the ability to set registers and stack, _e_t_c,  and  later  to
       dump  them do they can be confirmed.  This may be observable

       
       Testing?  What testing? Peter Miller                  Page 3





                                   - 4 -



       _e_._g_.  as  some  interesting  opcodes  only  present  in  the
       simulator,  and  not  the  real  machine,  maybe  to get the
       simulator to exit with a success/fail indicator.

       33..11  IInniittiiaall SSttaattee

       The system under test needs a way to be  placed  in  a  well
       defined initial state.  This is something that most programs
       are reasonably good at.  Word processors can  load  a  file,
       image processing systems can load an image, databases can be
       created and populated with test sets, _e_t_c.

       It was mentioned above that transactions can actually  be  a
       series of transactions.  Sometimes, getting the system under
       test into a defined state requires starting from the default
       state  and  applying a series of known-to-work transactions.
       Provided that  you can _g_e_t the  system  under  test  into  a
       defined state automatically, it can be tested automatically.

       33..22  TTrraannssaaccttiioonnss

       Automating transactions can often be  the  hardest  part  of
       automated  testing.   Usually,  this  means  automating  the
       simulation of input.  This could be user input, or a network
       connection,   or  a  hardware  simulation  for  an  embedded
       application.

       _3_._2_._1  _C_o_m_m_a_n_d _L_i_n_e

       The design  of  UNIX  makes  the  testing  of  command  line
       programs  relatively  simple, because you can redirect input
       from a file.  This means that you  don't  actually  need  to
       change your software (or not much, anyway).

       _3_._2_._2  _F_u_l_l _S_c_r_e_e_n

       Full-screen  programs  are  often  similar, with input again
       directed from a file, although  you  may  need  to  make  it
       tolerant  of  non-tty  input possibly under the control of a
       command line option.  The trickier cases can be handled with
       _e_x_p_e_c_t.

       _3_._2_._3  _G_U_I

       On the other hand GUI interfaces are harder.  There are some
       utilities, such as _T_k_R_e_p_l_a_y which help.  But they lead us to
       looking  at the problem differently: where can we inject the
       input?
       We can inject it into the X server (or have a fake X  server
       which exists solely to provide test input).
       We  can  proxy  the  X  server, and inject the input via the
       proxy.
       We can inject it into the event  loop  of  our  application.
       This, of course, requires changing the system under test.

       
       Testing?  What testing? Peter Miller                  Page 4





                                   - 5 -



       We  can  have  alternate  input classes, a "real" one and an
       "automated" one.  This, of course,  means  that  the  "real"
       input  class  doesn't get tested, but the rest of the system
       does, and that may be enough.

       _3_._2_._4  _C_l_i_e_n_t _S_e_r_v_e_r

       Most of the techniques useful for X programs work for client
       server   systems  as  well.   Fake  clients,  fake  servers,
       proxies, alternative input classes, _e_t_c.

       _3_._2_._5  _O_b_s_e_r_v_a_t_i_o_n

       In order to test the system, some aspect of it was  changed.
       Auxiliary  test support, more tolerant input, multiple input
       sources.

       33..33  VVeerriiffyy SSttaattee

       Some programs, such as the  _s_e_d  example  given  above,  are
       relatively  easy to test.  Many programs store a significant
       amount of state when you save to a file,  and  this  may  be
       compared  with  _d_i_f_f(1)  or _c_m_p(1).  Other systems, however,
       are more challenging.

       _3_._3_._1  _F_u_l_l _S_c_r_e_e_n

       Many _c_u_r_s_e_s(3) programs need a special command to  dump  the
       screen into a text file for comparison using _d_i_f_f(1).  It is
       also possible to use _e_x_p_e_c_t in many cases.

       _3_._3_._2  _G_U_I

       Many of the input solutions also work for  output,  but  you
       will probably need special commands or options to get screen
       dumps at strategic moments, for comparison.

       Wholesale capture and comparison of  the  output  stream  is
       problematic,  usually  because of gratuitous differences not
       relevant to the test.

       _3_._3_._3  _C_l_i_e_n_t _S_e_r_v_e_r

       You can use bogus clients, bogus servers, or clever proxies.

       _3_._3_._4  _O_b_s_e_r_v_a_t_i_o_n

       In  order to test the system, some aspect of it was changed.
       Auxiliary test support,  captured  output,  multiple  output
       destinations.


       44..  DDiissccuussssiioonn


       
       Testing?  What testing? Peter Miller                  Page 5





                                   - 6 -





       There  are  some  things  which  arise from consideration of
       these ideas.

       44..11  NNoo RReessuulltt

       In coming up with a  testing  regime,  it  is  necessary  to
       remember that tests do not simply _p_a_s_s or _f_a_i_l.

       This  is  further  complicated by the inverted sense of some
       tests.  For example, your development  process  may  require
       that  a  bug fix be accompanied by a test which _f_a_i_l_s on the
       unfixed system, and _p_a_s_s_e_s on the fixed system.

       Consider the issues in achieving a necessary  initial  state
       by  applying transactions to an initial state.  What happens
       when  one  of  these  transactions,  which   are   not   the
       transaction under test, _f_a_i_l?  In such a case it can't _f_a_i_l,
       because the bug fix case will give  a  false  _p_o_s_i_t_i_v_e,  but
       equally  it  can't  succeed  because  this  renders the test
       meaningless.

       The solution is to have a  third  result,  often  called  _n_o
       _r_e_s_u_l_t, which when negated still means _n_o _r_e_s_u_l_t.

       Similar   problems   can  occur  with  the  transaction  and
       verification stages of the test.

       44..22  NNeeggaattiivvee TTeessttiinngg

       Some other examples of negative testing will be given  (i.e.
       _d_i_d_n_'_t  arrive  in  the right state, or invalid transactions
       resulting in an invalid state change).

       44..33  WWaattcchh MMee

       A useful facility for creating tests is a "watch  me"  mode.
       This  is a mode or tool or whatnot that allows the system to
       record  inputs  and  output  for  replay  and   confirmation
       (respectively)  at  a  later time.  While this is _n_o_t one of
       the necessary attributes, it is often a useful side  effect.

       44..44  AAsssseerrtt

       This  simple  model of testing gives a different spin on the
       humble assert statement.  The use of assert can  be  thought
       of as verifying that the system is in a particular state, or
       that the transaction (input) is valid.  This is not the kinf
       of  artifact  you  _w_a_n_t  to  see  in  production code; it is
       usually compiled out of production code.




       
       Testing?  What testing? Peter Miller                  Page 6





                                   - 7 -



       44..55  TTrraaccee oonn RReeqquueesstt

       Another thing which is often compiled out of production code
       is  a  variety of tracing macros, which allow you to see the
       state  of  various  portions  of  the  system  as  they  are
       executed.   You  sometimes  see  this in production systems,
       whene there is little performance impact;  it  is  extremely
       useful feature for tech support, as well as testing.


       55..  TTeessttiinngg??  WWhhaatt tteessttiinngg??



       I  once  worked  on an image processing system for which the
       company had partial source, and  the  inner  workings  where
       supplied   as  a  library  from  the  vendor.   One  of  the
       transforms had some trouble, and I  fixed  it,  but  then  I
       wondered  how  I should test it.  How many of us can confirm
       visually that  a  2D  Walsh-Hadamard  transform  has  worked
       correctly?   While  the destination state was visible on the
       screen, giving humans 2 side-by-side pictures  (a  "does  it
       look like this" manual test) you will almost certainly get a
       false positive.  _E_._g_. those "find  10  differences"  cartoon
       pictures on the funnies section of the newspaper.  If humans
       are so bad at spotting _g_r_o_s_s differences, how can we  expect
       them to find one pixel different in a million?  So, I looked
       for the tool to compare two images  and  tell  me  how  many
       pixels  were  different.   _T_h_e_r_e  _w_a_s_n_'_t  _o_n_e_.   How did the
       vendor test their product?

       If you have testability as a requirement of  your  software,
       you  will  write  different software than if testability was
       not a requirement.

       Do  all  the  tools  we  use  every  day  have  these  three
       properties: Can their initial state be loaded automatically?
       Can their transactions be applied automatically?  Can  their
       destination state be confirmed automatically?  If any one of
       these is missing (but usually the last one), what  gives  us
       any confidence that they were tested at all?














       
       Testing?  What testing? Peter Miller                  Page 7


