OMNeT++ Regression Tests
========================

This file documents the opp_test tool. Reading it will help you understand
.test files in the test/ directory and write new test cases for your simulation
models or for the simulation kernel.


Each test case is a single *.test file. All NED code, C++ code and
other data necessary to run the test case, and also PASS criteria are
embedded in the test case file. The advantages for this design are twofold:
 - test cases are self-contained (easy to share, etc)
 - enforces test cases that are compact and to the point

How to run tests (steps 2-4 are automated with the test/runtest script):

1. write the *.test files

  The syntax of test files is described in a later section of this doc.

2. export all source and other files to a working directory

  This is done with the following command:

  % opp_test -g *.test

  This will export all C++ and NED code, ini files, etc. from
  the .test files into separate files. All files will be created
  in the test tool's work directory. (It defaults to ./work,
  but it can be orverridden.

3. create a makefile and build the test program

  You can build an executable test program from the source files
  in the work directory by running:

  % opp_makemake -u cmdenv; make

  The code from all test cases will be combined into a single executable.
  (If there are many test cases, this results in a dramatical
  improvement in terms of necessary disk space and compilation time,
  compared to the case when every test is a separate executable.)

4. execute the tests and see the results

  Typing

  % opp_test -r -v *.test

  will execute all test cases, using the test executable built
  in step 3. For each test case, opp_test will check the output from
  the test program (stdout, stderr and the generated output files)
  against the specification in the test file and decide if it is
  a PASS or FAIL or UNRESOLVED. Finally, it will print a summary
  with the number of test cases PASSed, FAILed and UNRESOLVED.

In practice you would write a small script that automates the job
for you. I use the following 'runtest' script:

  #! /bin/sh
  perl opp_test.pl -g -v *.test || exit 1
  (cd work; opp_makemake -f -u cmdenv; make) || exit 1
  perl opp_test.pl -r -v *.test

Note that you only need the test tool (opp_test) in steps 2 and 4.
One might wonder why steps 2-3-4 are not combined into a single
step, controlled by opp_test. The .test files could also contain
the necessary opp_makemake options too, then. The reason why this is
not done is the following:

Since we're building a single executable that contains all
test cases, we'd have to merge the opp_makemake options from
all .test files, and that is neither trivial nor a clean way.
Also, it was intended to omit everything from the .test files
that needs to be configured. With the present solution,
only the 'runtest' script contains settings that need to
be configured.

Let us see a small test file:

file cMessage_properties_1.test:

  %description:
  Test the name and length properties of cMessage.

  %activity:
  cMessage msg;
  msg.setName("Joe");
  msg.setLength(456);
  ev << "name: " << msg.name() << endl;
  ev << "length: " << msg.length() << endl;

  %contains: stdout
  name: Joe
  length: 456

The test case involves creating a simple module with only an activity()
function that has the C++ code after the %activity line as a function body.
This module is run then in a simulation as the only module. The module C++
code and the corresponding NED definition as well as omnetpp.ini are
auto-generated by opp_test, you don't need to care about them.
The standard output is matched against the lines after the %contains
line, and if it matches, a PASS is declared.

%activity is only a convenience feature to free you from having to write
a lot of boilerplate C++ and NED code when you don't really need it.
There are three other ways to proceed:

- %module lets you define a module class and run it as the only module
  in the simulation. The corresponding NED code and omnetpp.ini is
  auto-generated for you. The module won't have any gates or parameters.

- %module_a and %module_b lets you define a module pair where each module
  has an 'in' and an 'out' gate, and they are connected in both directions.
  The corresponding NED code and omnetpp.ini is auto-generated for you.
  The modules won't have parameters.

- %file lets you take full control of the source and explicitly spell out
  all NED and C++ code (and also additional data files if you need them).

Note that since all sources are compiled into a single test executable,
actions have to be taken to prevent accidental name clashes among test
cases. One has currently two choices to achieve this (yes, namespaces will be 
used for this purpose soon):
 - the string @TESTNAME@ will be replaced with the test case name everywhere
   in the sources, so if you use names like @TESTNAME@_Module, they will be
   unique;
 - words enclosed in curly braces (like {Module}) will be automatically
   prefixed with the test name, so {Module} and @TESTNAME@_Module yield the
   same results.
 - {} is a shorthand for @TESTNAME@.

What needs to be prefixed?
 - module name, channel names, network names (but not submodule names)
 - C++ class names (but not their members)
 - global variables, if there are any (one should avoid them if possible)
 - module and file names as arguments to the %module, %file, etc. entries (!)

The full list of possible % entries in a test file:

Generic entries:
   %description:
   <test-description-lines>

Test case components:
   %activity:
   <body-of-activity()>

   %global:
   <global-code-pasted-before-activity>

   %module: <modulename>
   <simple-module-C++-definition>

   %module_a: <module_a-name>
   <simple-module-C++-definition-for-module_a>
   %module_b: <module_b-name>
   <simple-module-C++-definition-for-module_b>

   %file: <file-name>
   <file-contents>

   %inifile: [<inifile-name>]
   <inifile-contents>

Conditions for PASS:
   %exitcode: <numeric-exit-code>

   %ignore-exitcode: 1

   %contains: <output-file-to-check>
   <substring>

   %contains-regex: <output-file-to-check>
   <regexp-pattern>

   %not-contains: <output-file-to-check>
   <substring>

   %not-contains-regex: <output-file-to-check>
   <regexp-pattern>

Enjoy!

--Andras
