
#include "bacula_template.inc" title="Regression Testing"

<H1>Bacula Regression Testing</h1>

<h2>General</h2>
This document is intended mostly for developers who wish to
ensure that their changes to Bacula don't introduce bugs
in the base code.
<p>
You can find the existing regression script in the
Bacula CVS on the SourceForge CVS in the project tree named
<b>regress</b>.
<p>
There are two different aspects of regression testing that
this document will discuss: 1. Running the Regression Script,
2. Writing a Regression test.  

<h2>Running the Regression Script</h2>
There are a number of different tests that may
be run, such as: the standard set that uses disk
Volumes and runs under any userid; a small set of
tests that write to tape; another set of tests
where you must be root to run them. To date,
each subset of tests runs no more than about
15 minutes.

<h3>Setting the Configuration Parameters</h3>
Once you have the regression directory loaded, you will 
first need to change the variables
that are at the top of <b>regress/Makefile</b>:
<pre>
\#                                        
\# Makefile for Bacula regression testing
\#
                                                                                        
\# Where to get the source to be tested
BACULA_SOURCE="${HOME}/bacula/k"
                                                                                        
\# Where to send email   !!!! Change me !!!!!!!
EMAIL=your-email@domain.com
                                                                                        
\# Full path where to find sqlite
DEPKGS="${HOME}/bacula/depkgs/sqlite"
                                                                                        
TAPE_DRIVE="/dev/nst0"

\# if you don't have an autochanger set AUTOCHANGER to /dev/null
AUTOCHANGER="/dev/sg0"

# This must be the path to the autochanger including its name
AUTOCHANGER_PATH="/bin/mtx"
                                                                                        
</pre>

<ul type="disc">
<li><b>BACULA_SOURCE</b> should be the full path to the Bacula source code
    that you wish to test.</li>
<li><b>EMAIL</b> should be your email addres. Please remember
    to change this or I will get a flood of unwanted 
    messages. You may or may not want to see these emails. In
    my case, I don't need them so I direct it to the bit bucket.</li>
<li><b>DEPKGS</b> should be the full path to the sqlite package,
    must be build before running a Bacula regression. SQLite
    is used for all regression testing of Bacula. Perhaps at
    some future time the tests will include MySQL.</li>
<li><b>TAPE_DRIVE</b> is the full path to your tape drive.
    The base set of regression tests do not use a tape, so
    this is only important if you want to run the full tests.</li>
<li><b>AUTOCHANGER</b> is the name of your autochanger device.
    Set this to /dev/null if you do not have one.</li>
<li><b>AUTOCHANGER_PATH</b> is the full path including the
    program name for your autochanger program (normally
    <b>mtx</b>. Leave the default value if you do not have one.</li>

</ul>
<p>
<h3>Building the Test Bacula</h3>
Once the above variables are set, you can build Bacula
by entering:
<pre>
make setup
</pre>
This will make a copy of the source code withing the regression
tree (in directory regress/build), configure it, and build it.
There should be no errors. If there are, please correct
them before continuing.

<h3>Running the Disk Only Regression</h3>
Once Bacula is built, you can run the basic disk
only non-root regression test by
entering:
<pre>
make test
</pre>
This will run the base set of tests using disk Volumes, currently
(25 Sep 2003), there are current 18 separate tests that run. If you
are testing on a non-Linux machine two of the tests will not
be run. In any case, as we add new tests, the number will
vary. It will take about 5 or 10 minutes if you have
a fast (2 GHz) machine, and you don't need to be
root to run these tests (I run under my regular
userid). 
The result should be something similar to:
<pre>
Test results
  
  ===== Backup Bacula Test OK =====
  ===== Verify Volume Test OK =====
  ===== sparse-test OK =====
  ===== compressed-test OK =====
  ===== sparse-compressed-test OK =====
  ===== Weird files test OK =====
  ===== two-jobs-test OK =====
  ===== two-vol-test OK =====
  ===== six-vol-test OK =====
  ===== bscan-test OK =====
  ===== Weird files2 test OK =====
  ===== concurrent-jobs-test OK =====
  ===== four-concurrent-jobs-test OK =====
  ===== bsr-opt-test OK =====
  ===== bextract-test OK =====
  ===== recycle-test OK =====
  ===== span-vol-test OK =====
  ===== restore-by-file-test OK =====
</pre>
Each separate test is self contained in that it initializes
to run Bacula from scratch (i.e. newly created database).
It will also kill any Bacula session that is currently running.
In addition, it uses ports 8101, 8102, and 8103 so that
it does not intefere with a production system.
<p>
<h3>Other Tests</h3>
There are a number of other tests that can be
run as well.  All the tests are a simply shell
script keep in the regress directory.  For
example the &quot;make test&quot; simply
executes <b>./all-non-root-tests</b>.  The
other tests are:
<dl>
  <dt>all_non-root-tests</dt><dd>All non-tape tests not requiring root.
      This is the standard set of tests, that in general, backup some
      data, then restore it, and finally compares the restored data
      with the original data.</dd>
  <dt>all-root-tests</dt><dd>All non-tape tests requiring root permission.
      These are a relatively small number of tests that require running
      as root. The amount of data backed up can be quite large. For
      example, one test backs up /usr, another backs up /etc.  One
      or more of these tests reports an error -- I'll fix it one
      day. </dd>
  <dt>all-non-root-tape-tests</dt><dd>All tape test not requiring root.
       There are currently three tests, all run without being root,
       and backup to a tape. The first two tests use one volume,
       and the third test requires an autochanger, and uses two
       volumes. If you don't have an autochanger, then this script
       will probably produce an error. </dd>
  <dt>all-tape-and-file-tests</dt><dd>All tape and file tests not requiring
      root.  This includes just about everything, and I don't run it 
      very often. </dd>
</dl>

<h3>If a Test Fails</h3>
If you one or more tests fail, the line output will be
similar to:
<pre>
  !!!!! concurrent-jobs-test failed!!! !!!!!
</pre>
If you want to determine why the test failed, you will
need to modify the script so that it prints.
Do so by finding the file in <b>regress/tests</b>
that corresponds to the name printed. For example,
the script for the above error message is in:
<b>regress/tests/concurrent-jobs-test</b>.
<p>
In order to see the output produced by Bacula, you
need only change the lines that start with <b>@output</b>
to <b>@tee</b>, then rerun the test by hand. it is
very important to start the test from
the <b>regress</b> directory.
<p>
To modify the test mentioned above so that you can
see the output, modify the following:
<pre>
...
bin/console -c bin/console.conf &lt;&lt;END_OF_DATA
@output /dev/null
messages
@output tmp/log1.out
label storage=File volume=TestVolume001
run job=CompressedTest level=Full yes
run job=CompressedTest level=Full yes
run job=CompressedTest level=Full yes
run job=CompressedTest level=Full yes
wait
messages
@#
@# now do a restore
@#
@tee    tmp/log2.out
restore where=${cwd}/tmp/bacula-restores
5
</pre>
to be:
<pre>
bin/console -c bin/console.conf &lt;&lt;END_OF_DATA
@output /dev/null
messages
@tee    tmp/log1.out
label storage=File volume=TestVolume001
run job=CompressedTest level=Full yes
run job=CompressedTest level=Full yes
run job=CompressedTest level=Full yes
run job=CompressedTest level=Full yes
wait
messages
@#
@# now do a restore
@#
@output tmp/log2.out
restore where=${cwd}/tmp/bacula-restores
5
</pre>
this will ensure that the output is written both
to file (tmp/log1.out and tmp/log2.out) as well
as to your terminal.
<p>
In rare cases, you may want to see debug output
that is generated by the Director or one of
the other daemons. To do so modify:
</pre>
bin/bacula start 2&gt;&amper;1 &gt;/dev/null
</pre>
to
<pre>
bin/bacula start
</pre>

<h2>Writing a Regression Test</h2>
Any developer, who implements
a major new feature, should write a regression 
test that exercises and validates the new feature.
Each regression test is a complete test by itself.
It terminates any running Bacula, initializes the
database, starts Bacula, then runs the test by using
the console program.  
<h3>Running the Tests by Hand</h3>
You can run any individual test by hand by cd'ing
to the <b>regress</b> directory and entering:
<pre>
tests/&lt;test-name&gt;
</pre>
<h3>Directory Structure</h3>
The directory structure of the regression tests is:
<pre>
  regress                   - Makefile, scripts to start tests
    |------ scripts         - Scripts and conf files    
    |-------tests           - All test scripts are here
    |
    |------------------------ All directories below this are used
    |                          for testing, but are created from the
    |                          above directories and are removed with
    |                          &quot;make distclean&quot;
    |------ bin             - This is the install directory for
    |                           Bacula to be used testing
    |------ build           - Where the Bacula source build tree is
    |------ tmp             - Most temp files go here
    |------ working         - Bacula working directory
    |------ weird-files     - Weird files used in two of the tests.
</pre>
<h3>Adding a New Test</h3>
If you want to write a new regression test,
it is best to start with one of the existing
test scripts, and modify it to do the new
test. 
<p>
When adding a new test, be extremely
careful about adding anything to any of
the daemons' configuration files.  The reason
is that it may change the prompts that are
sent to the console. For example, adding a Pool  
means that the current scripts, which
assume that Bacula automatically selects a
Pool, will now be presented with a new
prompt, so the test will fail.  If you
need to enhance the configuration files,
consider making your own versions.


<page_prev name="Porting Bacula" url="porting.html.html">
<page_next name="Implementing a GUI" url="gui-interface.html">
