
#include "bacula_template.inc" title="When Bacula crashes (Kaboom)" num=19 


<h1>What To Do When Bacula Crashes (Kaboom)</h1>
If you are running on a Linux system, and you have a set of working
configuration files, it is very unlikely that <b>Bacula</b>
will crash.  As with all software, however, it is inevitable that
someday, it may crash, particularly if you are running on another
operating system or using a new or unusual feature.
<p>
This chapter explains what you should do if one of the three
<b>Bacula</b> daemons (Director, File, Storage) crashes.  
 
<h2>Traceback</h2>
Each of the three Bacula daemons has a built-in exception
handler which, in case of an error, will attempt to produce
a traceback. If successful the traceback will be emailed to you.
<p>
For this to work, you need to ensure that a few things are setup
correctly on your system:
<ol>
  <li>You must have an installed copy of <b>gdb</b> (the GNU debugger),
      and it must be on <b>Bacula's</b> path. </li>
  <li>The Bacula installed script file <b>btraceback</b> must 
      be in the same directory as the daemon which dies, and it must
      be marked as executable.</li>
  <li>The script file <b>btraceback.gdb</b> must
      have the correct
      path to it specified in the <b>btraceback</b> file.</li>
  <li>You must have a <b>mail</b> program which is on <b>Bacula's</b>
      path.</li>
</ol>
If all the above conditions are met, the daemon that crashes will 
produce a traceback report and email it to you.  If the above conditions
are not true, you can either run the debugger by hand as described
below, or you may be able to correct the problems by editing the
<b>btraceback</b> file. I recommend not spending too much time
on trying to get the traceback to work as it can be very
difficult. 
<p>
The changes that might needed are to add a correct path to
the <b>gdb</b> program, correct the path to the <b>btraceback.gdb</b>
file, change the <b>mail</b> program or its path, or change your
email address. The key line in the <b>btraceback</b> file is:
<pre>
gdb -quiet -batch -x /home/kern/bacula/bin/btraceback.gdb \\
     $1 $2 2>&1 | mail -s "Bacula traceback" your-address@xxx.com
</pre>
Since each daemon has the same traceback code, a single btraceback file                                    
is sufficient if you are running more than one daemon on a machine.

<h2>Testing The Traceback</h2>
To &quot;manually&quot; test the traceback feature, you simply start
<b>Bacula</b> then obtain the <b>PID</b> of the main daemon thread
(there are multiple threads). Unfortunately, the output had to be split
to fit on this page:
<pre>
[kern@rufus kern]$ ps fax --columns 132 | grep bacula-dir
 2103 ?        S      0:00 /home/kern/bacula/k/src/dird/bacula-dir -c 
                                       /home/kern/bacula/k/src/dird/dird.conf
 2104 ?        S      0:00  \_ /home/kern/bacula/k/src/dird/bacula-dir -c 
                                       /home/kern/bacula/k/src/dird/dird.conf
 2106 ?        S      0:00      \_ /home/kern/bacula/k/src/dird/bacula-dir -c 
                                       /home/kern/bacula/k/src/dird/dird.conf
 2105 ?        S      0:00      \_ /home/kern/bacula/k/src/dird/bacula-dir -c 
                                       /home/kern/bacula/k/src/dird/dird.conf
</pre>
which in this case is 2103.  Then while Bacula is running, you
call the program giving it the path to the Bacula executable 
and the <b>PID</b>. In this case, it is:
<pre>
./btraceback /home/kern/bacula/k/src/dird 2103
</pre>
It should produce an email showing you the current state of the
daemon (in this case the Director), and then exit leaving <b>Bacula</b>
running as if nothing happened.  If this is not the case, you will need
to correct the problem by modifying the <b>btraceback</b> script.
<p>
Typical problems might be that <b>gdb</b> is not on the default
path. Fix this by specifying the full path to it in the
<b>btraceback</b> file. Another common problem is that the 
<b>mail</b> program doesn't work or is not on the default path.
On some systems, it is preferable to use <b>Mail</b> rather than <b>mail</b>.

<h2>Getting A Traceback On Other Systems</h2>
It should be possible to produce a similar traceback on systems
other than Linux, either using <b>gdb</b> or some other debugger.
Solaris with <b>gdb</b> loaded works quite fine.  On other
systems, you will need to modify the <b>btraceback</b> program
to invoke the correct debugger, and possibly correct the
<b>btraceback.gdb</b> script to have appropriate commands for your
debugger.  If anyone succeeds in making this work with another
debugger, please send us a copy of what you modified.

<a name="ManuallyDebugging">
<h2>Manually Running Bacula Under The Debugger</h2></a>
If for some reason you cannot get the automatic traceback, or
if you want to interactively examine the variable contents after
a crash, you can run Bacula under the debugger.
Assuming you want to run the Storage daemon under the 
debugger (the technique is the same for the other daemons,
only the name changes), you would do the following:
<ol>
  <li>Start the Director and the File daemon.  If the
      Storage daemon also starts, you will need to find its PID
      as shown above (ps fax | grep bacula-sd) and kill it
      with a command like the following:
      <pre>
      kill -15 PID
      </pre>
      where you replace <b>PID</b> by the actual value. </li>
  <li>At this point, the Director and the File daemon should
      be running but the Storage daemon should not.</li>
  <li>cd to the directory containing the Storage daemon</li>
  <li>Start the Storage daemon under the debugger:
    <pre>
    gdb ./bacula-sd
    </pre>
   <li>Run the Storage daemon:
     <pre>
     run -s -f -c ./bacula-sd.conf
     </pre>
     You may replace the <b>./bacula-sd.conf</b> with the full path
     to the Storage daemon's configuration file.</li>
   <li>At this point, Bacula will be fully operational.</li>
   <li>In another shell command window, start the Console program 
       and do what is necessary to cause Bacula to die.</b>
   <li>When Bacula crashes, the <b>gdb</b> shell window will
       become active and <b>gdb</b> will show you the error that 
       occurred.</b>
   <li>To get a general traceback of all threads, issue the following
       command:
       <pre>
       thread apply all bt
       </pre>
       After that you can issue any debugging command.</li>
</ol>

<h2>Getting Debug Output from Bacula</h2>
Each of the daemons normally has debug compiled into the program, but
disabled. There are two ways to enable the debug output. One is
to add the <b>-d nnn</b> option on the command line when starting
the debugger. The <b>nnn</b> is the debug level, and generally anything
between 50 and 200 is reasonable. The higher the number, the more
output is produced. The output is written to standard output.
<p>
The second way of getting debug output is to dynamically turn it
on using the Console using the <b>setdebug</b> command. The full
syntax of the command is:
<pre>
 setdebug level=nnn client=client-name storage=storage-name dir
</pre>
If none of the options are given, the command will prompt you.
You can selectively turn on/off debugging in any or all the
daemons (i.e. it is not necessary to specify all the components
of the above command).

<page_prev name="Tape Testing" url="tapetesting.html">
<page_next name="Win32 Implementation" url="win32.html">
