
#include "bacula_template.inc" title="When Bacula crashes (Kaboom)" num=17


<h1>What To Do When Bacula Crashes (Kaboom)</h1>
If you are running on a Linux system, and you have a set of working
configuration files, it is very unlikely that <b>Bacula</b>
will crash.  As with all software, however, it is inevitable that
someday, it may crash, particularly if you are running on another
operating system or using a new or unusual feature.
<p>
This chapter explains what you should do if one of the three
<b>Bacula</b> daemons (Director, File, Storage) crashes.  
 
<h2>Traceback</h2>
Each of the three <b>Bacula</b> daemons has a built-in exception
handler which, in case of an error, will attempt to produce
a traceback. If successful the traceback will be emailed to you.
<p>
For this to work, you need to ensure that a few things are setup
correctly on your system:
<ol>
  <li>You must have an installed copy of <b>gdb</b> (the GNU debugger),
      and it must be on <b>Bacula's</b> path. </li>
  <li>The <b>Bacula</b> installed script file <b>btraceback</b> must 
      be in the same directory as the daemon which dies, and it must
      be marked as executable.</li>
  <li>The script file <b>btraceback.gdb</b> must
      have the correct
      path to it specified in the <b>btraceback</b> file.</li>
  <li>You must have a <b>mail</b> program which is on <b>Bacula's</b>
      path.</li>
</ol>
If all the above conditions are met, the daemon that crashes will 
produce a traceback report and email it to you.  If the above conditions
are not true, you may be able to correct them by editing the
<b>btraceback</b> file. In doing so, you can add a correct path to
the <b>gdb</b> program, correct the path to the <b>btraceback.gdb</b>
file, change the <b>mail</b> program or its path, or change your
email address. The key line in the <b>btraceback</b> file is:
<pre>
gdb -quiet -batch -x /home/kern/bacula/bin/btraceback.gdb \\
     $1 $2 2>&1 | mail -s "Bacula traceback" your-address@xxx.com
</pre>
Since each daemon has the same traceback code, a single btraceback file                                    
is sufficient if you are running more than one daemon on a machine.

<h2>Testing The Traceback</h2>
To &quot;manually&quot; test the traceback feature, you simply start
<b>Bacula</b> then obtain the <b>PID</b> of the main daemon thread
(there are multiple threads). Unfortunately, the output had to be split
to fit on this page:
<pre>
[kern@rufus kern]$ ps fax --columns 132 | grep bacula-dir
 2103 ?        S      0:00 /home/kern/bacula/k/src/dird/bacula-dir -c 
                                       /home/kern/bacula/k/src/dird/dird.conf
 2104 ?        S      0:00  \_ /home/kern/bacula/k/src/dird/bacula-dir -c 
                                       /home/kern/bacula/k/src/dird/dird.conf
 2106 ?        S      0:00      \_ /home/kern/bacula/k/src/dird/bacula-dir -c 
                                       /home/kern/bacula/k/src/dird/dird.conf
 2105 ?        S      0:00      \_ /home/kern/bacula/k/src/dird/bacula-dir -c 
                                       /home/kern/bacula/k/src/dird/dird.conf
</pre>
which in this case is 2103.  Then while <b>Bacula</b> is running, you
call the program giving it the path to the <b>Bacula</b> executable 
and the <b>PID</b>. In this case, it is:
<pre>
./btraceback /home/kern/bacula/k/src/dird 2103
</pre>
It should produce an email showing you the current state of the
daemon (in this case the Director), and then exit leaving <b>Bacula</b>
running as if nothing happened.  If this is not the case, you will need
to correct the problem by modifying the <b>btraceback</b> script.
<p>
Typical problems might be that <b>gdb</b> is not on the default
path. Fix this by specifying the full path to it in the
<b>btraceback</b> file. Another common problem is that the 
<b>mail</b> program doesn't work or is not on the default path.
On some systems, it is preferable to use <b>Mail</b> rather than <b>mail</b>.

<h2>Getting A Traceback On Other Systems</h2>
It should be possible to produce a similar traceback on systems
other than Linux, either using <b>gdb</b> or some other debugger.
Solaris with <b>gdb</b> loaded works quite fine.  On other
systems, you will need to modify the <b>btraceback</b> program
to invoke the correct debugger, and possibly correct the
<b>btraceback.gdb</b> script to have appropriate commands for your
debugger.  If anyone succeeds in making this work with another
debugger, please send us a copy of what you modified.

<h2>Manually Running Bacula Under The Debugger</h2>
If for some reason you cannot get the automatic traceback, or
if you want to interactively examine the variable contents after
a crash, you can run <b>Bacula</b> under the debugger.
Assuming you want to run the Storage daemon under the 
debugger, you would do the following:
<ol>
  <li>Start the Director and the File daemon.  If the
      Storage daemon also starts, you will need to find its PID
      as shown above (ps fax | grep bacula-sd) and kill it
      with a command like the following:
      <pre>
      kill -15 PID
      </pre>
      where you replace <b>PID</b> by the actual value. </li>
  <li>At this point, the Director and the File daemon should
      be running but the Storage daemon should not.</li>
  <li>cd to the directory containing the Storage daemon</li>
  <li>Start the Storage daemon under the debugger:
    <pre>
    gdb ./bacula-sd
    </pre>
   <li>Run the Storage daemon:
     <pre>
     run -s -f -c ./bacula-sd.conf
     </pre>
     You may replace the <b>./bacula-sd.conf</b> with the full path
     to the Storage daemon's configuration file.</li>
   <li>At this point, <b>Bacula</b> will be fully operational.</li>
   <li>In another shell command window, start the Console program 
       and do what is necessary to cause <b>Bacula</b> to die.</b>
   <li>When <b>Bacula</b> crashes, the <b>gdb</b> shell window will
       become active and <b>gdb</b> will show you the error that 
       occurred.</b>
   <li>To get a general traceback of all threads, issue the following
       command:
       <pre>
       thread apply all bt
       </pre>
       After that you can issue any debugging command.</li>
</ol>


<page_prev name="Tape Testing" url="tapetesting.html">
<page_next name="Win32 Implementation" url="win32.html">
