
#include "bacula_template.inc" title="Disaster Recovery Using Bacula" num=9

<h1>Disaster Recovery Using Bacula</h1>

<h2>General</h2>
When disaster strikes, you must have a plan, and you must have
prepared in advance otherwise the work of recovering your
system and your files will be considerably greater.  For example, if you
have not previously saved the partitioning information for your hard
disk, how can you properly rebuild it if the disk must be replaced?
<p>
Unfortunately, many of the steps one must take before and immediately
after a disaster are very operating system dependent.  As a consequence,
this chapter will discuss in detail disaster recovery (also called Bare
Metal Recovery) for <b>Linux</b> and <b>Solaris</b>. For Solaris, the
procedures are still quite manual. For FreeBSD the same procedures
may be used but they are not yet developed.  For Win32, no luck.
Apparently an &quot;emergency boot&quot; disk allowing
access to the full system API without interference does not exist.

<h2>Important Considerations</h2>
Here are a few important considerations concerning disaster recovery
that you should take into account before a disaster strikes.
<ul>
 <li>If the building which houses your computers burns down or is otherwise
     destroyed, do you have off-site backup data?</li>
 <li>Disaster recovery is much easier if you have several machines. If
     you have a single machine, how will you handle unforeseen events
     if your only machine is down?</li>
 <li>Do you want to protect your whole system and use Bacula to 
     recover everything? or do you want to try to restore your system from
     the original installation disks and apply any other updates and
     only restore user files?
</ul>
<h2>Bare Metal Recovery on Linux</h2>
The remainder of this section concerns recovering a <b>Linux</b> computer,
and parts of it relate to the Red Hat version of Linux.  The <b>Solaris</b>
procedures can be found below.
<p>
A so called &quot;Bare Metal&quot; recovery is one where you start with
an empty hard disk and you restore your machine.
There are also cases where you may lose a file or a directory and want
it restored. Please see the previous chapter for more details for
those cases.                
<p>Bare Metal Recovery assumes that you have the following four
items for your system:
<ul>
  <li>An emergency boot disk allowing you to boot without a 
      hard disk</li>
  <li>A Bacula Rescue Disk containing your disk information and
      a number of helpful scripts (described below) including a
      statically linked version of the Bacula File daemon </li>
  <li>A full Bacula backup of your system possibly including
      Incremental or Differential backups since the last Full
      backup </li>
  <li>A second system running the Bacula Director, the Catalog,
      and the Storage daemon. (this is not an absolute requirement,
      but how to get around it is not yet documented here)</li>
</ul>

<h2>Restrictions</h2>
In addition, to the above assumptions, the following conditions or
restrictions apply:
<ul>
  <li>Linux only -- tested only on Red Hat, but should work on other Linuxes</li>
  <li>The scripts handle only SCSI and IDE disks</li>
  <li>All partitions will be recreated, but only <b>ext2</b>,
      <b>ext3</b>, and <b>swap</b> partitions will be reformatted.
      Any other partitions such as Windows FAT partitions will
      not be formatted by the scripts, but you can do it by hand</li>
  <li>You are using either <b>lilo</b> or <b>grub</b> as a boot
      loader, and you know which one (not automatically detected)</li>
</ul>

<h2>Scripts</h2>
The scripts discussed below can be found in the <b>rescue/linux</b>
subdirectory of the Bacula source code.

<h2>Preparation for a Bare Metal Recovery</h2>
There are two things you should do immediately on all (Linux) 
systems for which you wish to do a bare metal recovery:
<ol>
  <li>Create a system emergency boot disk or alternatively
      a system installation boot floppy.  This step can be skipped
      if you have an Installation CDROM and your machine will boot
      from CDROM (most modern computers will).</li>
  <li>Create a Bacula Rescue disk, which captures the current working
      state of your computer and creates scripts to restore it. In
      addition, it creates a statically linked version of the Bacula
      File daemon (Client) program, which is key to successfully restoring
      from scratch. </li>
</ol>
<h3>Creating an Emergency Boot Disk</h2>
Here you have several choices:
<ul>
  <li>Create a tomsrtbt disk (any Linux)</li>
  <li>Create an emergency boot disk (any Linux I think) </li>
  <li>Create a Red Hat Installation disk (Red Hat specific)</li>
  <li>Others </li>
</ul>

<h4>tomsrtbt</h4>
My preference is to create and use a <b>tomsrtbt</b> emergency
boot disk because it gives you a very clean Linux environment
(with a 2.2 kernel) and the most utilities.
See <a href="http://www.toms.net/rb/">http://www.toms.net/rb/</a>
for more details on this. It is very easy to do and well worth 
the effort.  However, I recommend that you create both especially
if you have non-standard hardware. You may find that <b>tomsrtbt</b>
will not work with your network driver (he surely has one, but
you must explicitly put it on the disk), whereas the Linux rescue
is more likely to work.

<h4>Emergency Boot Disk</h4>
To create a standard Linux emergency boot disk you must first know the name
of the kernel, which you can find with:
<pre>
  ls -l /boot
</pre>
and looking on the <b>vmlinux-...</b> line or alternative do
an
<pre>
 uname -a
</pre>
then become root and with a blank floppy in the drive, enter
the following command:
<pre>
  mkbootdisk --device /dev/fd0 2.4.18-18
</pre>
where you replace &quot;2.4.18-18&quot; by your system name.
<p>This disk can then be booted and you will be in an environment  
with a number of important tools available.  Some disadvantages
of this environment as opposed to <b>tomsrtbt</b> are that
you must enter <b>linux rescue</b> at the boot prompt or
the boot will fail without a hard disk; it requires a disk
boot image or a CDROM to be mounted, if the CDROM is
released, you will loose a large number of the tools.

<h4>Red Hat Installation Disk</h4>
Specific to Red Hat Linux, is to create an
Installation floppy, which can also be used as an emergency
boot disk.
The advantage of this method is that it works in conjunction
with the installation CDROM and hence during the first part
of restoring the system, you have a much larger number of
tools available (on the CDROM). This can be extremely useful
if you are not sure what really happened and you need to
examine your system in detail.
<p>To make a Red Hat Linux
installation disk, do the following:
<pre>
mount the Installation CDROM (/mnt/cdrom)
cd /mnt/cdrom/images
dd if=boot.img of=/dev/fd0 bs=1440k
</pre>
Now that you have either an emergency boot disk or an installation
floppy, you will be able to reboot your system in the absence of
your hard disk or with a damaged hard disk.  This method
has the same disadvantages compared to <b>tomsrtbt</b> disk
as mentioned above for the Emergency Boot Disk.

<h3>Creating a Bacula Rescue Disk</h3>
Simply having a boot disk is not sufficient to re-create things
as they were. To solve this problem, we will create a Bacula
Rescue disk.  Everything that will be written to this disk
will first be placed into the <b>&lt;bacula-src&gt;/rescue/linux</b>
directory. 
<p>The first step is while your system is up and
running normally, you use a <b>Bacula</b> script called
<b>getdiskinfo</b> to capture certain important information
about your hard disk configuration (partitioning, formatting,
mount points, ...). <b>getdiskinfo</b> will also create a
number of scripts using the information found that
can be used in an emergency to repartition your disks, reformat them,
and restore a statically linked version of the <b>Bacula</b> file daemon
so that your disk can be restored from within a minimal boot 
environment.
<p>The first step is to run <b>getdiskinfo</b> as follows:
<pre>
   su
   cd &lt;bacula-src&gt;/rescue/linux
   ./getdiskinfo
</pre>
<b>getdiskinfo</b> works for either IDE or SCSI drives and
recognizes both ext2 and ext3 file systems. If you wish
to restore other file systems, you will need to modify
the code.  This script can be run multiple times, but really
only needs to be run once unless you change your hard disk
configuration.
<p>
Assuming you have a single hard disk on device /dev/hda,
<b>getdiskinfo</b> will create the following files:
<dl>
  <dt>partition.hda</dt><dd>This file contains the shell commands
      to repartition your hard disk drive /dev/hda to the current
      state.  If you have additional drives (e.g. /dev/hdc), you
      will find one of these files for each drive. 
      DO NOT EXECUTE THIS SCRIPT UNLESS YOU WANT YOUR HARD DISK
      REPARTITIONED</dd>

  <dt>format.hda</dt><dd>This file contains the shell commands that
      will format each of the partitions on your hard drive. It
      knows about ext2, ext3, and swap partitions. All other partitions,
      you must manually format. It is recommended that any Microsoft
      partitions be partitioned with Microsoft's <b>format</b> command
      rather than using Unix tools.
      DO NOT EXECUTE THIS SCRIPT UNLESS YOU WANT YOUR HARD DISK
      REFORMATTED</dd>

  <dt>mount_drives</dt><dd>This script will mount all ext2 and ext3
      drives that were previously mounted. They will be mounted on
      <p>/mnt/drive/. This is used just before running the statically
      linked Bacula so that it can access your drives for the restore.</dd>

  <dt>restore_bacula</dt><dd>This script will restore the File daemon
      from the Bacula Rescue disk. Building the Bacula Rescue disk
      will be described later. This will provide your emergency boot
      environment with a Bacula file daemon. Note, this is a special
      statically linked version of the file daemon (i.e. it does not
      need or use shared libraries).</dd>

  <dt>start_network</dt><dd>This script will start your network using
      the simplest possible commands. You will need to verify that the
      IP address used in this script is correct.  In addition, if you
      have several ethernet cards, you may need to make other modifications
      to this script.
      </dd>

  <dt>sfdisk</dt><dd>This is the program that will repartition your hard disk,
      and it is normally found in <b>/sbin/sfdisk</b>. It is placed in
      this directory so that it will be included on the rescue disk as
      it is not normally available with all emergency boot environments.</dd>

   <dt>sfdisk.gz</dt><dd>This is the version of sfdisk that works with
      <b>tomsrtbt</b>. The standard sfdisk described above will not
      run under tomsrtbt. </dd>

</dl>
The <b>getdiskinfo</b> program (actually a shell script) will
also create a subdirectory named <b>diskinfo</b>,
which contains the following files:
<pre>
df.bsi
disks.bsi
fstab.bsi
ifconfig.bsi
mount.bsi
mount.ext2.bsi
mount.ext3.bsi
mtab.bsi
route.bsi
sfdisk.disks.bsi
sfdisk.hda.bsi
sfdisk.make.hda.bsi
</pre>
Each of these files contains some important piece of information
(sometimes redundant) about your hard disk setup or your
network. Normally, you will not need this information, but it
will be written to the Bacula Rescue disk just in case. Since
it is normally not used, we will leave it to you to examine
those files at your leisure.
<p>
<h4>Building a Static File Daemon</h4>
The second of the three steps in creating your Bacula Rescue disk is
to build a static version of the File daemon. Do so by either
configuring Bacula as follows or by allowing the <b>make_rescue_disk</b>
script described below make it for you:
<pre>
cd &lt;bacula-src&gt;
./configure &lt;normal-options&gt; --enable-static-fd
make 
cd src/filed
strip bacula-fd
cp bacula-fd ../../rescue/linux
cp bacula-fd.conf ../../rescue/linux
</pre>
Finally, in &lt;bacula-src&gt;/rescue/linux, ensure that
the WorkingDirectory, PIDDirectory, and SubSysDirectory
all point to reasonable locations on a stripped down system.
If you are using <b>tomsrtbt</b> you will also want to
replace machine names with IP addresses since there is
no resolver running. With the Linux Rescue disk, network
address mapping seems to work.
Don't forget that at the time this version of the Bacula File
daemon runs, your file system will not be restored.
In my bacula-fd.conf, I use <b>/var/working</b>.

<h4>Writing the Bacula Rescue Floppy</h4>
When you have everything you need (output of getdiskinfo,
Bacula File daemon, ...), you create your rescue floppy
by putting a blank tape into your floppy disk drive and
entering:
<pre>
su
./make_rescue_disk
</pre>
This script will reformat the floppy and
write everything in the current directory and
all files in the <b>diskinfo</b> directory to the floppy.
If you supply the appropriate command line options, it
will also build a static version of the Bacula file daemon
and copy it along with the configuration file to the disk.
Also using a command line option, you can make it
write a compressed tar file containing all the files 
whose names are in <b>backup.etc.list</b> to the floppy.
The list as provided contains names of files in
<b>/etc</b> that you might need in a disaster situation.
It is not needed, but in some cases such as a complex 
network setup, you may find it useful.
<h4>Options for make_rescue_disk</h4>
The following command line options are available for
the make_rescue_disk script:
<pre>
Usage: make_rescue_disk
  -h, --help             print this message
  --make-static-bacula   make static File daemon and add to diskette
  --copy-static-bacula   copy static File daemon to diskette
  --copy-etc-files       copy files in etc list to diskette
</pre>
Briefly the options are:
<dl>
  <dt>--make-static-bacula</dt><dd>If this option is specified, the
      script will assume that you have already configured and
      built Bacula. It will then proceed to build a statically
      linked version and copy it along with bacula-fd.conf to the
      current directory, then write it to the rescue disk.</dd>
  <dt>--copy-static-bacula,/dt><dd>If this option is given, the
      script will assume that you already have a copy of the
      statically linked Bacula in the current directory named
      <b>bacula-fd</b> as well as the configuration script. They
      will then be written to the rescue disk.</dd>
  <dt>--copy-etc-files</dt><dd>If this option is specified, the
      script will tar the files in <b>backup.etc.list</b> and write
      them to the rescue disk.</dd>
</dl>
Please examine the contents of the rescue floppy to ensure that
it has everything you want and need. If not modify the scripts
as necessary and re-run it until it is correct.
<p>
Now that you have both a system boot floppy and a Bacula Rescue
floppy, assuming you have a full backup of your system made by
Bacula, you are ready to handle nearly any kind of emergency
restoration situation.

<h2>Restoring Your System</h2>
Now, let's assume that your hard disk has just died and that
you have replaced it with an new identical drive. In addition,
we assume that you have:
<ol>
  <li>A recent Bacula backup (Full plus Incrementals)</li>
  <li>An emergency boot floppy (preferably <b>tomsrtbt</b>)</li>
  <li>A Bacula Rescue Disk</li>
  <li>Your Bacula Director, Catalog, and Storage daemon running
      on another machine on your local network.</li>
</ol>
This is a relatively simple case, and later in this chapter,
as time permits, we will discuss how you might recover from a situation
where the machine that crashes is your main Bacula server (i.e.  has the
Director, the Catalog, and the Storage daemon).
<p>You will take the following steps to get your system
back up and running:
<ol>
  <li>Boot with your Emergency Floppy</li>
  <li>Mount your Bacula Rescue floppy</li>
  <li>Start the Network (local network)</li>
  <li>Re-partition your hard disk(s) as it was before</li>
  <li>Re-format your partitions</li>
  <li>Restore the Bacula File daemon (static version)</li>
  <li>Perform a Bacula restore of all your files</li>
  <li>Re-install your boot loader</li>
  <li>Reboot</li>
</ol>
Now for the details ...

<h3>Boot with your Emergency Floppy</h3>
First you will boot with your emergency floppy.  If you use
the Installation floppy described above, when you get to
the boot prompt:
<pre>
boot:
</pre>
you enter <b>linux rescue</b>.
<p>If you are booting from <b>tomsrtbt</b> simply enter the
default responses.
<p>
When your machine finishes booting, you should be at the command
prompt possibly with your hard disk mounted on <b>/mount/sysimage</b>
(Linux emergency only).
To see what is actually mounted, use:
<pre>
df
</pre>
<h4>Mount your Bacula Rescue Floppy</h4>
Make sure that the mount point <b>/mnt/floppy</b> exists.
If not, enter:
<pre>
mkdir -p /mnt/floppy
</pre>
the mount your <b>Bacula Rescue</b> disk and cd to it with:
<pre>
mount /dev/fd0 /mnt/floppy
cd /mnt/floppy
</pre>
To simplify running the scripts make sure the current directory
is on your path by:
<pre>
PATH=$PATH:.
</pre>
<h4>Start the Network</h4>
At this point, you should bring up your network. Normally,
this is quite simple and requires just a few commands.
To simplify your task, we have created a script that should
work in most cases by typing:
<pre>
./start_network
</pre>
You can test it by pinging another machine, or pinging your broken
machine machine from another machine.  Do not proceed until your network
is up.

<h4>Unmount Your Hard Disk (if mounted)</h4>
When you are sure you want to repartition your disk, normally,
if your disk was damaged or if you are using <b>tomsrtbt</b>
your hard disk will not be mounted. However, if it is
you must first unmount it so that it is not in use. Do so by
entering <b>df</b> and then enter the correct commands
to unmount the disks. For example:
<pre>
umount /mnt/sysimage/boot
umount /mnt/sysimage/usr
umount /mnt/sysimage/proc
umount /mnt/sysimage/
</pre>
where you explicitly unmount (<b>umount</b>) each sysimage partition
and finally, the last one being the root.  Do another <b>df</b>
command to be sure you successfully unmount all the sysimage 
partitions.  
<p>This is necessary because <b>sfdisk</b> will refuse to partition
a disk that is currently mounted. As mentioned, this should never
be necessary with <b>tomsrtbt</b>.

<h4>Partition Your Hard Disk(s)</h4>
If you are using <b>tomsrtbt</b>, you will need to
do the following steps to get the correct
sfdisk:
<pre>
rm -f sfdisk
bzip2 -d sfdisk.bz2
</pre>
<b>
Do not do the above steps if you are using a standard
Linux boot disk.</b>
<p>
Then proceed with partitioning your hard disk by:
<pre>
./partition.hda
</pre>
If you have multiple disks, do the same for each of them.  For SCSI
disks, the repartition script will be named: <b>partition.sda</b>.  If
the script complains about the disk being in use, simply go back and
redo the <b>df</b> command and <b>umount</b> commands until you no
longer have your hard disk mounted.  Note, in many cases, if your hard
disk was seriously damaged or a new one installed, it will not
automatically be mounted.  If it is mounted, it is because the emergency
kernel found one or more possibly valid partitions.
<p>
If for some reason this proceedure does not work,
you can use the information in <b>partition.hda</b> to 
re-partition your disks by hand using <b>fdisk</b>.

<h4>Format Your Hard Disk(s)</h4>
After partitioning your disk, you must format it appropriately.
The formatting script will put back swap partitions, normal Unix
partitions (ext2) and journaled partitions (ext3). Do so by
entering for each disk:
<pre>
./format.hda
</pre>
The format script will ask you if you want a block check done. We
recommend to answer yes, but realize that for very large disks this
can take hours.

<h4>Mount the Newly Formatted Disks</h4>
Once the disks are partitioned and formatted, you can remount them
with the <b>mount_drives</b> script.  
All your drives
must be mounted for Bacula to be able to access them. 
Run the script as follows:
<pre>
./mount_drives
df
</pre>
The <b>df</b> will tell you if the drives are mounted.
If not, re-run the script again. It isn't always easy
to figure out and create the mount points and the mounts
in the proper order, so repeating the <b>./mount_drives</b>
command will not cause any harm and will most likely work
the second time.  If not, correct it by hand before
continuing.

<h4>Unmount the CDROM</h4>
Next, if you are using the Red Hat installation
disk, unmount the CDROM drive by doing:
<pre>
umount /mnt/cdrom
</pre>
This is not necessary if you are running <b>tomsrtbt</b>.
In doing this, I find it is always busy, and I haven't
figured out how to unmount it (Linux boot only).

<h4>Restore and Start the File Daemon</h4>
Now, change (cd) to some directory where you want to put the image of
the Bacula File daemon.  I use the tmp directory my hard disk (mounted
as <b>/mnt/disk/tmp</b>) because it is easy.  Then install into the current
directory Bacula by running the <b>restore_bacula</b> script from the
floppy drive.  For example:
<pre>
cd /mnt/disk
mkdir -p /mnt/disk/tmp
mkdir -p /mnt/disk/tmp/working
/mnt/floppy/restore_bacula
ls -l
</pre>
Make sure <b>bacula-fd</b> and <b>bacula-fd.conf</b>
are both there.
<p>
Edit the Bacula configuration file, create the
working/pid/subsys directory if you haven't
already done so above, and start Bacula
by entering:
<pre>
chroot /mnt/disk /tmp/bacula-fd -c /tmp/bacula-fd.conf
</pre>
The above command starts the Bacula File daemon
with your the proper root disk location (i.e.
<b>/mnt/disk/tmp</b>.
If Bacula does not start correct the problem and
start it.  You can check if it is running by
entering:
<pre>
ps fax
</pre>
You can kill Bacula by entering:
<pre>
kill -TERM &lt;pid&gt;
</pre>
where <b>pid</b> is the first number printed in front
of the first occurrence of <b>bacula-fd</b> in the
<b>ps fax</b> command.
<p>
Now, you should be able to use another computer
with Bacula installed to check the status by entering:
<pre>
status client=xxxx
</pre>
into the Console program, where xxxx is the name of the
client you are restoring.
<p>
One common problem is that your <b>bacula-fd.conf</b> 
may contain machine addresses that are not properly
resolved on this stripped down system because it is not
running DNS. In that case, be prepared to edit 
<b>bacula-fd.conf</b> to replace the name of the
Director's machine with its IP address. Or better
yet, do this before building the Bacula rescue disk.

<h4>Restoring using the RedHat Installation Disk</h4>
Suppose your system was damaged for one reason or another, so that
the hard disk and the partitioning and much of the filesystems
are intact, but you want to do a full restore.  If you have 
booted into your system with the RedHat Installation Disk by
specifying <b>linux rescue</b> at the <b>boot:</b> prompt, you
will find yourself in a shell command with your disks already
mounted (if it was possible) in <b>/mnt/sysimage</b>.  In this
case, you can do much like you did above to restore your 
system:
<pre>
cd /mnt/sysimage/tmp
mkdir -p /mnt/sysimage/tmp/working
/mnt/floppy/restore_bacula
ls -l
</pre>
Make sure that <b>bacula-fd</b> and <b>bacula-fd.conf</b> are both
in the current directory and that the directory names in the
<b>bacula-fd.conf</b> correctly point to the appropriate 
directories.  Then start <b>Bacula</b> with:
<pre>
chroot /mnt/sysimage /tmp/bacula-fd -c /tmp/bacula-fd.conf
</pre>


   
<h4>Restore Your Files</h4>
On the computer that is running the Director, you now
run a <b>restore</b> command and select the files to
be restored (normally everything), but before starting
the restore, there is one final change you must make 
using the <b>mod</b> option.  You must change the 
<b>Where</b> directory to be the root by using the 
<b>mod</b> option just before running the job and
selecting <b>Where</b>. Set it to:
<pre>
/
</pre>
then run the restore.
<p>
You might be tempted to avoid using <b>chroot</b> and
running Bacula directly and then using a <b>Where</b>
to specify a destination of <b>/mnt/disk</b>. This is
possible, however, the current version of Bacula always
restores files to the new location, and thus any
soft links that have been specified with absolute
paths will end up with <b>/mnt/disk</b> prefixed to them.
In general this is not fatal to getting your system
running, but be aware that you will have to fix these
links if you do not use <b>chroot</b>.

<h4>Final Step</h4>
At this point, the restore should have finished with no errors,
and all your files will be restored. One last task remains and
that is to write a new boot sector so that your machine will
boot.  For <b>lilo</b>, you enter the following command:
<pre>
run_lilo
</pre>
If you are using grub instead of lilo, you must enter
the following:
<pre>
run_grub
</pre>
Note, I've had quite a number of problems with <b>grub</b>
because it is rather complicated and not designed to install
easily under a simplified system. So, if you experience
errors or end up unexpectedly in a <b>chroot</b> shell,
simply exit back to the normal shell and type in the
appropriate commands from the <b>run_grub</b> script
by hand until you get it to install.
 
<h4>Reboot</h4>
Reboot your machine by simply entering <b>exit</b> until
you get to the main prompt then enter <b>ctl-d</b>.
<p>
If everything went well, you should now be back up and running.
If not, re-insert the emergency boot floppy, boot, and figure
out what is wrong.
<p>At this point, you will probably want to remove
the temporary copy of Bacula that you installed. Do
so with:
<pre>
rm -f /bacula-fd /bacula-fd.conf
rm -rf /working
</pre>

<h2>Problems or Bugs</h2>
Since every flavor and every release of Linux is different,
there are likely to be some small difficulties with the scripts,
so please be prepared to edit them in a minimal environment. 
A rudimentary knowledge of <b>vi</b> is very useful.  Also,
these scripts do not do everything. You will need to reformat
Windows partitions by hand, for example.
<p>
Getting the boot loader back can be a problem if you are
using <b>grub</b> because it is so complicated. If all else
fails, reboot your system from your floppy but using the 
restored disk image, then proceed to a reinstallation of
grub (looking at the run-grub script can help).  By contrast,
lilo is a piece of cake.

<h3>Bugs</h3>
When performing the bare metal recovery using the Red Hat 
emergency boot disk (actually the installation boot disk),
I was never able to release the cdrom, and when the system
came up <b>/mnt/cdrom</b> was soft linked to <b>/mnt/disk/dev/hdd</b>,
which is not correct. I fixed this in each case by deleting and
simply remaking it with <b>mkdir -p /mnt/cdrom</b>.

<h2>tomsrtbt</h2>
This is a single floppy (1.722Meg) that really has A LOT of software.
For example, by default (version 2.0.103) you get:
<p>
AHA152X AHA1542 AIC7XXX BUSLOGIC DAC960 DEC_ELCP(TULIP)
EATA EEXPRESS/PRO/PRO100 EL2 EL3 EXT2 EXT3 FAT FD IDE-CD/DISK/TAPE IMM INITRD
ISO9660 JOLIET LOOP MATH_EMULATION MINIX MSDOS NCR53C8XX NE2000 NFS NTFS
PARPORT PCINE2K PCNET32 PLIP PPA RTL8139 SD SERIAL/_CONSOLE SLIP SMC_ULTRA SR
ST VFAT VID_SELECT VORTEX WD80x3 .exrc 3c589_cs agetty ash badblocks basename
boot.b buildit.s busybox bz2bzImage bzip2 cardmgr cardmgr.pid cat chain.b
chattr chgrp chmod chown chroot clear clone.s cmp common config cp cpio cs
cut date dd dd-lfs debugfs ddate df dhcpcd-- dirname dmesg domainname ds du
dumpe2fs e2fsck echo egrep elvis ex false fdflush fdformat fdisk filesize
find findsuper fmt fstab grep group gunzip gzip halt head hexdump hexedit
host.conf hostname hosts httpd i82365 ifconfig ile init inittab insmod
install.s issue kernel key.lst kill killall killall5 ld ld-linux length
less libc libcom_err libe2p libext2fs libtermcap libuuid lilo lilo.conf ln
loadkmap login ls lsattr lsmod lua luasocket man map md5sum miterm mkdir
mkdosfs mke2fs mkfifo mkfs.minix mknod mkswap more more.help mount mt mtab mv
nc necho network networks nmclan_cs nslookup passwd pax pcmcia_core pcnet_cs
pidof ping poweroff printf profile protocols ps pwd rc.0 rc.S rc.custom
rc.custom.gz rc.pcmcia reboot rescuept reset resolv.conf rm rmdir rmmod route
rsh rshd script sed serial serial_cs services setserial settings.s sh shared
slattach sleep sln sort split stab strings swapoff swapon sync tail tar tcic
tee telnet telnetd termcap test tomshexd tomsrtbt.FAQ touch traceroute true
tune2fs umount undeb-- unpack.s unrpm-- update utmp vi vi.help view watch
wc wget which xargs xirc2ps_cs yecho yes zcat
</p>
<p>
In addition, at <a href="http://www.toms.net/rb">Tom's Web Site</a>, you
can find a lot of additional kernel drivers and other software (such as
<b>sdisk</b>, which is used by Bacula.
<p>Building his floppy is a piece of cake. Simply download
his .tar.gz file then:
<pre>
- detar the .tar.gz archive
- become root
- cd to the tomsrtbt-&lt;version&gt; directory
- load a blank floppy with no bad sectors
- ./install.s 
</pre>

<h2>Solaris Bare Metal Recovery</h2>
The same basic techniques as described above apply to Solaris:
<ul>
  <li>the same restrictions as those given for Linux apply</li>
  <li>you will need to create a Bacula Rescue disk</li>
</ul>
However, during the recovery phase, the boot and disk preparation
procedures are different:
<ul>
  <li>there is no need to create an emergency boot disk 
      since it is an integrated part of the boot.</li>
  <li>you must partition and format your hard disk by hand
      following manual procedures as described in W. Curtis Preston's
      book &quot;Unix Backup & Recovery&quot;</li>
</ul>
Once the disk is partitioned, formatted and mounted,
you can continue with bringing up the network and
reloading Bacula.
</ul>

<h3>Preparing Solaris Before a Disaster</h3>
As mentioned above, before a disaster strikes, you should
prepare the information needed in the case of problems.
To do so, in the <b>rescue/solaris</b> subdirectory enter:
<pre>
su
./getdiskinfo
./make_rescue_disk
</pre>
The <b>getdiskinfo</b> script will, as in the case of Linux described
above, create a subdirectory <b>diskinfo</b> containing the output
from several system utilities. In addition, it will contain the
output from the <b>SysAudit</b> program as described in 
Curtis Preston's book. This file <b>diskinfo/sysaudit.bsi</b>
will contain the disk partitioning information that will allow
you to manually follow the procedures in the &quot;Unix Backup & Recovery&quot;
book to repartition and format your hard disk. In addition, the
<b>getdiskinfo</b> script will create a <b>start_network</b>
script.
<p>
Once you have your your disks repartitioned and formatted, do   
the following:
<ul>
  <li>Start Your Network with the <b>start_network</b> script</li>
  <li>Restore the Bacula File daemon as documented above</li>
  <li>Perform a Bacula restore of all your files using the same
      commands as described above for Linux</li>
  <li>Re-install your boot loader using the instructions outlined
      in the &quot;Unix Backup & Recovery&quot; book
      using installboot</li>
</ul>
 

<h2>Recovering a Server</h2>
Above, we considered how to recover a client machine where a
valid <b>Bacula</b> server was running on another machine.  However,
what happens if your server goes down and you no longer have a running
Director, Catalog, or Storage daemon?  There are several solutions:
<ol>
  <li>Bring up static versions of your Director, Catalog, and Storage
      daemon. </li>                               
  <li>Move your server to another machine.</li>
</ol>
The first option, is very difficult because it requires you to have
created a static version of the Director and the Storage daemon as
well as the Catalog. If the Catalog uses MySQL, this may or may not
be possible. In addition, to loading all these programs on a bare
system (quite possible), you will need to make sure you have a valid
driver for your tape drive.
<p>
The second suggestion is probably a much simpler solution, and one
I have done myself. To do so, you might want to consider the following
steps:
<ul>
  <li>If you are using MySQL, configure, build and install MySQL from
      source (or user rpms) on your new system.</li>
  <li>Load the <b>Bacula</b> source code onto your new system, configure,
      install it, and create the Bacula database.</li>
  <li>If you have a valid saved Bootstrap file as created for your
      damaged machine with WriteBootstrap, use it to restore the
      files to the damaged machine, where you have loaded a Bacula
      File daemon using the Bacula Rescue disk).</li>   
  <li>If you have the Bootstrap file, you should now be back up and
      running, if you do not have a Bootstrap file, continue with the
      suggestions below.</li>
  <li>Using <b>bscan</b> scan the last set of backup tapes into your
      MySQL or SQLite database.</li>     
  <li>Start Bacula, and using the Console <b>restore</b> command, 
      restore the last valid copy of the Bacula database and the
      the Bacula configuration files.</li>
   <li>Move the database to the correct location. </li>
   <li>Start and restart Bacula, and with the full database using
       the Console <b>restore</b> command, restore all the files 
       on the damaged machine, where you have loaded a Bacula File
       daemon using the Bacula Rescue disk.</li>
</ul>

<h2>Bugs and Other Considerations</h2>
<h4>Directory Modification and Access Times are Modified</h4>
When <b>Bacula</b> restores a directory, it first must create the
directory, then it populates the directory with its files and
subdirectories. The act of creating the files and subdirectories
updates both the modification and access times associated with
the directory itself.
As a consequence, all modification and access times of all directories
will be updated to the time of the restore. This could be &quot;corrected&quot;
by saving a list of all directories created during the restore, then when
all files are restored, visit each of those directories and reset their
modification and access times. This could possibly fail due to an out
of memory condition -- don't forget that during a bare metal recovery, there
is generally no swap file active. 
<p>I'm not too worried about this, and will probably provide restoration of
exact directory modification times in a future release. If anyone feels this
is more important that I do, please let me know.

<h4>Strange Bootstrap Files</h4>
If any of you look closely at the bootstrap file that is produced
and used for the restore (I sure do), you will probably notice that
the FileIndex item does not include all the files saved to the tape.
This is because in some instances there are duplicates (especially
in the case of an Incremental save), and in such circumstances, <b>Bacula</b>
restores only the last of multiple copies of a file or directory.

<h2>Disaster Recovery of Win32 Systems</h2>
<b>Bacula</b> does not yet really work completely well on Win32 systems,
particularly WinXP and Win2K because of permission problems. However,
Ludovic Strappazon has suggested an interesting way to backup and 
restore complete Win32 partions. Simply boot your Win32 system with
a Linux Rescue disk as described above for Linux, install a statically
linked Bacula, and backup any of the raw partitions you want. Then to
restore the system, you simply restore the raw partion or partions.
Here is the email that Ludovic recently sent on that subject:
<pre>
I've just finished testing my brand new cd LFS/Bacula 
with a raw Bacula backup and restore of my portable.

I can't resist sending you the results: look at the rates !!!

hunt-dir: Start Backup JobId 100, Job=HuntBackup.2003-04-17_12.58.26
hunt-dir: Bacula 1.30 (14Apr03): 17-Apr-2003 13:14
JobId:                  100
Job:                    HuntBackup.2003-04-17_12.58.26
FileSet:                RawPartition
Backup Level:           Full
Client:                 sauvegarde-fd
Start time:             17-Apr-2003 12:58
End time:               17-Apr-2003 13:14
Files Written:          1
Bytes Written:          10,058,586,272
Rate:                   10734.9 KB/s
Software Compression:   None
Volume names(s):        000103
Volume Session Id:      2
Volume Session Time:    1050576790
Last Volume Bytes:      10,080,883,520
FD termination status:  OK
SD termination status:  OK
Termination:            Backup OK

hunt-dir: Begin pruning Jobs.
hunt-dir: No Jobs found to prune.
hunt-dir: Begin pruning Files.
hunt-dir: No Files found to prune.
hunt-dir: End auto prune.


hunt-dir: Start Restore Job RestoreFilesHunt.2003-04-17_13.21.44
hunt-sd: Forward spacing to file 1.
hunt-dir: Bacula 1.30 (14Apr03): 17-Apr-2003 13:54
JobId:                  101
Job:                    RestoreFilesHunt.2003-04-17_13.21.44
Client:                 sauvegarde-fd
Start time:             17-Apr-2003 13:21
End time:               17-Apr-2003 13:54
Files Restored:         1
Bytes Restored:         10,056,130,560
Rate:                   5073.7 KB/s
FD termination status:  OK
Termination:            Restore OK

hunt-dir: Begin pruning Jobs.
hunt-dir: No Jobs found to prune.
hunt-dir: Begin pruning Files.
hunt-dir: No Files found to prune.
hunt-dir: End auto prune.
</pre>

<h2>Additional Resources</h2>
Many thanks to Charles Curley who wrote 
<a href="http://www.tldp.org/HOWTO/Linux-Complete-Backup-and-Recovery-HOWTO/index.html">
Linux Complete Backup and Recovery HOWTO</a> for the 
<a href="http://www.tldp.org/">The Linux Documentation Project</a>.
This is an excellent document on how to do Bare Metal Recovery on Linux
systems, and it was this document that made me realize that Bacula could
do the same thing. 
<p>
You can find quite a few additional resources, both commercial and
free at <a href="http://www.backupcentral.com">Storage Mountain</a>, formerly
known as Backup Central.
<p>And finally, the O'Reilly book,  &quot;Unix Backup & Recovery&quot; by
W. Curtis Preston covers virtually every backup and recovery topic including
bare metal recovery for a large range of Unix systems.

<page_prev name="The Bacula Console Restore Command" url="restore.html">
<page_next name="Catalog Maintenance" url="catmaintenance.html">
