Subsections

- General
- Important Considerations
- Steps to Take Before Disaster Strikes
- Bare Metal Recovery on Linux with a Bacula Rescue CD
- Requirements
- Directories
- Preparation for a Bare Metal Recovery
- Creating a Bacula Rescue CDROM
- Putting Multiple Systems on Your Rescue Disk
- Restoring a Client System
- Boot with your Bacula Rescue CDROM
  - Start the Network:
  - Partition Your Hard Disk(s):
  - Format Your Hard Disk(s):
  - Mount the Newly Formatted Disks:
  - Start the Network:
  - Restore and Start the File Daemon:
  - Restore Your Files:
  - Final Step:
  - Reboot:
- Restoring a Server
- Linux Problems or Bugs
- Bare Metal Recovery using a LiveCD
- FreeBSD Bare Metal Recovery
- Solaris Bare Metal Recovery
- Preparing Solaris Before a Disaster
- Bugs and Other Considerations
  - Directory Modification and Access Times are Modified on pre-1.30 Baculas :
  - Strange Bootstrap Files:
- Disaster Recovery of Win32 Systems
- Ownership and Permissions on Win32 Systems
- Alternate Disaster Recovery Suggestion for Win32 Systems
- Restoring to a Running System
- Additional Resources

#
Disaster Recovery Using Bacula

# General

When disaster strikes, you must have a plan, and you must have prepared in advance otherwise the work of recovering your system and your files will be considerably greater. For example, if you have not previously saved the partitioning information for your hard disk, how can you properly rebuild it if the disk must be replaced?

Unfortunately, many of the steps one must take before and immediately after a disaster are very operating system dependent. As a consequence, this chapter will discuss in detail disaster recovery (also called Bare Metal Recovery) for Linux and Solaris. For Solaris, the procedures are still quite manual. For FreeBSD the same procedures may be used but they are not yet developed. For Win32, a number of Bacula users have reported success using BartPE.

# Important Considerations

Here are a few important considerations concerning disaster recovery that you should take into account before a disaster strikes.

- If the building which houses your computers burns down or is otherwise destroyed, do you have off-site backup data?
- Disaster recovery is much easier if you have several machines. If you have a single machine, how will you handle unforeseen events if your only machine is down?
- Do you want to protect your whole system and use Bacula to recover everything? or do you want to try to restore your system from the original installation disks and apply any other updates and only restore user files?

# Steps to Take Before Disaster Strikes

- Create a Bacula Rescue CDROM for each of your Linux systems. Note, it is possible to create one CDROM by copying the bacula-hostname directory from each machine to the machine where you will be burning the CDROM, so if the Linux distro/version is the same, you can have a single CDROM that can recover multiple systems.
- Ensure that you always have a valid bootstrap file for your backup and that it is saved to an alternate machine. This will permit you to easily do a full restore of your system.
- If possible copy your catalog nightly to an alternate machine. If you have a valid bootstrap file, this is not necessary, but can be very useful if you do not want to reload everything. .
- Ensure that you always have a valid bootstrap file for your catalog backup that is saved to an alternate machine. This will permit you to restore your catalog more easily if needed.
- Test using the Bacula Rescue CDROM before you are forced to use it in an emergency situation.
- Make a copy of your Bacula .conf files, particularly your bacula-dir.conf, and your bacula-sd.conf files, because if your server goes down, these files will be needed to get it back up and running, and they can be difficult to rebuild from memory.

# Bare Metal Recovery on Linux with a Bacula Rescue CD

As an alternative to creating a Bacula Rescue CD, please see the section below entitled Bare Metal Recovery using a LiveCD.

The remainder of this section concerns recovering a Linux client computer (i.e. one running just the Bacula File daemon). The Solaris procedures can be found below under the Solaris Bare Metal Recovery section of this chapter.

Previously Bacula supported a floppy rescue disk. This code has been removed in 1.37.40 and later.

A so called "Bare Metal" recovery is one where you start with an empty hard disk and you restore your machine. There are also cases where you may lose a file or a directory and want it restored. Please see the previous chapter for more details for those cases.

The primary goals of the Bacula rescue CD are:

- NOT to be a general or universal recovery disk.
- to capture and setup a restore environment for a single system running as a Client.
- to capture the current state of the hard disks on your system, so that they can be easily restored from pre-generated scripts. Note, this is not done by any other rescue CDROM, as far as I am aware.
- to create and save a statically linked copy of your current Bacula FD. Thus you need no packages or other software to be installed before using this CDROM and the Bacula File daemon on it.
- to be relatively easy to create. In most cases you simply type make all in the rescue/linux/cdrom directory, then burn the ISO image created. In contrast, if you have looked at any of the documentation on how to remaster a CD or how to roll your own rescue CD, your head will spin (at least mine did).
- to be easy for you to add any additional files, binaries, or libraries to the CD.
- to build and work on any (or almost any) Linux flavor or release.
- you might ask why I don't use Knoppix or some other preprepared recovery disk, especially since Knoppix is very kind and provides the Bacula FD on their disk. The answer is that: I am more comfortable having my Linux boot up in rescue mode rather than another flavor. In addition, the Bacula rescue CDROM contains a complete snapshot of your disk partitioning, which is not the case with any other rescue disk. If your harddisk dies, do you remember all the partitions you had and how big they are? I don't, and without that information, you have little hope of reformatting your harddisk and rebuilding your system.

One of the main of the advantages of a Bacula Rescue CDROM is that it contains a bootable copy of your system, so you should be familiar with it.

Bare Metal Recovery assumes that you have the following items for your system:

- A Bacula Rescue CDROM containing a copy of your OS and a copy of your hard disk information, as well as a statically linked version of the Bacula File daemon. This chapter describes how to build such a CDROM.
- A full Bacula backup of your system possibly including Incremental or Differential backups since the last Full backup
- A second system running the Bacula Director, the Catalog, and the Storage daemon. (this is not an absolute requirement, but how to get around it is not yet documented here)

# Requirements

In addition, to the above assumptions, the following conditions or restrictions apply:

- Linux only -- tested only on SuSE and Fedora Core 4, but should work on other Linux distros.
- The scripts handle only SCSI and IDE disks.
- All partitions will be recreated, but only ext2, ext3, rfs and swap partitions will be reformatted. Any other partitions such as Windows FAT partitions will not be formatted by the scripts, but you can do it by hand.
- You are using either lilo or grub as a boot loader, and you know which one (not automatically detected).
- The partitioning and reformatting scripts *should* work with RAID devices, but probably not with other "complicated" disk partitioning/formatting schemes. They also should work with Reiser filesystems. Please check them carefully. You will probably need to edit the scripts by hand to make them work.
- You will need mkisofs (might be part of cdrtools, but is a separate rpm on my system); cdrecord or some other tool for burning the CDROM.

# Directories

To build the Bacula Rescue CDROM, you must get a copy of the rescue files. In version 1.37 and later, they are separate from the Bacula source. The rescue files are distributed as a compressed tar file on the Source Forge Bacula release area with the name bacula-rescue-xx.yy.zz.tar.gz. They are also automatically installed in /etc/bacula/rescue when installing by rpms. Another place you can find the rescue files is in the Source Forge Bacula SVN module named rescue.

Please read the README file in the main directory of the Rescue source code. Before using it, you must run configure and specify the location of the Bacula source code (not necessary if installed from rpms). This permits the rescue build scripts to automatically create a statically linked Bacula File daemon.

You will find the necessary scripts in linux/cdrom subdirectory of the rescue source code. If you installed the bacula rpm package the scripts will be found in the /etc/bacula/rescue/linux/cdrom directory.

# Preparation for a Bare Metal Recovery

Before you can do a Bare Metal recovery, you must create a Bacula Rescue CDROM, which will contain everything you need to begin recovery. This assumes that you will have your Director and Storage daemon running on a different machine. If you want to recover a machine where the Director and/or the database were previously running, things will be much more complicated.

# Creating a Bacula Rescue CDROM

You should probably make a new rescue CDROM each time you upgrade a major version of Bacula and whenever you modify your disk partitioning.

To build the rescue CDROM from source, you must first configure the rescue package, which is distributed separately from the source. The simplest procedure is for you to pre-build a static-bacula-fd taking care to use a minimum configuration such as:

    cd <bacula-source>
    ./configure \
       --prefix=/usr \
       --sbindir=/usr/sbin \
       --sysconfdir=/etc/bacula \
       --with-scriptdir=/etc/bacula \
       --enable-smartalloc \
       --enable-client-only \
       --enable-static-fd
    make

Then to copy the src/filed/static-bacula-fd, and a valid working copy of your bacula-fd.conf file to some specific directory. You can then proceed to configure the rescue package with:

    cd <bacula-rescue>
    ./configure \
       --with-static-fd=<directory-containing-static-bacula-fd> \
       --with-bacula-scripts=<directory-containing-bacula-fd.conf>
    cd linux/cdrom
    su
    (enter root password)
    make

The above instructions were for building the rescue CDROM from a bacula-rescue release. The advantage of the above procedure is that you have explicitly built your static-bacula-fd and you will supply the configuration with a working copy of bacula-fd.conf containing the correct Director name and password.

Alternatively when you configure the rescue package, you could supply it with the path to your Bacula source code, and when building the rescue disk, it will attepmpt to build a static-bacula-fd for you. We suggest you manually build your static Bacula File daemon and use the --with-static-fd option rather than letting the script attempt to build it (as shown below) because by manually building it, you can ensure that there are no errors, and you can execute it prior to putting it on the CD (e.g. ./bacula-fd -t).

To have the rescue scripts automatically build a static File daemon for you, use:

    cd <bacula-rescue>
    ./configure \
       --with-bacula=<bacula-source-directory>
    cd linux/cdrom
    su
    (enter root password)
    make

If you have multiple kernels installed on your system, you can specify which one using the following configuration option:

    cd <bacula-rescue>
    ./configure \
       --with-kernel=<kernel-version> \
       ...

For example a kernel-version might be 2.6.14-1.1653.

One additional option that can be useful is to specify the device name of your CDROM on the ./configure. To do so use:

    cd <bacula-rescue>
    ./configure \
       --with-dev=<device> \
       ...

Where <device> is typically replaced with something like /dev/hdc. This option is needed only if you have a recent OS that used device specifications rather than rather than ATA addresses, and you want to use the Bacula script make burn to automatically burn your ISO onto a CDROM.

For users of the bacula-rescue rpm the static bacula-fd has already been built and placed in /etc/bacula/rescue/linux/cdrom/bin/ along with a symbolic link to your /etc/bacula/bacula-fd.conf file. Rpm users only need to do the second step:

    cd /etc/bacula/rescue/linux/cdrom
    su (become root)
    make

At this point, if the scripts are successful, they should have done the following things:

- Made a copy of your kernel and its essential files.
- Copied a number of binary files from your system.
- Copied all the necessary shared libraries to run the above binary files.
- Made a statically-linked version of your File daemon and copied it into the CDROM build area.
- Made an ISO image and left it in bootcd.iso

Once this is accomplished, you need only burn it into a CDROM. This can be done directly from the makefile with:

    make burn

However, you may need to modify the Makefile to properly specify your CD burner as the detection process is complicated especially if you have two CDROMs or do not have cdrecord loaded on your system. Users of the rescue rpm package should definitely examine the Makefile since it was configured on the host used to produce the rpm package. If you find that the make burn does not work for you, try doing a:

    make scan

and use the output of that to modify the Makefile accordingly.

The "make" that you did above actually does the equivalent to the following:

    make kernel
    make binaries
    make bacula
    make iso

If you wish, you can modify what you put on the CDROM and redo any part of the make that you wish. For example, if you want to add a new directory, you might do the first three makes, then add a new directory to the CDROM, and finally do a "make iso". Please see the README file in the rescue/linux/cdrom or /etc/bacula/rescue/linux/cdromdirectory for instructions on changing the contents of the CDROM.

At the current time, the size of the CDROM is about 100MB (compressed to about 20MB), so there is quite a bit more room for additional programs. Keep in mind that when this CDROM is booted, *everything* is in memory, so the total size cannot exceed your memory size, and even then you will need some reserve memory for running programs, ...

Finally, if you want to be completely responsible for getting your own FD binary on the disk, you can do the following:

    cd linux/cdrom
    touch rpm_release
    make kernel
    make binaries
    make bacula
    (add your own Bacula FD to the bacula/bin directory)
    make iso
    rm -f rpm_release

The rpm_release file prevents the make bacula from attempting to build or copy a File daemon, so that you can do it before the "make iso" step. Once "make iso" is run, you can no longer add anything to the in-memory part of the image. You can still add files to the cdtree directory, and when you do a "make burn" they will be written to the CDROM. However, to access them, you must be able to mount the CDROM after booting it, then copy them into memory.

# Putting Multiple Systems on Your Rescue Disk

You can put multiple systems on the same rescue CD if you wish. This is because the information that is specific to your OS will be stored in the /bacula-hostname directory, where hostname is the name of the host on which you are building the CD. Suppose for example, you have two systems. One named client1 and one named client2. Assume also that your CD burner is on client1, and that is the machine we start on, and that we can ssh into client2 and also client2's disks are mounted on client1.

    ssh client2
    cd <bacula-source>
    ./configure --with-static-fd (our options)
    make
    cd <bacula-rescue-source>
    ./configure --with-bacula=<path-to-bacula-source>
    cd linux/cdrom
    su (become root)
    make bacula
    exit

Again, for rpm package users the above command set would be:

    ssh client2
    cd /etc/bacula/rescue/linux/cdrom
    su
    (enter root password)
    make bacula
    exit

Thus we have just built a Bacula rescue directory on client2. Now, on client1, we copy the appropriate directory to two places (explained below), then build an ISO and burn it:

    cd <bacula-source>
    ./configure (your options)
    make
    cd <bacula-rescue-source>
    ./configure --with-bacula=<path-to-bacula-source>
    cd linux/cdrom
    su (become root)
    c=/mnt/client2/home/user/bacula/rescue/linux/cdrom
    cp -a $c/roottree/bacula-client2 cdtree
    make
    make burn
    exit

And with the rpm package:

    cd /etc/bacula/rescue/linux/cdrom
    su
    (enter root password)
    c=/mnt/client2/etc/bacula/rescue/linux/cdrom
    cp -a $c/roottree/bacula-client2 cdtree
    make
    make burn
    exit

In summary, with the above commands, we first build a Bacula directory on client2 in roottree/bacula-client2, then we copied the bacula-client2 directory into the client1's cdtree so it will also be on the CD as a separate directory and thus can be read without booting the CDROM. Then we made and burned the CDROM for client1, which of course, contains the client2 data.

# Restoring a Client System

Now, let's assume that your hard disk has just died and that you have replaced it with an new identical drive. In addition, we assume that you have:

1. A recent Bacula backup (Full plus Incrementals)
2. A Bacula Rescue CDROM.
3. Your Bacula Director, Catalog, and Storage daemon running on another machine on your local network.

This is a relatively simple case, and later in this chapter, as time permits, we will discuss how you might recover from a situation where the machine that crashes is your main Bacula server (i.e. has the Director, the Catalog, and the Storage daemon).

You will take the following steps to get your system back up and running:

1. Boot with your Bacula Rescue CDROM.
2. Start the Network (local network)
3. Re-partition your hard disk(s) as it was before
4. Re-format your partitions
5. Restore the Bacula File daemon (static version)
6. Perform a Bacula restore of all your files
7. Re-install your boot loader
8. Reboot

Now for the details ...

# Boot with your Bacula Rescue CDROM

When the CDROM boots, you will be presented with a script that looks like:

          Welcome to the Bacula Rescue Disk 2.0.0
    To proceed, press the <ENTER> key or type "linux <runlevel>"

       linux 1     -> shell
       linux 2     -> login  (default if ENTER pressed)
       linux 3     -> network started and login (network not working yet)
       linux debug -> print debug during boot then login

Normally, at this point, you simply press ENTER. However, you may supply options for the boot if you wish.

Once it has booted, you will be requested to login something like:

    bash-3.1#

You will be in the root directory, and you can proceed to examine your system.

The complete Bacula rescue part of the CD will be in the directory: /bacula-hostname, where hostname is replaced by the name of the host machine on which you did the build for the CDROM. This naming procedure allows you to put multiple restore environments for each of your machines on a single CDROM if you so wish to do. Please see the README document in the rescue/linux/cdrom directory for more information on adding to the CDROM.

#### Start the Network:

At this point, you should bring up your network. Normally, this is quite simple and requires just a few commands. Please cd into the /bacula-hostname directory before continuing. To simplify your task, we have created a script that should work in most cases by typing:

    cd /bacula-hostname
    ./start_network

You can test it by pinging another machine, or pinging your broken machine machine from another machine. Do not proceed until your network is up.

#### Partition Your Hard Disk(s):

Assuming that your hard disk crashed and needs repartitioning, proceed with:

    ./partition.hda

If you have multiple disks, do the same for each of them. For SCSI disks, the repartition script will be named: partition.sda. If the script complains about the disk being in use, simply go back and redo the df command and umount commands until you no longer have your hard disk mounted. Note, in many cases, if your hard disk was seriously damaged or a new one installed, it will not automatically be mounted. If it is mounted, it is because the emergency kernel found one or more possibly valid partitions.

If for some reason this procedure does not work, you can use the information in partition.hda to re-partition your disks by hand using fdisk.

#### Format Your Hard Disk(s):

If you have repartitioned your hard disk, you must format it appropriately. The formatting script will put back swap partitions, normal Unix partitions (ext2) and journaled partitions (ext3) as well as Reiser partitions (rei). Do so by entering for each disk:

    ./format.hda

The format script will ask you if you want a block check done. We recommend to answer yes, but realize that for very large disks this can take hours.

#### Mount the Newly Formatted Disks:

Once the disks are partitioned and formatted, you can remount them with the mount_drives script. All your drives must be mounted for Bacula to be able to access them. Run the script as follows:

    ./mount_drives
    df

The df command will tell you if the drives are mounted. If not, re-run the script again. It isn't always easy to figure out and create the mount points and the mounts in the proper order, so repeating the ./mount_drives command will not cause any harm and will most likely work the second time. If not, correct it by hand before continuing.

#### Start the Network:

Before starting the File Daemon, you must bring up the network so that it can communicate with the Director and Storage daemon. Generally you can do so by running:

    ./start_network

#### Restore and Start the File Daemon:

If you have booted with a Bacula Rescue CDROM, your statically linked Bacula File daemon and the bacula-fd.conf file will be in the /bacula-hostname/bin directory. Make sure bacula-fd and bacula-fd.conf are both there.

If you did not already install a correct conf file, please edit the Bacula configuration file, create the working/pid/subsys directory if you haven't already done so above, and start Bacula. Before starting Bacula, you will need to move it and bacula-fd.conf from /bacula-hostname/bin, to the /mnt/disk/tmp directory so that it will be on your hard disk. Then start it with the following command:

    chroot /mnt/disk /tmp/bacula-fd -c /tmp/bacula-fd.conf

The above command starts the Bacula File daemon with the proper root disk location (i.e. /mnt/disk/tmp. If Bacula does not start, correct the problem and start it. You can check if it is running by entering:

    ps fax

You can kill Bacula by entering:

    kill -TERM <pid>

where pid is the first number printed in front of the first occurrence of bacula-fd in the ps fax command.

Now, you should be able to use another computer with Bacula installed to check the status by entering:

    status client=xxxx

into the Console program, where xxxx is the name of the client you are restoring.

One common problem is that your bacula-dir.conf may contain machine addresses that are not properly resolved on the stripped down system to be restored because it is not running DNS. This is particularly true for the address in the Storage resource of the Director, which may be very well resolved on the Director's machine, but not on the machine being restored and running the File daemon. In that case, be prepared to edit bacula-dir.conf to replace the name of the Storage daemon's domain name with its IP address.

#### Restore Your Files:

On the computer that is running the Director, you now run a restore command and select the files to be restored (normally everything), but before starting the restore, there is one final change you must make using the mod option. You must change the Where directory to be the root by using the mod option just before running the job and selecting Where. Set it to:

    /

then run the restore.

You might be tempted to avoid using chroot and running Bacula directly and then using a Where to specify a destination of /mnt/disk. This is possible, however, the current version of Bacula always restores files to the new location, and thus any soft links that have been specified with absolute paths will end up with /mnt/disk prefixed to them. In general this is not fatal to getting your system running, but be aware that you will have to fix these links if you do not use chroot.

#### Final Step:

At this point, the restore should have finished with no errors, and all your files will be restored. One last task remains and that is to write a new boot sector so that your machine will boot. For lilo, you enter the following command:

    ./run_lilo

If you are using grub instead of lilo, you must enter the following:

    ./run_grub

Note, I've had quite a number of problems with grub because it is rather complicated and not designed to install easily under a simplified system. In fact, the ./run_grub script is not going to work on most Linux 2.6 kernels with the latest grub, because grub-install references /usr/share/grub/... and it uses /dev/pts, which will not be in /dev if you are using udev (as do many 2.6 kernels).

So, if you experience errors or end up unexpectedly in a chroot shell, simply exit back to the normal shell and type in the appropriate commands from the run_grub script by hand until you get it to install. When you run the run_grub script, it will print the commands that you should manually enter if that is necessary.

In my more recent tests on FC4 running a 2.6.14 kernel and udev, I see that because of the above mentioned problems with grub, you will need version 1.8.2 rescue disk or later, and you may be more successful in getting grub to run by running it directly from the command line while logged into the rescue kernel using:

    /sbin/grub-install --root-directory=/mnt/disk /dev/hda

Note, in this case, you omit the chroot command, and you must replace /dev/hda with your boot device. If you don't know what your boot device is, run the ./run_grub script once and it will tell you.

Finally, I've even run into a case where grub-install was unable to rewrite the boot block. In my case, it produced the following error message:

    /dev/hdx does not have any corresponding BIOS drive.

The solution is to insure that all your disks are properly mounted on /mnt/disk, then do the following:

    chroot /mnt/disk
    mount /dev/pts

Then edit the file /boot/grub/grub.conf and uncomment the line that reads:

    #boot=/dev/hda

So that it reads:

    boot=/dev/hda

Note, the /dev/hda may be /dev/sda or possibly some other drive depending on your configuration, but in any case, it is the same as the one that you previously tried with grub-install.

Then, enter the following commands:

    grub --batch --device-map=/boot/grub/device.map \
      --config-file=/boot/grub/grub.conf --no-floppy
    root (hd0,0)
    setup (hd0)
    quit

If the grub call worked, you will get a prompt of grub> before the root, setup, and quit commands, and after entering the setup command, it should indicate that it successfully wrote the MBR (master boot record).

#### Reboot:

First unmount all your hard disks, otherwise they will not be cleanly shutdown, then reboot your machine by entering exit until you get to the main prompt then enter Ctrl-d. Once back to the main CDROM prompt, you will need to turn the power off, then back on to your machine to get it to reboot.

If everything went well, you should now be back up and running. If not, re-insert the emergency boot CDROM, boot, and figure out what is wrong.

# Restoring a Server

Above, we considered how to recover a client machine where a valid Bacula server was running on another machine. However, what happens if your server goes down and you no longer have a running Director, Catalog, or Storage daemon? There are several solutions:

1. Bring up static versions of your Director, Catalog, and Storage daemon on the damaged machine.

2. Move your server to another machine.

3. Use a Hot Spare Server on another Machine.

The first option, is very difficult because it requires you to have created a static version of the Director and the Storage daemon as well as the Catalog. If the Catalog uses MySQL or PostgreSQL, this may or may not be possible. In addition, to loading all these programs on a bare system (quite possible), you will need to make sure you have a valid driver for your tape drive.

The second suggestion is probably a much simpler solution, and one I have done myself. To do so, you might want to consider the following steps:

- If you are using MySQL or PostgreSQL, configure, build and install it from source (or use rpms) on your new system.
- Load the Bacula source code onto your new system, configure, install it, and create the Bacula database.
- Ideally, you will have a copy of all the Bacula conf files that were being used on your server. If not, you will at a minimum need create a bacula-dir.conf that has the same Client resource that was used to backup your system.
- If you have a valid saved Bootstrap file as created for your damaged machine with WriteBootstrap, use it to restore the files to the damaged machine, where you have loaded a static Bacula File daemon using the Bacula Rescue disk). This is done by using the restore command and at the yes/mod/no prompt, selecting mod then specifying the path to the bootstrap file.
- If you have the Bootstrap file, you should now be back up and running, if you do not have a Bootstrap file, continue with the suggestions below.
- Using bscan scan the last set of backup tapes into your MySQL, PostgreSQL or SQLite database.
- Start Bacula, and using the Console restore command, restore the last valid copy of the Bacula database and the Bacula configuration files.
- Move the database to the correct location.
- Start the database, and restart Bacula. Then use the Console restore command, restore all the files on the damaged machine, where you have loaded a Bacula File daemon using the Bacula Rescue disk.

For additional details of restoring your database, please see the Restoring When Things Go Wrong section of the Console Restore Command chapter of this manual.

# Linux Problems or Bugs

Since every flavor and every release of Linux is different, there are likely to be some small difficulties with the scripts, so please be prepared to edit them in a minimal environment. A rudimentary knowledge of vi is very useful. Also, these scripts do not do everything. You will need to reformat Windows partitions by hand, for example.

Getting the boot loader back can be a problem if you are using grub because it is so complicated. If all else fails, reboot your system from your floppy but using the restored disk image, then proceed to a reinstallation of grub (looking at the run-grub script can help). By contrast, lilo is a piece of cake.

# Bare Metal Recovery using a LiveCD

Rather than building a full Bacula Rescue CDROM, you can use any system rescue or LiveCD to recover your system. The big problem with most rescue or LiveCDs is that they are not designed to capture the current state of your system, so when you boot them on a damaged system, you might be somewhat lost -- e.g. how many of you remember your exact hard disk partitioning.

This lack can be easily corrected by running the part of the Bacula Rescue code that creates a directory containing a static-bacula-fd, a snapshot of your current system disk configuration, and scripts that help restoring it.

The procedure is similar to creating and your Bacula Rescue CDROM described above, but with the following differences:

Before a disaster strikes:

1. Run only the make bacula part of the Bacula Rescue procedure to create the static Bacula File daemon, and system disk snapshot.
2. Save the directory generated (more details below) preferrably on a CDROM or alternatively to some other system.
3. Possibly run make bacula every night as part of your backup process to ensure that you have a current snapshot of your system.

Then when disaster strikes, do the following:

1. Boot with your system rescue disk or LiveCD (e.g. Knoppix).
2. Start the Network (local network).
3. Copy the Bacula recovery directory to the damaged system using ftp, scp, wget or if your boot disk permits it reading it directly from a CDROM.
4. Continue as documented above as if you were using the Bacula Rescue CDROM -- that is.
5. Re-partition your hard disk(s) as it was before, if necessary.
6. Re-format your partitions, if necessary.
7. Restore the Bacula File daemon (static version).
8. Perform a Bacula restore of all your files.
9. Re-install your boot loader.
10. Reboot.

In order to create the Bacula recovery directory, you need a copy of the Bacula Rescue code as described above, and you must first configure that directory (and possibly your Bacula source) as described above in the section entitled Creating a Bacula Rescue CDROM.

Once the configuration is done, you can do the following to create the Bacula recovery directory:

    cd <bacula-rescue-source>/linux/cdrom
    su (become root)
    make bacula

The directory you want to save will be created in the current directory with the name bacula. You need only save that directory either as a directory or possibly as a compressed tar file. If you run this procedure on multiple machines, you will probably want to rename this directory to something like bacula-hostname.

# FreeBSD Bare Metal Recovery

The same basic techniques described above also apply to FreeBSD. Although we don't yet have a fully automated procedure, Alex Torres Molina has provided us with the following instructions with a few additions from Jesse Guardiani and Dan Langille:

1. Boot with the FreeBSD installation disk
2. Go to Custom, Partition and create your slices and go to Label and create the partitions that you want. Apply changes.
3. Go to Fixit to start an emergency console.
4. Create devs ad0 .. .. if they don't exist under /mnt2/dev (in my situation) with MAKEDEV. The device or devices you create depend on what hard drives you have. ad0 is your first ATA drive. da0 would by your first SCSI drive. Under OS version 5 and greater, your device files are most likely automatically created for you.
5. mkdir /mnt/disk this is the root of the new disk
6. mount /mnt2/dev/ad0s1a /mnt/disk mount /mnt2/dev/ad0s1c /mnt/disk/var mount /mnt2/dev/ad0s1d /mnt/disk/usr ..... The same hard drive issues as above apply here too. Note, under OS version 5 or higher, your disk devices may be in /dev not /mnt2/dev.
7. Network configuration (ifconfig xl0 ip/mask + route add default ip-gateway)
8. mkdir /mnt/disk/tmp
9. cd /mnt/disk/tmp
10. Copy bacula-fd and bacula-fd.conf to this path
11. If you need to, use sftp to copy files, after which you must do this: ln -s /mnt2/usr/bin /usr/bin
12. chmod u+x bacula-fd
13. Modify bacula-fd.conf to fit this machine
14. Copy /bin/sh to /mnt/disk, necessary for chroot
15. Don't forget to put your bacula-dir's IP address and domain name in /mnt/disk/etc/hosts if it's not on a public net. Otherwise the FD on the machine you are restoring to won't be able to contact the SD and DIR on the remote machine.
16. mkdir -p /mnt/disk/var/db/bacula
17. chroot /mnt/disk /tmp/bacula-fd -c /tmp/bacula-fd.conf to start bacula-fd
18. Now you can go to bacula-dir and restore the job with the entire contents of the broken server.
19. You must create /proc

# Solaris Bare Metal Recovery

The same basic techniques described above apply to Solaris:

- the same restrictions as those given for Linux apply
- you will need to create a Bacula Rescue disk

However, during the recovery phase, the boot and disk preparation procedures are different:

- there is no need to create an emergency boot disk since it is an integrated part of the Solaris boot.
- you must partition and format your hard disk by hand following manual procedures as described in W. Curtis Preston's book "Unix Backup & Recovery"

Once the disk is partitioned, formatted and mounted, you can continue with bringing up the network and reloading Bacula.

# Preparing Solaris Before a Disaster

As mentioned above, before a disaster strikes, you should prepare the information needed in the case of problems. To do so, in the rescue/solaris subdirectory enter:

    su
    ./getdiskinfo
    ./make_rescue_disk

The getdiskinfo script will, as in the case of Linux described above, create a subdirectory diskinfo containing the output from several system utilities. In addition, it will contain the output from the SysAudit program as described in Curtis Preston's book. This file diskinfo/sysaudit.bsi will contain the disk partitioning information that will allow you to manually follow the procedures in the "Unix Backup & Recovery" book to repartition and format your hard disk. In addition, the getdiskinfo script will create a start_network script.

Once you have your disks repartitioned and formatted, do the following:

- Start Your Network with the start_network script
- Restore the Bacula File daemon as documented above
- Perform a Bacula restore of all your files using the same commands as described above for Linux
- Re-install your boot loader using the instructions outlined in the "Unix Backup & Recovery" book using installboot

# Bugs and Other Considerations

#### Directory Modification and Access Times are Modified on pre-1.30 Baculas :

When a pre-1.30 version of Bacula restores a directory, it first must create the directory, then it populates the directory with its files and subdirectories. The act of creating the files and subdirectories updates both the modification and access times associated with the directory itself. As a consequence, all modification and access times of all directories will be updated to the time of the restore.

This has been corrected in Bacula version 1.30 and later. The directory modification and access times are reset to the value saved in the backup after all the files and subdirectories have been restored. This has been tested and verified on normal restore operations, but not verified during a bare metal recovery.

#### Strange Bootstrap Files:

If any of you look closely at the bootstrap file that is produced and used for the restore (I sure do), you will probably notice that the FileIndex item does not include all the files saved to the tape. This is because in some instances there are duplicates (especially in the case of an Incremental save), and in such circumstances, Bacula restores only the last of multiple copies of a file or directory.

# Disaster Recovery of Win32 Systems

Due to open system files, and registry problems, Bacula cannot save and restore a complete Win2K/XP/NT environment.

A suggestion by Damian Coutts using Microsoft's NTBackup utility in conjunction with Bacula should permit a Full bare metal restore of Win2K/XP (and possibly NT systems). His suggestion is to do an NTBackup of the critical system state prior to running a Bacula backup with the following command:

    ntbackup backup systemstate /F c:\systemstate.bkf

The backup is the command, the systemstate says to backup only the system state and not all the user files, and the /F c:\systemstate.bkf specifies where to write the state file. this file must then be saved and restored by Bacula. This command can be put in a Client Run Before Job directive so that it is automatically run during each backup, and thus saved to a Bacula Volume.

To restore the system state, you first reload a base operating system, then you would use Bacula to restore all the users files and to recover the c:\systemstate.bkf file, and finally, run NTBackup and catalogue the system statefile, and then select it for restore. The documentation says you can't run a command line restore of the systemstate.

This procedure has been confirmed to work by Ludovic Strappazon -- many thanks!

A new tool is provided in the form of a bacula plugin for the BartPE rescue CD. BartPE is a self-contained WindowsXP boot CD which you can make using the PeBuilder tools available at http://www.nu2.nu/pebuilder/ and a valid Windows XP SP1 CDROM. The plugin is provided as a zip archive. Unzip the file and copy the bacula directory into the plugin directory of your BartPE installation. Edit the configuration files to suit your installation and build your CD according to the instructions at Bart's site. This will permit you to boot from the cd, configure and start networking, start the bacula file client and access your director with the console program. The programs menu on the booted CD contains entries to install the file client service, start the file client service, and start the WX-Console. You can also open a command line window and CD Programs\Bacula and run the command line console bconsole.

# Ownership and Permissions on Win32 Systems

Bacula versions after 1.31 should properly restore ownership and permissions on all WinNT/XP/2K systems. If you do experience problems, generally in restores to alternate directories because higher level directories were not backed up by Bacula, you can correct any problems with the SetACL available under the GPL license at: http://sourceforge.net/projects/setacl/.

# Alternate Disaster Recovery Suggestion for Win32 Systems

Ludovic Strappazon has suggested an interesting way to backup and restore complete Win32 partitions. Simply boot your Win32 system with a Linux Rescue disk as described above for Linux, install a statically linked Bacula, and backup any of the raw partitions you want. Then to restore the system, you simply restore the raw partition or partitions. Here is the email that Ludovic recently sent on that subject:

    I've just finished testing my brand new cd LFS/Bacula
    with a raw Bacula backup and restore of my portable.
    I can't resist sending you the results: look at the rates !!!
    hunt-dir: Start Backup JobId 100, Job=HuntBackup.2003-04-17_12.58.26
    hunt-dir: Bacula 1.30 (14Apr03): 17-Apr-2003 13:14
    JobId:                  100
    Job:                    HuntBackup.2003-04-17_12.58.26
    FileSet:                RawPartition
    Backup Level:           Full
    Client:                 sauvegarde-fd
    Start time:             17-Apr-2003 12:58
    End time:               17-Apr-2003 13:14
    Files Written:          1
    Bytes Written:          10,058,586,272
    Rate:                   10734.9 KB/s
    Software Compression:   None
    Volume names(s):        000103
    Volume Session Id:      2
    Volume Session Time:    1050576790
    Last Volume Bytes:      10,080,883,520
    FD termination status:  OK
    SD termination status:  OK
    Termination:            Backup OK
    hunt-dir: Begin pruning Jobs.
    hunt-dir: No Jobs found to prune.
    hunt-dir: Begin pruning Files.
    hunt-dir: No Files found to prune.
    hunt-dir: End auto prune.
    hunt-dir: Start Restore Job RestoreFilesHunt.2003-04-17_13.21.44
    hunt-sd: Forward spacing to file 1.
    hunt-dir: Bacula 1.30 (14Apr03): 17-Apr-2003 13:54
    JobId:                  101
    Job:                    RestoreFilesHunt.2003-04-17_13.21.44
    Client:                 sauvegarde-fd
    Start time:             17-Apr-2003 13:21
    End time:               17-Apr-2003 13:54
    Files Restored:         1
    Bytes Restored:         10,056,130,560
    Rate:                   5073.7 KB/s
    FD termination status:  OK
    Termination:            Restore OK
    hunt-dir: Begin pruning Jobs.
    hunt-dir: No Jobs found to prune.
    hunt-dir: Begin pruning Files.
    hunt-dir: No Files found to prune.
    hunt-dir: End auto prune.

# Restoring to a Running System

If for some reason you want to do a Full restore to a system that has a working kernel (not recommended), you will need to take care not to overwrite the following files:

    /etc/grub.conf
    /etc/X11/Conf
    /etc/fstab
    /etc/mtab
    /lib/modules
    /usr/modules
    /usr/X11R6
    /etc/modules.conf

# Additional Resources

Many thanks to Charles Curley who wrote Linux Complete Backup and Recovery HOWTO for the The Linux Documentation Project. This is an excellent document on how to do Bare Metal Recovery on Linux systems, and it was this document that made me realize that Bacula could do the same thing.

You can find quite a few additional resources, both commercial and free at Storage Mountain, formerly known as Backup Central.

And finally, the O'Reilly book, "Unix Backup & Recovery" by W. Curtis Preston covers virtually every backup and recovery topic including bare metal recovery for a large range of Unix systems.