
#include "bacula_template.inc" title="Using Pools to Manage Disk Volumes" num=13

<h1>Using Pools to Manage Disk Volumes</h1>
If you manage 5 or 10 machines and have a nice tape
backup, you don't need Pools, and you may wonder what they are good
for. In this chapter, you will see that Pools can help you optimize
disk storage space. The same techniques can be applied to a shop
that has multiple tape drives, or that wants to mount various
different Volumes to meet their needs.

<h2>The Problem</h2>
A site that I administer (a charitable organization) had a tape
DDS-3 tape drive that was failing.  The exact reason for the
failure is still unknown.  Worse yet, their full backup size is
about 15GB whereas the capacity of their broken DDS-3 was at best
8GB (rated 6/12).  A new DDS-4 tape drive and the necessary
cassettes was more expensive than their budget could handle.

<h2>The Solution</h2>
They want to maintain 6 months of backup data, and be able to
access the old files on a daily basis for a week, a weekly basis
for a month, then monthly for 6 months.  In addition, and offsite
capability was not needed (well perhaps it really is, but it was
never used).  Their daily changes amount to about 300MB on the
average, or about 2GB per week.
<p>As a consequence, the total volume of data they need to keep to 
meet their needs is about 100GB (15GB x 6 + 2GB x 5 + 0.3 x 7) = 102.1GB.
<p>
The chosen solution was to buy a 120GB hard disk for next to
nothing -- far less than 1/10th the price of a tape drive and the
cassettes to handle the same amount of data, and to have Bacula
write to disk files.
<p>
The rest of this chapter will explain how to setup Bacula so
that it would automatically manage a set of disk files with the 
minimum intervention on my part. The system has been running since
22 January 2004 until today (08 April 2004) with no intervention. 
Since we have not yet crossed the six month boundary, 
we still lack some data to be sure the system performs as
desired.

<h2>Overall Design</h2>
Getting Bacula to write to disk rather than tape in the simplest
case is rather easy, and is documented in the previous chapter. 
In addition, all the directives discussed here are
explained in that chapter. 
We'll leave it to you to look at the details there.
If you haven't read it and are not familiar with Pools, you
probably should at least read it once quickly for the ideas 
before continuing here.
<p>
One needs to consider about what happens if we have only a single
large Bacula Volume defined on our hard disk.  Everything
works fine until the Volume fills, then Bacula will ask you
to mount a new Volume. Being a hard disk and the only one
you have, this will be a bit of a problem.
It should be obvious that it is better to use a number
of smaller Volumes and arrange for Bacula to automatically
recycle them so that the disk storage space can be reused.
The other problem with a single Volume, is that at the current
time (1.34.0) Bacula does not seek within a disk Volume, so
restoring a single file can take more time than one would expect.
<p>
As mentioned, the solution is to have multiple Volumes, or files
on the disk.  To do so, we need to limit the use and thus the size of a
single Volume, by time, by number of jobs, or by size.  Any of
these would work, but we chose to limit the use of a single Volume
by putting a single job in each Volume with the exception of
Volumes containing Incremental backup where there will be
6 jobs (a week's worth of data) per volume. The details of
this will be discussed shortly.
<p>
The next problem to resolve is recycling of Volumes. As you
noted from above, the requirements are to be able to restore
monthly for 6 months, weekly for a month, and daily for a 
week. So to simplify things, why not do a Full save once
a month, a Differential save once a week, and Incremental
saves daily. Now since each of these different kinds of
saves needs to remain valid for differing periods, the simplest
way to do this (and possibly the only) is to have a separate
Pool for each backup type.
<p>
The decision was to use three Pools: one for Full saves, one for
Differential saves, and one for Incremental saves, and each
would have a different number of volumes and a
different Retention period to accomplish the requirements.

<h3>Full Pool</h3>
Putting a single Full backup on each Volume, will require six
Full save Volumes, and a retention period of six months.  The
Pool needed to do that is:

<pre>
Pool {
  Name = Full-Pool
  Pool Type = Backup
  Recycle = yes         
  AutoPrune = yes       
  Volume Retention = 6 months
  Accept Any Volume = yes 
  Maximum Volume Jobs = 1
  Label Format = Full-
  Maximum Volumes = 6
}
</pre>
Since these are disk Volumes, no space is lost by having separate
Volumes for each backup (done once a month in this case). The items to
note are the retention period of six months (i.e. they are
recycled after 6 months), that there is one
job per volume (Maximum Volume Jobs = 1), the volumes will be labeled
Full-0001, ... Full-0006 automatically.  One could have labeled these
manual from the start, but why not use the features of Bacula.

<h3>Differential Pool</h3>
For the Differential backup Pool, we choose a retention period of a bit
longer than a month and ensure that there is at least one Volume
for each of the maximum of five weeks in a month. So the following works:
<pre>
Pool {
  Name = Diff-Pool
  Pool Type = Backup
  Recycle = yes        
  AutoPrune = yes      
  Volume Retention = 40 days
  Accept Any Volume = yes 
  Maximum Volume Jobs = 1
  Label Format = Diff-
  Maximum Volumes = 6
}
</pre>
As you can see, the Differential Pool can grow to a maximum of
six volumes, and the Volumes are retained 40 days and there after
can be recycled.  Finally there is one job per volume.  This, of
course, could be tightened up a lot, but the expense here is a
few GB which is not too serious.

<h3>Incremental Pool</h3>
Finally, here is the resource for the Incremental Pool:
<pre>
Pool {
  Name = Inc-Pool
  Pool Type = Backup
  Recycle = yes   
  AutoPrune = yes
  Volume Retention = 20 days
  Accept Any Volume = yes
  Maximum Volume Jobs = 6
  Label Format = Inc-
  Maximum Volumes = 5
}
</pre>
We keep the data for 20 days
rather than just a week as the needs require. To reduce the
proliferation of volume names, we keep a week's worth
of data (6 incremental backups) in each Volume.  In practice,
the retention period should be set to just a bit more than a week
and keep only two or three volumes instead of five.  Again, the
lost is very little and as the system reaches the full steady
state, we can adjust these values so that the total disk usage
doesn't exceed the disk capacity.


<a name="Example"></a>
<h2>The Actual Conf Files</h2>
The following example shows you the actual files
used, with only a few minor modifications
to simplify things.


<p>The Director's configuration file is as follows:
<pre>

Director {          # define myself
  Name = bacula-dir
  DIRport = 9101 
  QueryFile = "/home/bacula/bin/query.sql"
  WorkingDirectory = "/home/bacula/working"
  PidDirectory = "/home/bacula/working"
  Maximum Concurrent Jobs = 1
  Password = " "
  Messages = Standard
}

#   By default, this job will back up to disk in /tmp
Job {
  Name = client
  Type = Backup
  Client = client-fd 
  FileSet = "Full Set"
  Schedule = "WeeklyCycle"
  Storage = File
  Messages = Standard
  Pool = Default
  Full Backup Pool = Full-Pool
  Incremental Backup Pool = Inc-Pool
  Differential Backup Pool = Diff-Pool
  Write Bootstrap = "/home/bacula/working/client.bsr"
  Priority = 10
}


# List of files to be backed up
FileSet {
  Name = "Full Set"
  Include = signature=SHA1 compression=GZIP9 {
    /
    /usr
    /home
  }
  Exclude = { 
     /proc /tmp /.journal /.fsck 
  }
}

Schedule {
  Name = "WeeklyCycle"
  Run = Full 1st sun at 1:05
  Run = Differential 2nd-5th sun at 1:05
  Run = Incremental mon-sat at 1:05
}

Client {
  Name = client-fd
  Address = client
  FDPort = 9102
  Catalog = MyCatalog
  Password = " "
  AutoPrune = yes      # Prune expired Jobs/Files
  Job Retention = 6 months
  File Retention = 60 days
}

Storage {
  Name = File
  Address = localhost
  SDPort = 9103
  Password = " "
  Device = FileStorage
  Media Type = File
}

Catalog {
  Name = MyCatalog
  dbname = bacula; user = bacula; password = ""
}

Pool {
  Name = Full-Pool
  Pool Type = Backup
  Recycle = yes           # automatically recycle Volumes
  AutoPrune = yes         # Prune expired volumes
  Volume Retention = 6 months
  Accept Any Volume = yes # write on any volume in the pool
  Maximum Volume Jobs = 1
  Label Format = Full-
  Maximum Volumes = 6
}

Pool {
  Name = Inc-Pool
  Pool Type = Backup
  Recycle = yes           # automatically recycle Volumes
  AutoPrune = yes         # Prune expired volumes
  Volume Retention = 20 days
  Accept Any Volume = yes 
  Maximum Volume Jobs = 6
  Label Format = Inc-
  Maximum Volumes = 5
}

Pool {
  Name = Diff-Pool
  Pool Type = Backup
  Recycle = yes           
  AutoPrune = yes         
  Volume Retention = 40 days  
  Accept Any Volume = yes   
  Maximum Volume Jobs = 1
  Label Format = Diff-
  Maximum Volumes = 6
}

Messages {
  Name = Standard
  mailcommand = "bsmtp -h mail.domain.com -f \"\(Bacula\) %r\" 
      -s \"Bacula: %t %e of %c %l\" %r"
  operatorcommand = "bsmtp -h mail.domain.com -f \"\(Bacula\) %r\" 
      -s \"Bacula: Intervention needed for %j\" %r"
  mail = root@domain.com = all, !skipped            
  operator = root@domain.com = mount
  console = all, !skipped, !saved
  append = "/home/bacula/bin/log" = all, !skipped
}

</pre>

and the Storage daemon's configuration file is:

<pre>

Storage {               # definition of myself
  Name = bacula-sd
  SDPort = 9103       # Director's port      
  WorkingDirectory = "/home/bacula/working"
  Pid Directory = "/home/bacula/working"
}

Director {
  Name = bacula-dir
  Password = " "
}

Device {
  Name = FileStorage
  Media Type = File
  Archive Device = /files/bacula
  LabelMedia = yes;    # lets Bacula label unlabeled media
  Random Access = Yes;
  AutomaticMount = yes;   # when device opened, read it
  RemovableMedia = no;
  AlwaysOpen = no;
}

Messages {
  Name = Standard
  director = bacula-dir = all
}
</pre>
 
<page_prev name="Backing Up to Disk Volumes" url="disk.html">
<page_next name="Backup Strategies" url="strategies.html">
