Bacula 1.36 User's Guide |
Basic Volume Management | Index | Backup Strategies
# Using Pools to Manage Volumes If you manage 5 or 10 machines and have a nice tape backup, you don't need Pools, and you may wonder what they are good for. In this chapter, you will see that Pools can help you optimize disk storage space. The same techniques can be applied to a shop that has multiple tape drives, or that wants to mount various different Volumes to meet their needs. The rest of this chapter will give an example involving backup to disk Volumes, but most of the information applies equally well for tape Volumes. ## The Problem A site that I administer (a charitable organization) had a tape DDS-3 tape drive that was failing. The exact reason for the failure is still unknown. Worse yet, their full backup size is about 15GB whereas the capacity of their broken DDS-3 was at best 8GB (rated 6/12). A new DDS-4 tape drive and the necessary cassettes was more expensive than their budget could handle. ## The Solution They want to maintain 6 months of backup data, and be able to access the old files on a daily basis for a week, a weekly basis for a month, then monthly for 6 months. In addition, and offsite capability was not needed (well perhaps it really is, but it was never used). Their daily changes amount to about 300MB on the average, or about 2GB per week. As a consequence, the total volume of data they need to keep to meet their needs is about 100GB (15GB x 6 + 2GB x 5 + 0.3 x 7) = 102.1GB. The chosen solution was to buy a 120GB hard disk for next to nothing -- far less than 1/10th the price of a tape drive and the cassettes to handle the same amount of data, and to have Bacula write to disk files. The rest of this chapter will explain how to setup Bacula so that it would automatically manage a set of disk files with the minimum intervention on my part. The system has been running since 22 January 2004 until today (08 April 2004) with no intervention. Since we have not yet crossed the six month boundary, we still lack some data to be sure the system performs as desired. ## Overall Design Getting Bacula to write to disk rather than tape in the simplest case is rather easy, and is documented in the previous chapter. In addition, all the directives discussed here are explained in that chapter. We'll leave it to you to look at the details there. If you haven't read it and are not familiar with Pools, you probably should at least read it once quickly for the ideas before continuing here. One needs to consider about what happens if we have only a single large Bacula Volume defined on our hard disk. Everything works fine until the Volume fills, then Bacula will ask you to mount a new Volume. This same problem applies to the use of tape Volumes if your tape fills. Being a hard disk and the only one you have, this will be a bit of a problem. It should be obvious that it is better to use a number of smaller Volumes and arrange for Bacula to automatically recycle them so that the disk storage space can be reused. The other problem with a single Volume, is that at the current time (1.34.0) Bacula does not seek within a disk Volume, so restoring a single file can take more time than one would expect. As mentioned, the solution is to have multiple Volumes, or files on the disk. To do so, we need to limit the use and thus the size of a single Volume, by time, by number of jobs, or by size. Any of these would work, but we chose to limit the use of a single Volume by putting a single job in each Volume with the exception of Volumes containing Incremental backup where there will be 6 jobs (a week's worth of data) per volume. The details of this will be discussed shortly. The next problem to resolve is recycling of Volumes. As you noted from above, the requirements are to be able to restore monthly for 6 months, weekly for a month, and daily for a week. So to simplify things, why not do a Full save once a month, a Differential save once a week, and Incremental saves daily. Now since each of these different kinds of saves needs to remain valid for differing periods, the simplest way to do this (and possibly the only) is to have a separate Pool for each backup type. The decision was to use three Pools: one for Full saves, one for Differential saves, and one for Incremental saves, and each would have a different number of volumes and a different Retention period to accomplish the requirements. ### Full Pool Putting a single Full backup on each Volume, will require six Full save Volumes, and a retention period of six months. The Pool needed to do that is:
    Pool {
      Name = Full-Pool
      Pool Type = Backup
      Recycle = yes
      AutoPrune = yes
      Volume Retention = 6 months
      Accept Any Volume = yes
      Maximum Volume Jobs = 1
      Label Format = Full-
      Maximum Volumes = 6
    }
Since these are disk Volumes, no space is lost by having separate Volumes for each backup (done once a month in this case). The items to note are the retention period of six months (i.e. they are recycled after 6 months), that there is one job per volume (Maximum Volume Jobs = 1), the volumes will be labeled Full-0001, ... Full-0006 automatically. One could have labeled these manual from the start, but why not use the features of Bacula. ### Differential Pool For the Differential backup Pool, we choose a retention period of a bit longer than a month and ensure that there is at least one Volume for each of the maximum of five weeks in a month. So the following works:
    Pool {
      Name = Diff-Pool
      Pool Type = Backup
      Recycle = yes
      AutoPrune = yes
      Volume Retention = 40 days
      Accept Any Volume = yes
      Maximum Volume Jobs = 1
      Label Format = Diff-
      Maximum Volumes = 6
    }
As you can see, the Differential Pool can grow to a maximum of six volumes, and the Volumes are retained 40 days and there after can be recycled. Finally there is one job per volume. This, of course, could be tightened up a lot, but the expense here is a few GB which is not too serious. ### Incremental Pool Finally, here is the resource for the Incremental Pool:
    Pool {
      Name = Inc-Pool
      Pool Type = Backup
      Recycle = yes
      AutoPrune = yes
      Volume Retention = 20 days
      Accept Any Volume = yes
      Maximum Volume Jobs = 6
      Label Format = Inc-
      Maximum Volumes = 5
    }
We keep the data for 20 days rather than just a week as the needs require. To reduce the proliferation of volume names, we keep a week's worth of data (6 incremental backups) in each Volume. In practice, the retention period should be set to just a bit more than a week and keep only two or three volumes instead of five. Again, the lost is very little and as the system reaches the full steady state, we can adjust these values so that the total disk usage doesn't exceed the disk capacity. ## The Actual Conf Files The following example shows you the actual files used, with only a few minor modifications to simplify things. The Director's configuration file is as follows:
    Director {          # define myself
      Name = bacula-dir
      DIRport = 9101
      QueryFile = "/home/bacula/bin/query.sql"
      WorkingDirectory = "/home/bacula/working"
      PidDirectory = "/home/bacula/working"
      Maximum Concurrent Jobs = 1
      Password = " "
      Messages = Standard
    }

    #   By default, this job will back up to disk in /tmp
    Job {
      Name = client
      Type = Backup
      Client = client-fd
      FileSet = "Full Set"
      Schedule = "WeeklyCycle"
      Storage = File
      Messages = Standard
      Pool = Default
      Full Backup Pool = Full-Pool
      Incremental Backup Pool = Inc-Pool
      Differential Backup Pool = Diff-Pool
      Write Bootstrap = "/home/bacula/working/client.bsr"
      Priority = 10
    }

    # List of files to be backed up
    FileSet {
      Name = "Full Set"
      Include = signature=SHA1 compression=GZIP9 {
        /
        /usr
        /home
      }
      Exclude = {
         /proc /tmp /.journal /.fsck
      }
    }

    Schedule {
      Name = "WeeklyCycle"
      Run = Full 1st sun at 1:05
      Run = Differential 2nd-5th sun at 1:05
      Run = Incremental mon-sat at 1:05
    }

    Client {
      Name = client-fd
      Address = client
      FDPort = 9102
      Catalog = MyCatalog
      Password = " "
      AutoPrune = yes      # Prune expired Jobs/Files
      Job Retention = 6 months
      File Retention = 60 days
    }

    Storage {
      Name = File
      Address = localhost
      SDPort = 9103
      Password = " "
      Device = FileStorage
      Media Type = File
    }

    Catalog {
      Name = MyCatalog
      dbname = bacula; user = bacula; password = ""
    }

    Pool {
      Name = Full-Pool
      Pool Type = Backup
      Recycle = yes           # automatically recycle Volumes
      AutoPrune = yes         # Prune expired volumes
      Volume Retention = 6 months
      Accept Any Volume = yes # write on any volume in the pool
      Maximum Volume Jobs = 1
      Label Format = Full-
      Maximum Volumes = 6
    }

    Pool {
      Name = Inc-Pool
      Pool Type = Backup
      Recycle = yes           # automatically recycle Volumes
      AutoPrune = yes         # Prune expired volumes
      Volume Retention = 20 days
      Accept Any Volume = yes
      Maximum Volume Jobs = 6
      Label Format = Inc-
      Maximum Volumes = 5
    }

    Pool {
      Name = Diff-Pool
      Pool Type = Backup
      Recycle = yes
      AutoPrune = yes
      Volume Retention = 40 days
      Accept Any Volume = yes
      Maximum Volume Jobs = 1
      Label Format = Diff-
      Maximum Volumes = 6
    }

    Messages {
      Name = Standard
      mailcommand = "bsmtp -h mail.domain.com -f \"\(Bacula\) %r\"
          -s \"Bacula: %t %e of %c %l\" %r"
      operatorcommand = "bsmtp -h mail.domain.com -f \"\(Bacula\) %r\"
          -s \"Bacula: Intervention needed for %j\" %r"
      mail = root@domain.com = all, !skipped
      operator = root@domain.com = mount
      console = all, !skipped, !saved
      append = "/home/bacula/bin/log" = all, !skipped
    }
and the Storage daemon's configuration file is:
    Storage {               # definition of myself
      Name = bacula-sd
      SDPort = 9103       # Director's port
      WorkingDirectory = "/home/bacula/working"
      Pid Directory = "/home/bacula/working"
    }

    Director {
      Name = bacula-dir
      Password = " "
    }

    Device {
      Name = FileStorage
      Media Type = File
      Archive Device = /files/bacula
      LabelMedia = yes;    # lets Bacula label unlabeled media
      Random Access = Yes;
      AutomaticMount = yes;   # when device opened, read it
      RemovableMedia = no;
      AlwaysOpen = no;
    }

    Messages {
      Name = Standard
      director = bacula-dir = all
    }
Basic Volume Management | Index | Backup Strategies
Bacula 1.36 User's Guide The Network Backup Solution | Copyright © 2000-2004 Kern Sibbald and John Walker