
#include "bacula_template.inc" title="Utility Programs" num=17

<h1>Volume Utility Tools</h1>
This document describes the utility programs written
to aid Bacula users and developers in dealing with
Volumes external to Bacula.

<h3>Specifying the Configuration File</h3>
Starting with version 1.27, each of the following programs
requires a valid Storage daemon configuration
file (actually, the only part of the configuration file
that these programs need is
the <b>Device</b> resource definitions). 
This permits the programs to find the configuration parameters
for your archive device (generally a tape drive). By
default, they read <b>bacula-sd.conf</b> in the current directory,
but you may specify a different     
configuration file using the <b>-c</b> option.

<h3>Specifying a Device Name For a Tape</h3>
Each of these programs require a            
<b>device-name</b> where the Volume can be found.
In the case of a tape, this is the
physical device name such as <b>/dev/nst0</b> or 
<b>/dev/rmt/0ubn</b> depending on your system. For the
program to work, it must find the identical name in
the Device resource of the configuration file. 
See below for specifying Volume names.

<h3>Specifying a Device Name For a File</h3>
If you are attempting to read or write
an archive file rather than
a tape, the <b>device-name</b> should be the full 
path to the archive location including
the filename. The filename (last part of the specification) will
be stripped and used as the Volume name, and the path
(first part before the filename) must have the same
entry in the configuration file.  So, the path is equivalent
to the archive device name, and the filename is equivalent
to the volume name.

<h3>Specifying Volumes</h3>
In general, you must specify the Volume name to each of
the programs below (with the exception of <b>btape</b>).
The best method to do so is to specify a <b>bootstrap</b>
file on the command line with the <b>-b</b> option. As
part of the bootstrap file, you will then specify the
Volume name or Volume names if more than one volume
is needed. For
example, suppose you want to read tapes <b>tape1</b> and 
<b>tape2</b>.  First construct a <b>bootstrap</b> file
named say, <b>list.bsr</b> which contains:
<pre>
Volume=test1|test2
</pre>
where each Volume is separated by a vertical bar.
Then simply use:
<pre>
./bls -b list.bsr /dev/nst0
</pre>
In the case of Bacula Volumes that are on files, you may simply
append volumes as follows:
<pre>
./bls /tmp/test1\|test2
</pre>
where the backslash (\) was necessary as a shell escape to permit
entering the vertical bar (|).
<p>
And finally, if you feel that specifying a Volume name is a bit
complicated with a bootstrap file, you can use the <b>-V</b> option
(on all programs except <b>bcopy</b>) to specify one or more Volume
names separated by the vertical bar (|). For example,
<pre>
./bls -V Vol001 /dev/nst0
</pre>
You may also specify an asterisk (*) to indicate that the program
should accept any volume.
For example:
<pre>
./bls -V* /dev/nst0
</pre>
 
<h1><a name="bls">bls</a></h1>
<b>bls</b> can be used to do an <b>ls</b> type listing of a 
<b>Bacula</b> tape or file.  It is called:
<pre>
Usage: bls [-d debug_level] &lt;device-name&gt;
       -b &lt;file&gt;       specify a bootstrap file
       -c &lt;file&gt;       specify a configuration file
       -d &lt;level&gt;       specify a debug level
       -e &lt;file&gt;       exclude list
       -i &lt;file&gt;       include list
       -j              list jobs
       -k              list blocks
       -L              list tape label
    (none of above)    list saved files
       -p              proceed inspite of I/O errors
       -t              use default tape device
       -v              be verbose
       -V              specify Volume names (separated by |)
       -?              print this message
</pre>
For example, to list the contents of a tape:
<pre>
./bls -V Volume-name /dev/nst0
</pre>
Or to list the contents of a file:
<pre>
./bls /tmp/Volume-name
or
./bls -V Volume-name /tmp
</pre>
Note that, in the case of a file, the Volume name becomes 
the filename, so in the above example,
you will replace the <b>xxx</b> with the name of the
volume (file) you wrote.
<p>
Normally if no options are specified, <b>bls</b> will produce the
equivalent output to the <b>ls -l</b> command for each file
on the tape. Using other options listed above, it is possible
to display only the Job records, only the tape blocks, etc.
For example:
<pre> <font size=-2>
./bls /tmp/File002
bls: butil.c:148 Using device: /tmp
drwxrwxr-x   3 k  k  4096 02-10-19 21:08  /home/kern/bacula/k/src/dird/
drwxrwxr-x   2 k  k  4096 02-10-10 18:59  /home/kern/bacula/k/src/dird/CVS/
-rw-rw-r--   1 k  k    54 02-07-06 18:02  /home/kern/bacula/k/src/dird/CVS/Root
-rw-rw-r--   1 k  k    16 02-07-06 18:02  /home/kern/bacula/k/src/dird/CVS/Repository
-rw-rw-r--   1 k  k  1783 02-10-10 18:59  /home/kern/bacula/k/src/dird/CVS/Entries
-rw-rw-r--   1 k  k 97506 02-10-18 21:07  /home/kern/bacula/k/src/dird/Makefile
-rw-r--r--   1 k  k  3513 02-10-18 21:02  /home/kern/bacula/k/src/dird/Makefile.in
-rw-rw-r--   1 k  k  4669 02-07-06 18:02  /home/kern/bacula/k/src/dird/README-config
-rw-r--r--   1 k  k  4391 02-09-14 16:51  /home/kern/bacula/k/src/dird/authenticate.c
-rw-r--r--   1 k  k  3609 02-07-07 16:41  /home/kern/bacula/k/src/dird/autoprune.c
-rw-rw-r--   1 k  k  4418 02-10-18 21:03  /home/kern/bacula/k/src/dird/bacula-dir.conf
...
-rw-rw-r--   1 k  k    83 02-08-31 19:19  /home/kern/bacula/k/src/dird/.cvsignore
bls: Got EOF on device /tmp
84 files found.
</pre></font>

<h3>Listing Bacula Jobs</h3>
If you are listing a Volume to determine what Jobs
to restore, normally the <b>-j</b> option provides you
with most of what you will need as long as you don't 
have multiple clients.
For example, 
<pre>
./bls -j /tmp/test1
Volume Record: SessId=2 SessTime=1033762386 JobId=0 DataLen=144
Begin Session Record: SessId=2 SessTime=1033762386 JobId=1 Level=F Type=B
End Session Record: SessId=2 SessTime=1033762386 JobId=1 Level=F Type=B
Begin Session Record: SessId=3 SessTime=1033762386 JobId=2 Level=I Type=B
End Session Record: SessId=3 SessTime=1033762386 JobId=2 Level=I Type=B
Begin Session Record: SessId=4 SessTime=1033762386 JobId=3 Level=I Type=B
End Session Record: SessId=4 SessTime=1033762386 JobId=3 Level=I Type=B
bls: Got EOF on device /tmp
</pre>
shows a full save followed by two incremental saves.
<p>
Adding the <b>-v</b> option will display virtually all information
that is available for each record:
<pre.
./bls -j -v /tmp/test1

Volume Label:
Id                : Bacula 0.9 mortal
VerNo             : 11
VolName           : test1
PrevVolName       : 
VolFile           : 0
LabelType         : VOL_LABEL
LabelSize         : 144
PoolName          : Default
MediaType         : File
PoolType          : Backup
HostName          : 
Date label written: 04-Oct-2002 23:01

Begin Session Record:
JobId             : 1
VerNum            : 11
PoolName          : Default
PoolType          : Backup
JobName           : kernsave
ClientName        : Rufus
Job (unique name) : kernsave.2002-10-04.23:01:47
FileSet           : Kerns Files
JobType           : B
JobLevel          : F
Date written      : 04-Oct-2002 23:01

End Session Record:
JobId             : 1
VerNum            : 11
PoolName          : Default
PoolType          : Backup
JobName           : kernsave
ClientName        : Rufus
Job (unique name) : kernsave.2002-10-04.23:01:47
FileSet           : Kerns Files
JobType           : B
JobLevel          : F
JobFiles          : 84
JobBytes          : 4,180,203
StartBlock        : 0
EndBlock          : 64
StartFile         : 0
EndFile           : 0
JobErrors         : 0
JobStatus         : T
Date written      : 04-Oct-2002 23:01

...

</pre>

<h3>Listing Bacula Blocks</h3>
Normally, except for debugging purposes, you will not need to list
Bacula blocks (the &quot;primitive&quot; unit of Bacula data
on the Volume). However, you can do so with:

<pre>
./bls -k /tmp/File002
bls: butil.c:148 Using device: /tmp
Block: 1 size=64512
Block: 2 size=64512
...
Block: 65 size=64512
Block: 66 size=19195
bls: Got EOF on device /tmp
End of File on device
</pre>

By adding the <b>-v</b> option, you can get more information,
which can be useful in knowing what sessions were written 
to the volume:

<pre>
./bls -k -v /tmp/File002

Volume Label:
Id                : Bacula 0.9 mortal
VerNo             : 10
VolName           : File002
PrevVolName       : 
VolFile           : 0
LabelType         : VOL_LABEL
LabelSize         : 147
PoolName          : Default
MediaType         : File
PoolType          : Backup
HostName          : 
Date label written: 2002-10-19 at 21:16
<font size=-2>
Block: 1 blen=64512 First rec FI=VOL_LABEL SessId=1 SessTim=1035062102 Strm=0 rlen=147
Block: 2 blen=64512 First rec FI=6 SessId=1 SessTim=1035062102 Strm=DATA rlen=4087
Block: 3 blen=64512 First rec FI=12 SessId=1 SessTim=1035062102 Strm=DATA rlen=5902
Block: 4 blen=64512 First rec FI=19 SessId=1 SessTim=1035062102 Strm=DATA rlen=28382
...
Block: 65 blen=64512 First rec FI=83 SessId=1 SessTim=1035062102 Strm=DATA rlen=1873
Block: 66 blen=19195 First rec FI=83 SessId=1 SessTim=1035062102 Strm=DATA rlen=2973
bls: Got EOF on device /tmp
End of File on device
</font>
</pre>
Armed with the SessionId and the SessionTime, you can extract just about anything.
<p>If you want to know even more, add a second <b>-v</b> to the command 
line to get a dump of every record in every block.
<pre>
./bls -k -v -v /tmp/File002
bls: block.c:79 Dump block  80f8ad0: size=64512 BlkNum=1
               Hdrcksum=b1bdfd6d cksum=b1bdfd6d
bls: block.c:92    Rec: VId=1 VT=1035062102 FI=VOL_LABEL Strm=0 len=147 p=80f8b40
bls: block.c:92    Rec: VId=1 VT=1035062102 FI=SOS_LABEL Strm=-7 len=122 p=80f8be7
bls: block.c:92    Rec: VId=1 VT=1035062102 FI=1 Strm=UATTR len=86 p=80f8c75
bls: block.c:92    Rec: VId=1 VT=1035062102 FI=2 Strm=UATTR len=90 p=80f8cdf
bls: block.c:92    Rec: VId=1 VT=1035062102 FI=3 Strm=UATTR len=92 p=80f8d4d
bls: block.c:92    Rec: VId=1 VT=1035062102 FI=3 Strm=DATA len=54 p=80f8dbd
bls: block.c:92    Rec: VId=1 VT=1035062102 FI=3 Strm=MD5 len=16 p=80f8e07
bls: block.c:92    Rec: VId=1 VT=1035062102 FI=4 Strm=UATTR len=98 p=80f8e2b
bls: block.c:92    Rec: VId=1 VT=1035062102 FI=4 Strm=DATA len=16 p=80f8ea1
bls: block.c:92    Rec: VId=1 VT=1035062102 FI=4 Strm=MD5 len=16 p=80f8ec5
bls: block.c:92    Rec: VId=1 VT=1035062102 FI=5 Strm=UATTR len=96 p=80f8ee9
bls: block.c:92    Rec: VId=1 VT=1035062102 FI=5 Strm=DATA len=1783 p=80f8f5d
bls: block.c:92    Rec: VId=1 VT=1035062102 FI=5 Strm=MD5 len=16 p=80f9668
bls: block.c:92    Rec: VId=1 VT=1035062102 FI=6 Strm=UATTR len=95 p=80f968c
bls: block.c:92    Rec: VId=1 VT=1035062102 FI=6 Strm=DATA len=32768 p=80f96ff
bls: block.c:92    Rec: VId=1 VT=1035062102 FI=6 Strm=DATA len=32768 p=8101713
bls: block.c:79 Dump block  80f8ad0: size=64512 BlkNum=2
               Hdrcksum=9acc1e7f cksum=9acc1e7f
bls: block.c:92    Rec: VId=1 VT=1035062102 FI=6 Strm=contDATA len=4087 p=80f8b40
bls: block.c:92    Rec: VId=1 VT=1035062102 FI=6 Strm=DATA len=31970 p=80f9b4b
bls: block.c:92    Rec: VId=1 VT=1035062102 FI=6 Strm=MD5 len=16 p=8101841
...
</pre>


 
<h1><a name="bextract">bextract</a></h1>
Normally, you will restore files by running a <b>Restore</b> Job
from the <b>Console</b> program.  However,
<b>bextract</b> can be used to extract a single file
or a list of files from a Bacula tape or file. 
In fact, <b>bextract</b> can be a useful tool to restore
files to an empty system assuming you are able to
boot, you have statically linked <b>bextract</b> and 
you have an appropriate <b>bootstrap</b> file.
<p>
It is called:
<pre> 
Usage: bextract [-d debug_level] &lt;device-name&gt; &lt;directory-to-store-files&gt;
       -b &lt;file&gt;       specify a bootstrap file
       -dnn            set debug level to nn
       -e &lt;file&gt;       exclude list
       -i &lt;file&gt;       include list
       -p              proceed inspite of I/O errors
       -V              specify Volume names (separated by |)
       -?              print this message
</pre>

where <b>device-name</b> is the Archive Device (raw device name 
or full filename) of the device to be read, and <b>directory-to-store-files</b>
is a path prefix to prepend to all the files restored. 
<p>NOTE: On Windows systems, if you specify a prefix of say d:/tmp,
any file that would have been restored to <b>c:/My Documents</b>
will be restored to <b>d:/tmp/My Documents</b>. That is, the original
drive specification will be stripped.
If no prefix is specified, the file will be restored to the
original drive.

<h3>Extracting with Include or Exclude Lists</h3>
Using the <b>-e</b> option, you can
specify a file containing a list of files to be excluded. Wildcards
can be used in the exclusion list. This option will normally be
used in conjunction with the <b>-i</b> option (see below). Both the
<b>-e</b> and the <b>-i</b> options may be specified at the same
time as the <b>-b</b> option. The bootstrap filters will be applied
first, then the include list, then the exclude list.
<p>
Likewise, and probably more importantly, with the <b>-i</b> option, you
can specify a file that contains a list (one file per line) of files and
directories to include to be restored.  The list must contain the full
filename with the path.  If you specify a path name only, 
all files and subdirectories of that path
will be restored.  If you specify a line containing only the filename
(e.g.  <b>my-file.txt</b>) it probably will not be extracted because you
have not specified the full path.
<p>For example, if the file <b>include-list</b> contains:
<pre>
/home/kern/bacula 
/usr/local/bin
</pre>
<p>
Then the command:
<pre>
./bextract -i include-list -V Volume /dev/nst0 /tmp
</pre>
will restore from the Bacula archive <b>/dev/nst0</b> all files and
directories in the backup from <b>/home/kern/bacula</b> and
from <b>/usr/local/bin</b>. The restored
files will be placed in a file of the original name under the
directory <b>/tmp</b> (i.e. /tmp/home/kern/bacula/... and
/tmp/usr/local/bin/...).

<h3>Extracting With a Bootstrap File</h3>
The <b>-b</b> option is used to specify a <b>bootstrap</b> file 
containing the information needed to restore precisely the files
you want. Specifying a <b>bootstrap</b> file is optional but
recommended because it gives you the most control over which files
will be restored. For more details on the <b>bootstrap</b> file,
please see <a href="bootstrap.html">Restoring Files with the
Bootstrap File</a> chapter of this document. Note, you may also
use a bootstrap file produced by the <b>restore</b> command.  
For example:
<pre>
./bextract -b bootstrap-file /dev/nst0 /tmp
</pre>
The bootstrap file allows detailed specification of what files
you want restored (extracted).  You may specify a bootstrap file
and include and/or exclude files at the same time. The bootstrap
conditions will first be applied, and then each file record seen
will be compared to the include and exclude lists.

<h3>Extracting From Multiple Volumes</h3>
If you wish to extract files that span several Volumes, you
can specify the Volume names in the bootstrap file or you
may specify the Volume names
on the command line by separating them with a vertical 
bar.  See the section above
under the <b>bls</b> program entitled <b>Listing Multiple
Volumes</b> for more information. The same techniques
apply equally well to the <b>bextract</b> program.


<h1><a name="bscan">bscan</a></h1>
The <b>bscan</b> program can be used to re-create a
database (catalog) from the backup information written
to one or more Volumes.  This is normally needed only
if one or more Volumes have been pruned or purged from
your catalog so that the records on the Volume are
no longer in the catalog.
<p>
With some care,
it can also be used to synchronize your existing catalog with
a Volume. Since <b>bscan</b> modifies your catalog, we
strongly recommend that you do a simple ASCII backup of
your database before running <b>bscan</b> just to be
sure. See <a href="catmaintenance.html#CompactingMySQL">Compacting
Your Database</a>.
<p>
<b>bscan</b> can also be useful in a disaster recovery
situation, after the loss of a hard disk, if you do not
have a valid <b>bootstrap</b> file for reloading
your system, or if a Volume has been recycled but not
overwritten, you can use <b>bscan</b>
to re-create your database, which can then be used
to <b>restore</b> your system or a file to its previous state.

<p>
It is called:
<pre> 
Usage: bscan [options] &lt;bacula-archive&gt;
       -b bootstrap   specify a bootstrap file
       -c &lt;file&gt;      specify configuration file
       -d &lt;nn&gt;        set debug level to nn
       -m             update media info in database
       -n &lt;name&gt;      specify the database name (default bacula)
       -u &lt;user&gt;      specify database user name (default bacula)
       -P &lt;password&gt;  specify database password (default none)
       -h &lt;host&gt;      specify database host (default NULL)
       -p             proceed inspite of I/O errors
       -r             list records
       -s             synchronize or store in database
       -v             verbose
       -V &lt;Volumes&gt;   specify Volume names (separated by |)
       -w &lt;dir&gt;       specify working directory (default from conf file)
       -?             print this message
</pre>

<p>If you are using MySQL or PostgreSQL, there is no need to
supply a working directory since in that case, bscan knows
where the databases are.  However, if you have provided security on
your database, you may need to supply either the database name
(<b>-b</b> option), the user name (<b>-u</b> option), and/or the
password (<b>-p</b>) options.
<p>
As an example, let's suppose that you did a backup to Volume  
&quot;Vol001&quot; and that sometime later all record of that
Volume was pruned or purged from the database. By using <b>bscan</b> 
you can recreate the catalog entries for that Volume and then use
the <b>restore</b> command in the Console to restore whatever you want.
A command something like:
<pre>
bscan -c bacula-sd.conf -v -V Vol001 /dev/nst0
</pre>
will give you a give you an idea of what is going to happen without
changing your catalog. Of course, you may need to change the path
to the Storage daemon's conf file, the Volume name, and your tape 
(or disk) device name.  This command must read the entire tape,
so if it has a lot of data, it may take a long time, and thus
you might want to immediately use the command listed below.
Note, if you are writing to a disk file,
replace the device name with the path to the directory that contains
the Volume. This must correspond to the Archive Device in the
conf file. 
<p>
Then to actually write or store the records in the catalog,
add the <b>-s</b> option as follows:
<pre>
 bscan -s -m -c bacula-sd.conf -v -V Vol001 /dev/nst0
</pre>
When writing to the database, if bscan finds existing records,
it will generally either update them if something is wrong or 
leave them alone. Thus if the Volume you are scanning is all or
partially in the catalog already, no harm will be done to that
existing data. Any missing data will simply be added.
<p>
If you have multiple tapes, you can scan them with:
<pre>
 bscan -s -m -c bacula-sd.conf -v -V Vol001\|Vol002\|Vol003 /dev/nst0
</pre>
You should, where ever possible try to specify the tapes in the
order they are written. However, bscan can handle scanning tapes
that are not sequential. Any incomplete records at the end of the 
tape will simply be ignored in that case.
<p>
Note, the restoration process using bscan is not identical to the
original creation of the catalog data. This is because certain
non-essential data such as volume reads, volume mounts, etc is not
stored on the Volume, and thus is not restored by bscan. The results
of bscanning are, however, perfectly valid, and will permit restoration
of any or all the files in the catalog using the normal Bacula
console commands.

<h3>Using bscan to Compare a Volume to an existing Catalog</h3>
If you wish to compare the contents of a Volume to an existing
catalog without changing the catalog, you can safely do so if and
only if you do <b>not</b> specify either the <b>-m</b> or the 
<b>-s</b> options.  However, at this time (Bacula version 1.26),
the comparison routines are not as good or as thorough as they
should be, so we don't particularly recommend this mode other than
for testing.

<h3>Using bscan to Recreate a Catalog from a Volume</h3>
This is the mode for which <b>bscan</b> is most useful.
You can either <b>bscan</b> into a freshly created
catalog, or directly into your existing catalog (after
having made an ASCII copy as described above).
Normally, you should start with a freshly created
catalog that contains no data.  
<p>Starting with a single Volume named
<b>TestVolume1</b>, you run a command such as:
<pre>
./bscan -V TestVolume1 -v -s -m -c bacula-sd.conf /dev/nst0
</pre>
If there is more than one volume, simply append it to the
first one separating it with a vertical bar. You may need to
precede the vertical bar with a forward slash escape the
shell -- e.g. <b>TestVolume1\|TestVolume2</b>.
The <b>-v</b> option was added for verbose output (this can
be omitted if desired). The <b>-s</b> option that tells <b>bscan</b> to
store information in the database. The physical
device name <b>/dev/nst0</b> is specified after all the options.
<p>
For example, after having done a full backup of a directory, then two
incrementals, I reinitialized the SQLite database as described above, and
using the bootstrap.bsr file noted above, I entered the following command:
<pre>
./bscan -b bootstrap.bsr -v -s -c bacula-sd.conf /dev/nst0
</pre>
which produced the following output:
<pre>
bscan: bscan.c:182 Using Database: bacula, User: bacula
bscan: bscan.c:673 Created Pool record for Pool: Default
bscan: bscan.c:271 Pool type "Backup" is OK.
bscan: bscan.c:632 Created Media record for Volume: TestVolume1
bscan: bscan.c:298 Media type "DDS-4" is OK.
bscan: bscan.c:307 VOL_LABEL: OK for Volume: TestVolume1
bscan: bscan.c:693 Created Client record for Client: Rufus
bscan: bscan.c:769 Created new JobId=1 record for original JobId=2
bscan: bscan.c:717 Created FileSet record "Kerns Files"
bscan: bscan.c:819 Updated Job termination record for new JobId=1
bscan: bscan.c:905 Created JobMedia record JobId 1, MediaId 1
bscan: Got EOF on device /dev/nst0
bscan: bscan.c:693 Created Client record for Client: Rufus
bscan: bscan.c:769 Created new JobId=2 record for original JobId=3
bscan: bscan.c:708 Fileset "Kerns Files" already exists.
bscan: bscan.c:819 Updated Job termination record for new JobId=2
bscan: bscan.c:905 Created JobMedia record JobId 2, MediaId 1
bscan: Got EOF on device /dev/nst0
bscan: bscan.c:693 Created Client record for Client: Rufus
bscan: bscan.c:769 Created new JobId=3 record for original JobId=4
bscan: bscan.c:708 Fileset "Kerns Files" already exists.
bscan: bscan.c:819 Updated Job termination record for new JobId=3
bscan: bscan.c:905 Created JobMedia record JobId 3, MediaId 1
bscan: Got EOF on device /dev/nst0
bscan: bscan.c:652 Updated Media record at end of Volume: TestVolume1
bscan: bscan.c:428 End of Volume. VolFiles=3 VolBlocks=57 VolBytes=10,027,437
</pre>
The key points to note are that <b>bscan</b> prints a line when
each major record is created. Due to the volume of output, it does not print
a line for each file record unless you supply the <b>-v</b> option
twice or more on the command line.
<p>
In the case of a Job record, the new JobId
will not normally be the same as the original Jobid.
For example, for the first JobId above, the new JobId is 1, but the original
JobId is 2. This is nothing to be concerned about as it is the normal
nature of databases. <b>bscan</b> will keep everything straight.
<p>Although <b>bscan</b> claims that it created a Client record for
Client: Rufus three times, it was actually only created the first
time. This is normal.
<p>
You will also notice that it read an end of file after each Job
(Got EOF on device ...). Finally the last line gives the total
statistics for the bscan.
<p>
If you had added a second <b>-v</b> option to the command line,
Bacula would have been even more verbose, dumping virtually all the
details of each Job record it encountered.
<p>
Now if you start Bacula and enter a <b>list jobs</b> command
to the console program, you will get:
<pre>
<font size=-1>
+-------+----------+------------------+------+-----+----------+----------+---------+
| JobId | Name     | StartTime        | Type | Lvl | JobFiles | JobBytes | JobStat |
+-------+----------+------------------+------+-----+----------+----------+---------+
| 1     | kernsave | 2002-10-07 14:59 | B    | F   | 84       | 4180207  | T       |
| 2     | kernsave | 2002-10-07 15:00 | B    | I   | 15       | 2170314  | T       |
| 3     | kernsave | 2002-10-07 15:01 | B    | I   | 33       | 3662184  | T       |
+-------+----------+------------------+------+-----+----------+----------+---------+
</font>
</pre>
which corresponds virtually identically with what the database contained before
it was re-initialized and restored with bscan. All the Jobs and Files found on
the tape are restored including most of the Media record. The Volume (Media)
records restored will be marked as <b>Full</b> so that they cannot be rewritten
without operator intervention.
<p>
It should be noted that <b>bscan</b> cannot restore a database to the exact
condition it was in previously because a lot of the less important information
contained in the database is not saved to the tape. Nevertheless, the reconstruction
is sufficiently complete, that you can run <b>restore</b> against it and
get valid results.

<h3>Using bscan to Correct the Volume File Count</h3>
If the Storage daemon crashes during a backup Job, the catalog will no
be properly updated for the Volume being used at the time of the crash.
This means that the Storage daemon will have written say 20 files on the
tape, but the catalog record for the Volume indicates only 19 files.
<p>
Bacula refuses to write on a tape that
contains a different number of files from what is in the catalog.
To correct this situation, you may run a <b>bscan</b> with the
<b>-m</b> option (but <b>without</b> the <b>-s</b> option) to
update only the final Media record for the Volumes read. 

<h2>After bscan</h2>
If you use <b>bscan</b> to enter the contents of the Volume into
an existing catalog, you should be aware that the records you
entered may be immediately pruned during the next job particularly if the
Volume is very old or had been previously purged. To avoid this,
after running <b>bscan</b>, you can manually set the volume
status (VolStatus) to <b>Read-Only</b> by using the <b>update</b>
command in the catalog. This will allow you to restore from the
volume without having it immediately purged. When you have
restored and backed up the data, you can reset the VolStatus
to <b>Used</b> and the Volume will be purged from the catalog.


<h1><a name="bcopy">bcopy</a></h1>
The <b>bcopy</b> program can be used to copy one
<b>Bacula</b> archive file to another. For example,
you may copy a tape to a file, a file to a tape, a
file to a file, or a tape to a tape.  For tape to tape,
you will need two tape drives. (a later version is   
planned that will buffer it to disk). In the process
of making the copy, no record of the information written
to the new Volume is stored in the catalog. This means
that the new Volume, though it contains valid backup data,
cannot be accessed directly from existing catalog entries.
If you wish to be able to use the Volume with the Console
restore command, for example, you must first bscan the
new Volume into the catalog.

<h2>bcopy Command Options</h2>
<pre>
Usage: bcopy [-d debug_level] &lt;input-archive&gt; &lt;output-archive&gt;
       -b bootstrap      specify a bootstrap file
       -c &lt;file&gt;         specify configuration file
       -dnn              set debug level to nn
       -i                specify input Volume names (separated by |)
       -o                specify output Volume names (separated by |)
       -p                proceed inspite of I/O errors
       -v                verbose
       -w dir            specify working directory (default /tmp)
       -?                print this message

</pre>
 
By using a <b>bootstrap</b> file, you can copy parts
of a Bacula archive file to another archive.
<p>One of the objectives of this program is to be
able to recover as much data as possible from a 
damaged tape. However, the current version does not
yet have this feature.
<p>As this is a new program, any feedback on its use
would be appreciated.  In addition, I only have a single
tape drive, so I have never been able to test this program
with two tape drives.


<h1><a name="btape">btape</a></h1>
This program permits a number of elementary tape operations
via a tty command interface. 
The <b>test</b> command,
described below, can be very useful for testing older tape 
drive compatibility problems.
Aside from initial testing of tape drive compatibility
with <b>Bacula</b>, <b>btape</b> will be mostly used by
developers writing new tape drivers. 
<p>
<b>btape</b> can be dangerous to use with existing
<b>Bacula</b> tapes because it will relabel a tape or  
write on the tape if so requested regardless that   
the tape may contain valuable data, so please be
careful and use it only on blank tapes.


<p>To work properly, <b>btape</b> needs to read the Storage
daemon's configuration file. As a default, it will look for
<b>bacula-sd.conf</b> in the current directory. If your
configuration file is elsewhere, please use the <b>-c</b>
option to specify where.
<p>The physical device name must be specified
on the command line, and that this same device name must be present
in the Storage daemon's configuration file read by <b>btape</b>
<pre>
Usage: btape [-c config_file] [-d debug_level] [device_name]
       -c &lt;file&gt;   set configuration file to file
       -dnn        set debug level to nn
       -s          turn off signals
       -t          open the default tape device
       -?          print this message.
</pre>


<h3>Using btape to Verify your Tape Drive</h3>                 
<p>An important reason for this program is
to ensure that a Storage daemon configuration
file is defined so that Bacula will correctly read and write
tapes. 
<p>
It is highly recommended that you run the <b>test</b> command before
running your first Bacula job to ensure that the parameters you
have defined for your storage device (tape drive) will permit
<b>Bacula</b> to function properly. You only need to mount
a blank tape, enter the command, and the output should be
reasonably self explanatory. Please see the
<a href="tapetesting.html">Tape Testing</a>
Chapter of this manual for the details.

<h3>btape Commands</h3>
  
The full list of commands are:
<pre>
  Command    Description
  =======    ===========
  bsf        backspace file
  bsr        backspace record
  cap        list device capabilities
  clear      clear tape errors
  eod        go to end of Bacula data for append
  test       General test Bacula tape functions
  eom        go to the physical end of medium
  fill       fill tape, write onto second volume
  unfill     read filled tape
  fsf        forward space a file
  fsr        forward space a record
  help       print this command
  label      write a Bacula label to the tape
  load       load a tape
  quit       quit btape
  rd         read tape
  readlabel  read and print the Bacula tape label
  rectest    test record handling functions
  rewind     rewind the tape
  scan       read tape block by block to EOT and report
  status     print tape status
  test       test a tape for compatibility with Bacula
  weof       write an EOF on the tape
  wr         write a single record of 2048 bytes
</pre>
The most useful commands are:
<ul>
  <li>test -- test writing records and EOF marks and
      reading them back.
  <li>fill -- completely fill a volume with records, then
      write a few records on a second volume, and finally,
      both volumes will be read back. Please be aware that
      the data written will be quite similar every record, so
      you might want to turn compression off. One user found
      that the fill command wrote 750Gb to a tape that can
      hold 35Gb -- so you can see that the hardware compression
      really worked well! </li>
  <li>readlabel -- read and dump the label on a Bacula tape.</li>
  <li>cap -- list the device capabilities as defined in the
      configuration file and as perceived by the Storage daemon.</li>
</ul>


The <b>readlabel</b> command can be used to display the
details of a Bacula tape label. This can be useful
if the physical tape label was lost or damaged.

<p>In the event that you want to relabel a <b>Bacula</b>,
you can simply use the <b>label</b> command which will
write over any existing label.  However, please note
for labeling tapes, we recommend that you use the
<b>label</b> command in the <b>Console</b> program since
it will never overwrite a valid Bacula tape.


<h1>Other Programs</h1>
The following programs are general utility programs and
in general do not need a configuration file nor a device name.

<h1><a name="bsmtp">bsmtp</a></h1>
<b>bsmtp</b> is a simple mail transport program that permits
more flexibility than the standard mail programs typically
found on Unix systems. It can even be used on Windows machines.
<p>
It is called:
<pre>
Usage: bsmtp [-f from] [-h mailhost] [-s subject] [-c copy] [recipient ...]
       -c          set the Cc: field
       -dnn        set debug level to nn
       -f          set the From: field
       -h          use mailhost:port as the bsmtp server
       -s          set the Subject: field
       -?          print this message.
</pre>
If the <b>-f</b> option is not specified, <b>bsmtp</b> will use your 
userid. If the <-h> option is not specified <b>bsmtp</b> will
use the value in the environment variable <b>bsmtpSERVER</b> or if
there is none <b>localhost</b>. By default port 25 is used.
<p><b>recipients</b> is a space separated list of email recipients.
<p>The body of the email message is read from standard input.
<p>An example of the use of <b>bsmtp</b> would be to put the following
statement in the <b>Messages</b> resource of your <b>bacula-dir.conf</b>
file. Note, these commands should appear on a single line each.
<pre>
  mailcommand = "/home/bacula/bin/bsmtp -h mail.domain.com -f \"\(Bacula\) %r\"
                 -s \"Bacula: %t %e of %c %l\" %r"
  operatorcommand = "/home/bacula/bin/bsmtp -h mail.domain.com -f \"\(Bacula\) %r\"
                    -s \"Bacula: Intervention needed for %j\" %r"
</pre>
Where you replace <b>/home/bacula/bin</b> with the path to your
<b>Bacula</b> binary directory, and you replace <b>mail.domain.com</b>
with the fully qualified name of your bsmtp (email) server, which
normally listens on port 25.  For more details on the substitution
characters (e.g.  %r) used in the above line, please see the
documentation of the <a href="messagesres.html#mailcommand"> MailCommand
in the Messages Resource</a> chapter of this manual.
<p>
It is HIGHLY recommended that you test one or two cases by hand to
make sure that the <b>mailhost</b> that you specified is correct and
that it will accept your email requests.  Since <b>bsmtp</b> always uses
a TCP connection rather than writing in the spool file, you may find
that your <b>from</b> address is being rejected because it does not
contain a valid domain, or because your message is caught in your spam
filtering rules.  Generally, you should specify a fully qualified domain
name in the <b>from</b> field, and depending on whether your bsmtp gateway
is Exim or Sendmail, you may need to modify the syntax of the from part
of the message. Please test.
<p>
When running <b>bsmtp</b> by hand, you will need to terminate the
message by entering a ctl-d in column 1 of the last line.
<p>

<h1><a name="dbcheck">dbcheck</a></h1>
<b>dbcheck</b> is a simple program that will search for inconsistencies
in your database, and optionally fix them.  The <b>dbcheck</b> program can
be found in the <b>&lt;bacula-source&gt;/src/tools</b> directory of the
source distribution. Though it is built with the make process, 
it is not normally &quot;installed&quot;.
<p>
It is called:
<pre>
Usage: dbcheck [-c config] [-C catalog name] [-d debug_level] <working-directory> <bacula-database> <user> <password> [<dbhost>]
       -b              batch mode
       -C              catalog name in the director conf file
       -c              director conf filename
       -dnn            set debug level to nn
       -f              fix inconsistencies
       -v              verbose
       -?              print this message
</pre>
If the <b>-c</b> option is given with the Director's conf file, there is
no need to enter any of the command line arguments, in particular the
working directory as dbcheck will read them from the file.
<p>
If the <b>-f</b> option is specified, <b>dbcheck</b> will repair (<b>fix</b>)
the inconsistencies it finds. Otherwise, it will report only.
<p>
If the <b>-b</b> option is specified, <b>dbcheck</b> will run in batch
mode, and it will proceed to examine and fix (if -f is set) all programmed
inconsistency checks.  If the <b>-b</b> option is not specified, <b>dbcheck</b>
will enter interactive mode and prompt with the following:
<pre>
Hello, this is the database check/correct program.
Please select the function you want to perform.

     1) Toggle modify database flag
     2) Toggle verbose flag
     3) Repair bad Filename records
     4) Repair bad Path records
     5) Eliminate duplicate Filename records
     6) Eliminate duplicate Path records
     7) Eliminate orphaned Jobmedia records
     8) Eliminate orphaned File records
     9) Eliminate orphaned Path records
    10) Eliminate orphaned Filename records
    11) Eliminate orphaned FileSet records
    12) Eliminate orphaned Client records
    13) Eliminate orphaned Job records
    14) Eliminate all Admin records
    15) Eliminate all Restore records
    16) All (3-15)
    17) Quit
Select function number: 
</pre>
By entering 1 or 2, you can toggle the modify database flag (-f option) and
the verbose flag (-v). It can be helpful and reassuring to turn off the
modify database flag, then select one or more of the consistency checks
(items 3 through 9) to see what will be done, then toggle the modify
flag on and re-run the check.
       
<p>
The inconsistencies examined are the following:
<ul type="disc">
  <li>Duplicate filename records. This can happen if you accidentally run
      two copies of Bacula at the same time, and they are both adding
      filenames simultaneously. It is a rare occurrence, but will create
      an inconsistent database. If this is the case, you will receive
      error messages during Jobs warning of duplicate database records.
      If you are not getting these error messages, there is no reason
      to run this check. </li>
  <li>Repair bad Filename records. This checkes and corrects filenames
      that have a trailing slash. They should not.</li>
  <li>Repair bad Path records. This checks and corrects path names
      that do not have a trailing slash. They should.</li>
  <li>Duplicate path records. This can happen if you accidentally run
      two copies of Bacula at the same time, and they are both adding
      filenames simultaneously. It is a rare occurrence, but will create
      an inconsistent database. See the item above for why this occurs and
      how you know it is happening. </li>
  <li>Orphaned JobMedia records. This happens when a Job record is deleted
      (perhaps by a user issued SQL statement), but the corresponding
      JobMedia record (one for each Volume used in the Job) was not deleted.     
      Normally, this should not happen, and even if it does, these records
      generally do not take much space in your database. However, by running
      this check, you can eliminate any such orphans.</li>
  <li>Orphaned File records. This happens when a Job record is deleted
      (perhaps by a user issued SQL statement), but the corresponding
      File record (one for each Volume used in the Job) was not deleted.     
      Note, searching for these records can be <b>very</b> time consuming (i.e.
      it may take hours) for a large database. Normally this should not
      happen as Bacula takes care to prevent it. Just the same, this
      check can remove any orphaned File records. It is recommended that
      you run this once a year since orphaned File records can take a
      large amount of space in your database. </li>
  <li>Orphaned Path records. This condition happens any time a directory is
      deleted from your system and all associated Job records have been purged.
      During standard purging (or pruning) of Job records, Bacula does
      not check for orphaned Path records. As a consequence, over a period
      of time, old unused Path records will tend to accumulate and use
      space in your database. This check will eliminate them. It is strongly
      recommended that you run this check at least once a year. </li>    

  <li>Orphaned Filename records. This condition happens any time a file is
      deleted from your system and all associated Job records have been purged.
      This can happen quite frequently as there are quite a large number
      of files that are created and then deleted. In addition, if you
      do a system update or delete an entire directory, there can be
      a very large number of Filename records that remain in the catalog
      but are no longer used.
      <p>
      During standard purging (or pruning) of Job records, Bacula does
      not check for orphaned Filename records. As a consequence, over a period
      of time, old unused Filename records will accumulate and use
      space in your database. This check will eliminate them. It is strongly
      recommended that you run this check at least once a year, and for
      large database (more than 200 Megabytes), it is probably better to
      run this once every 6 months.</li>
  <li>Orphaned Client records. These records can remain in the database
      long after you have removed a client. </li>
  <li>Orphaned Job records. If no client is defined for a job or you
      do not run a job for a long time, you can accumulate old job
      records. This option allow you to remove jobs that are not
      attached to any client (and thus useless).</li>
  <li>All Admin records. This command will remove all Admin records,
      regardless of their age.</li>
  <li>All Restore records. This command will remove all Restore records,
      regardless of their age.</li>
</ul>

<h1><a name="testfind">testfind</a></h1>
<b>testfind</b> permits listing of files using the same search engine that
is used for the <b>Include</b> resource in Job resources. Note, much
of the functionality of this program (listing of files to be included)
is present in the <a href="console.html#estimate">estimate command</a>
in the Console program.
<p>
The original use of testfind
was to ensure that Bacula's file search engine was correct and to print
some statistics on file name and path length. However, you may find
it useful to see what bacula would do with a given <b>Include</b> resource.
The <b>testfind</b> program can
be found in the <b>&lt;bacula-source&gt;/src/tools</b> directory of the
source distribution. Though it is built with the make process, 
it is not normally &quot;installed&quot;.
<p>
It is called:
<pre>
Usage: testfind [-d debug_level] [-] [pattern1 ...]
       -a          print extended attributes (Win32 debug)
       -dnn        set debug level to nn
       -           read pattern(s) from stdin
       -?          print this message.

Patterns are used for file inclusion -- normally directories.
Debug level >= 1 prints each file found.
Debug level >= 10 prints path/file for catalog.
Errors are always printed.
Files/paths truncated is a number with len > 255.
Truncation is only in the catalog.
</pre>
Where a pattern is any filename specification that is valid within
an <b>Include</b> resource definition. If none is specified, <b>/</b>
 (the root directory) is assumed. For example:
<pre>
./testfind /bin
</pre>
Would print the following:
<pre>
Dir: /bin
Reg: /bin/bash
Lnk: /bin/bash2 -> bash
Lnk: /bin/sh -> bash
Reg: /bin/cpio
Reg: /bin/ed
Lnk: /bin/red -> ed
Reg: /bin/chgrp
...
Reg: /bin/ipcalc
Reg: /bin/usleep
Reg: /bin/aumix-minimal
Reg: /bin/mt
Lnka: /bin/gawk-3.1.0 -> /bin/gawk
Reg: /bin/pgawk
Total files    : 85
Max file length: 13
Max path length: 5
Files truncated: 0
Paths truncated: 0
</pre>
Even though <b>testfind</b> uses the same search engine as <b>Bacula</b>,
each directory to be listed, must be entered as a separate command
line entry or entered one line at a time to standard input if the <b>-</b>
option was specified.
<p>
Specifying a debug level of one (i.e.  <b>-d1</b>) on the command line
will cause <b>testfind</b> to print the raw filenames without showing
the Bacula internal file type, or the link (if any).  Debug
levels of 10 or greater cause the filename and the path to be separated
using the same algorithm that is used when putting filenames into the
Catalog database.

<h1><a name="bimagemgr">bimagemgr</a></h1>

<b>bimagemgr</b> is a utility for those who backup to disk volumes in order to commit 
them to CDR disk, rather than tapes. It is a web based interface written in 
perl, used to monitor when a volume file needs to be burned to disk. It requires:

<ul>
<li>A web server running on the bacula server</li>
<li>A CD recorder installed and configured on the bacula server</li>
<li>The cdrtools package installed on the bacula server.</li>
<li>perl, perl-DBI module, and either DBD-MySQL or DBD-PostgreSQL modules</li>
</ul>

SQLite databases and DVD burning are not supported by <b>bimagemgr</b> at this time, 
but both planned for future releases.

<h3>Installation</h3>
Please see the README file in the bimagemgr directory of the 
distribution for instructions.

<h3>Usage</h3>
Calling the program in your web browser, e.g. <code>http://localhost/cgi-bin/bimagemgr.pl</code> 
will produce a display as shown below in Figure 1. The program will query the bacula 
database and display all volume files with the date last written and the date last 
burned to disk. If a volume needs to be burned (last written is newer than last burn 
date) a "Burn" button will be displayed in the right most column.

<p><div align="center"><img src="bimagemgr1.png" ></div></p>
<div align="center">Figure 1</div>

<p>Place a blank CDR disk in your recorder and click a "Burn" button. This will cause 
a pop up window as shown in Figure 2 to display the burn progress.</p>

<p><div align="center"><img src="bimagemgr2.png"></div></p>
<div align="center">Figure 2</div>

<p>When the burn finishes the pop up window will display the results of cdrecord as 
shown in Figure 3. Close the pop up window and refresh the main window. The last burn 
date will be updated and the "Burn" button for that volume will disappear. Should you 
have a failed burn you can reset the last burn date of that volume by clicking it's 
"Reset" link.</p>

<p><div align="center"><img src="bimagemgr3.png"></div></p>
<div align="center">Figure 3</div>

<p>In the bottom row of the main display window are two more buttons labeled 
"Burn Catalog" and "Blank CDRW". "Burn Catalog" will place a copy of your bacula 
catalog on a disk. If you use CDRW disks rather than CDR then "Blank CDRW" allows 
you to erase the disk before re-burning it. Regularly committing your backup volume 
files and your catalog to disk with <b>bimagemgr</b> insures that you can rebuild easily 
in the event of some disaster on the bacula server itself.</p>

<hr>

<page_prev name="Tips and Suggestions" url="tips.html">
<page_next name="Tape Testing" url="tapetesting.html">
