


                DETAIL NOTES ON VARIOUS TOPICS



 CONTENTS
----------------
   I - EA & differential backup
  II - dar and remote backup server
 III - bytes, bits, kilo, mega, etc.
  IV - archive structure in brief
   V - EA Support & Compilation Problems
  VI - run DAR in background
 VII - run command or scripts from DAR
VIII - Makefile targets
  IX - Scrambling
   X - dar_manager
  XI - file extension used
 XII - dar and ssh (see also chapter II)
XIII - overflow in arithmetic integer operations
 XIV - using data protection with DAR & PAR
  XV - Dar User Command (DUC files)
 XVI - Examples of file filtering

I- EA & differential backup
============================

brief presentation of EA:
--------------------------
EA stands for Extended Attributes. In Unix filesystem a regular file is
composed of a set of byte (the data) and an inode. The inode add properties to
the file, such as owner, group, permission, dates (last modification date of
the data [mtime], last access date to data [atime], and last inode change date
[ctime]), etc). Last, the name of the file is not contained in the inode, but
in the directory(ies) it is linked to. When a file is linked more than once
in the directory tree, we speak about "hard links". This way the same data and
associated inode appears several time in the same or different directories.
This is not the same as a symbolic links, which is a file that contains the
path to another file (which may or may not exist). A symbolic link has its own
inode. OK, now let's talk about EA:

Extended attributes is a recent feature of Unix file system. They extend
attributes provided by the inode and associated to a data. They are not part of
the inode, nor part of the data, nor part of a given directory. They are
stored beside the inode and are a set of pair of key and value. The owner
of the file can add or define any key and eventually associate data to it. It
can also
list and remove a particular key. What they are used for ? A way to associate
information to a file.

One particular interesting use of EA, is ACL: Access Control List. ACL can be
implemented using EA and add a more fine grain in assigning access permission
to file. For more information one EA and ACL, see the site of
Andreas Grunbacher:

        http://acl.bestbits.at/

EA & Differential Backup
--------------------------
to determine that an EA has changed dar looks at the ctime value.
if ctime has changed, (due to EA change, but also to permission or owner
change) dar saves the EA. ctime also changes, if atime or mtime changes. So if
you access a file or modify it, dar will consider that the EA have changed
also. This is not really fair, I admit.

Something better would be to compare EA one by one, and record those that have
changed or have been deleted. But to be able to compare all EA and their value
reference EA must reside in memory. As EA can grow up to 64 KB by file, this
can lead to a quick saturation of the virtual memory, which is already enough
solicited by the catalogue

Theses two schemes implies a different pattern for saving EA in archive. In the
first case (no EA in memory except at time of operation on it), to avoid
skipping in the archive (and ask the user to change of disks too often), EA
must be stored beside the data of the file (if present). Thus they must be
distributed all along the archive (except at the end that only contains the
catalogue).

In the second case (EA are loaded in memory for comparison), EA must reside
beside or within the catalogue, in any case at the end of the archive, not
to have to user to need all disks to just take an archive as reference.

As the catalogue, grows already fast with the number of file to save (from a
few bytes for hard_link to 400 bytes around per directory inode), the memory
saving option has been adopted.

Thus, EA changes are based on the ctime change. Unfortunately, no system call
permits to restore ctime. Thus, restoring an differential backup after its
reference has been restored, will present restored inode as more recent than
those in the differential archive, thus the -r option would prevent any EA
restoration. In consequence, -r has been disabled for EA, it does only concern
data contents. If you don't want to restore any EA but just more recent data,
you can use the following : -r -u -U


II - dar and remote backup server
=================================

The situation is the following:
you have a host (called local in the following), on which resides an
operational system, which you want to backup regularly, without perturbing
users. For security reasons you want to store the backup on another host
(called remote host in the following), only used for backup. Of course you have
not much space on local host to store the archive.

Between theses two hosts, you could use NFS and nothing special would be
necessary to add, to use dar as usually. but if for security reasons you don't
want to use NFS (insecure network, local user must not have access to backups),
but prefer to communicate through an encrypted session, (using ssh for example)
then you need to use dar features brought by version 1.1.0:

- dar can now output its archive to stdout instead of a given file. To activate
it, use "-" as basename. Here is an example :

        dar -c - -R / -z | some_program
or
        dar -c - -R / -z > named_pipe_or_file

Note, that file splitting is not available as it has not much meaning when
writing to a pipe. (a pipe has no name, there is no way to skip (or seek) in a
pipe, and what would mean changing of file ? changing of pipe ?).

At the other end of the pipe (on the remote host), the data can be redirected
to a file, with proper filename (something that matches "*.1.dar").

        some_other_program > backup_name.1.dar

It is also possible to redirect the output to dar_xform which can in turn on
the remote host split the data flow in several files, pausing between them,
exactly as dar is able to do:

        some_other_program | dar_xform -s 100M - backup_name

this will create backup_name.1.dar and so on. The resulting archive is totally
compatible with those directly generated by dar.

OK, you are happy, you can backup the local filesystem to a remote server
through a secure socket session, in a full featured dar archive without
using NFS. But, now you want to make a differential backup taking this archive
as reference. How to do that ?

The simplest way is to use the new feature called "isolation", which extracts
the catalogue from the archive and stores it in a little file. On the remote
backup server you would type:

        dar -A backup_name -C CAT_backup_name -z

if the catalogue is too big to fit on a floppy, you can slit it as usually
using dar:

        dar -A backup_name -C CAT_backup_name -z -s 1440k

the generated archive (CAT_backup_name.1.dar, and so on), only contains the
catalogue, but can still be used as reference for a new backup. You just need
to transfer it back to the local host, either using floppies, or through a
secured socket session, or even directly isolating the catalogue to a pipe
that goes from the remote host to the local host:

on remote host:
        dar -A backup_name -C - -z | some_program

on local host:
        some_other_program > CAT_backup_name.1.dar

or use dar_xform as previously if you need splitting :

        some_other_program | dar_xform -s 1440k CAT_backup_name

then you can make your differential backup as usual:

        dar -A CAT_backup_name -c - -z -R / | some_program

or if this time you prefer to save the archive locally:

        dar -A CAT_backup_name -c backup_diff -z -R /

For differential backups instead of isolating the catalogue, it is also
possible to read an archive or its extracted catalogue through pipes. Yes, two
pipes are required for dar to be able to read an archive.
The first goes from dar to the external program "dar_slave" and carries orders
(asking some portions of the archive), and the other pipe, goes from
"dar_slave" back to "dar" and carries the asked data for reading.

By default, if you specify "-" as basename for -l, -t, -d, -x, or to -A (used
with -C or -c), dar and dar_slave will use their standard input and output to
communicate. Thus you need additional program to make the input of the first
going to the output to the second, and vice versa.
Warning: you cannot use named pipe that way, because dar and dar_slave would
get blocked upon opening of the first named pipe, waiting for the peer to
open it also, even before they has started (dead lock at shell level).
For named pipes, there is -i and -o options that helps, they receive a
filename as argument, which may be a named pipe. The -i argument is used
instead of stdin and -o instead of stdout. Note that for dar -i and -o are only
available if "-" is used as basename. Let's take an example:

You now want to restore an archive from your remote backup server. Thus on it
you have to run dar_slave this way

on remote server:
        some_prog | dar_slave backup_name | some_other_prog
or
        dar_slave -o /tmp/pipe_todar -i /tmp/pipe_toslave backup_name

and on the local host you have to run dar this way

        some_prog | dar -x - -v ... | some_other_prog
or
        dar -x - -i /tmp/pipe_todar -o /tmp/pipe_toslave -v ...

there is no order to run dar or dar_slave first, dar can use -i and/or -o,
while dar_slave does not. What is important here is to connect in a way or in
an other their input and output, it does not matter how. The only restriction
is that communication support must be perfect: no data loss, no duplication,
no order change, thus communication over TCP should be fine.

Of course, you can also isolate a catalogue through pipes, test an archive,
make difference, use a reference catalogue this way etc, and even then, output
the resulting archive to pipe ! If using -C or -c with "-" while using -A also
with "-", it is then mandatory to use -o: The output catalogue will generated
on standard output, thus to send order to dar_slave you must use another
channel with -o:

       LOCAL HOST                                   REMOTE HOST
   +-----------------+                     +-----------------------------+
   |   filesystem    |                     |     backup of reference     |
   |       |         |                     |            |                |
   |       |         |                     |            |                |
   |       V         |                     |            V                |
   |    +-----+      | backup of reference |      +-----------+          |
   |    | DAR |--<-]=========================[-<--| DAR_SLAVE |          |
   |    |     |-->-]=========================[->--|           |          |
   |    +-----+      | orders to dar_slave |      +-----------+          |
   |       |         |                     |      +-----------+          |
   |       +--->---]=========================[->--| DAR_XFORM |--> backup|
   |                 |        saved data   |      +-----------+ to slices|
   +-----------------+                     +-----------------------------+

on local host :
        dar -c - -A - -i /tmp/pipe_todar -o /tmp/pipe_toslave | some_prog

on the remote host :

        dar_slave -i /tmp/pipe_toslave -o /tmp/pipe_todar full_backup
 //dar_slave provides the full_backup for -A option

        some_other_prog | dar_xform - diff -s 140M -p ...
 //while dar_xform make slice of the output archive provided by dar

Last, if you don't want to mess with pipe, you have still the possibility to
create a VPN and mount a NFS partition over it. In some cases it may be
sufficient for what you want to do. See VPN-HOWTO for simple implementation
using sshd and pppd.


 III - bytes, bits, kilo, mega etc.
====================================

you probably know a bit the metric system, where a dimension is expressed by a
base unit (the meter for distance, the liter for volume, the joule for energy,
the volt for electrical potential, the bar for pressure, the watt for
power, the second for time, etc.), and declined using prefixes:

deci  (d) = 0.1
centi (c) = 0.01
milli (m) = 0.001
micro (u) = 0.000,001 (symbol is not "u" but the "mu" Greek letter)
nano  (n) = 0.000,000,001
pico  (p) = 0.000,000,000,001
femto (f) = 0.000,000,000,000,001
atto  (a) = 0.000,000,000,000,000,001
zepto (z) = 0.000,000,000,000,000,000,001
yocto (y) = 0.000,000,000,000,000,000,000,001
deca (da) = 10
hecto (h) = 100
kilo  (k) = 1,000
mega  (M) = 1,000,000
giga  (G) = 1,000,000,000
tera  (T) = 1,000,000,000,000
peta  (P) = 1,000,000,000,000,000
exa   (E) = 1,000,000,000,000,000,000
zetta (Z) = 1,000,000,000,000,000,000,000
yotta (Y) = 1,000,000,000,000,000,000,000,000

This way two milliseconds are 0.002 second, and 5 kilometers are 5,000 meters.
All was fine and nice up to the recent time when computer science appeared:
In this discipline, the need to measure the size of information storage raised.
The smallest size, is the bit (contraction of binary digit), binary because it
has two possible states: "0" and "1". Grouping bits by 8 computer scientists
called it a "byte". A byte has 256 different states, (2 power 8). The ASCII
(American Standard Code for Information Interchange) code arrived and assigned
a letter or more generally a character to some value of a byte, (A is assigned
to 65, space to 32, etc). And as most text is composed of a set of character,
they started to count size in byte. Time after time, following technology
evolution, memory size approached 1000 bytes, but for some obscure reason it
was decided that a "kilobyte" would be 1024 bytes not 1000 bytes. Some time
after, and by extension, a megabyte has been defined to be 1024 kilobytes,
a terabyte to be 1024 megabytes, etc.

In parallel, in the telecommunications domain, going from analogical to
digital signal made the bit to be used also. In place of the analogical signal,
took place a flow of bits, representing the samples of the original signal.
For telecommunications the problem was more a problem of size of flow:
how much bit could be transmitted by second. At some ancient time appeared
the 1200 bit by second, then 64000, also designed as 64 Kbit/s. Thus here,
kilo stays in the usual meaning of 1000 time the base unit. You can also find
Ethernet 10 Mbit/s which is 10,000 bits by seconds, same thing with Token-Ring
that had 4, 16 or 100 Mbit by seconds (4,000, 16,000 or 100,000 bits/s). But,
even for telecommunications, kilo is not always 1000 times the base unit:
the E1 bandwidth at 2Mbit/s for example, is in fact 32*64Kbit/s thus
2048 Kbit/s ... not 2000 Kbit/s

Anyway, back to dar, you have to possibility to give the size in byte or using
a single letter as suffix (k or K, M, T, P, E, Z, Y) thus the possibility to
provide a size in kilo, mega, tera, peta, exa, zetta or yotta byte, with the
computer science definition of theses terms (power of 1024).

Theses suffixes are for simplicity and not to have to computer
how much make powers of 1024. For example, if you want to fill a CD-R you will
have to use the "-s 650M" option which is equivalent to "-s 6815744400", choose
the one you prefer, the result is the same :-). Now, if you want 2 Megabytes
slices in the sense of the metric system, simply use "-s 2000000".

Yes, and to make things more confuse, marketing arrived and made sellers count
gigabits a third way: I remember the last hard disk I bought, was described
as "2.1 GB", (OK, that's several couple of years ago), but in fact it had only
2097152 bytes available. This is far from 2202009 bytes (= 2.1 GB for computer
science meaning), and a bit more than 2,000,000 bytes (metric system). OK, if
it had theses 2202009 bytes (computer science meaning of 2.1 GB), this hard
disk would have been sold under the label "2.5 GB"! Don't you think so?
... just kidding :-)


  IV - archive structure in brief
===================================

Slice level
-----------------

a slice is composed of a header and data

+--------+-------------------------------------------+
| header |  Data                                     |
|        |                                           |
+--------+-------------------------------------------+

the slice header is composed of
- a magic number that tells this is a dar slice
- a internal_name which is unique to a given archive
- a flag that tells if the slice is the last of the archive
- a extension flag, that says if some extension field is present
  possible extension field is the size of the slices (-s option) when
  -S is used

+-------+----------+------+-----------+
| Magic | internal | flag | extension |
| Num.  | name     | byte | byte      |
+-------+----------+------+-----------+

or for the first slice if -s and -S are used together
+-------+----------+------+-----------+-------------------+
| Magic | internal | flag | extension | following slices  |
| Num.  | name     | byte | byte      | size              |
+-------+----------+------+-----------+-------------------+

the header is the first thing to be written, and if it is not the last slice,
the flag field is overwritten. The header is also the first part to be read.

To know where a given position takes place in the archive, dar must first read
the first archive, to know the size of the first slice, and if extension is
present, the size of following slices is read. This is why at start-up dar
always asks the first slice.

Archive Level
----------------

the archive level describes the structure of the slice's data field,
when they are stick together across slices.

+---------+-----------------------------------------+-----------+------+
| version |     Data                                | catalogue | term |
| header  |                                         |           |      |
+---------+-----------------------------------------+-----------+------+

the header version is composed of:
- edition version of the archive
- compression algorithm used
- command line used for creating the archive
- flag that tells if root or user EA have been saved

+---------+------+---------------+------+
| edition | algo | command line  | flag |
|         |      |               |      |
+---------+------+---------------+------+

the data is a suite of file contents, with EA if present

  ....--+---------------------+----+------------+-----------+----+---....
        |  file data          | EA | file data  | file data | EA |
        | (may be compressed) |    | (no EA)    |           |    |
  ....--+---------------------+----+------------+-----------+----+---....

the catalogue, contains all inode, directory structure and hard_links
information. The directory structure is stored in a simple way:
the inode of a directory comes, then the inode of the files it contains, then
a special entry named "EOD" for End of Directory.

considering the following tree:

 - toto
    | titi
    | tutu
    | tata
    |   | blup
    |   +--
    | boum
    | coucou
    +---

it would generate the following sequence for catalogue storage:

+-------+------+------+------+------+-----+------+--------+-----+
|  toto | titi | tutu | tata | blup | EOD | boum | coucou | EOD |
|       |      |      |      |      |     |      |        |     |
+-------+------+------+------+------+-----+------+--------+-----+

EOD takes on byte, and this way no need to store the full path of
each file, just the filename is recorded.

the terminator stores the position of the beginning of the catalogue, it is the
last thing to be written. Thus dar first reads the terminator, then the
catalogue.

ALL together:
---------------------

here is an example of how data can be structured in a four slice archive:

+-------------+--------+------------------------+
| slice + ext | version|  file data + EA        |
| header      | header |                        |
+-------------+--------+------------------------+
the first slice has been defined smaller using the -S option

+--------+--------------------------------------------+
| slice  |           file data + EA                   |
| header |                                            |
+--------+--------------------------------------------+

+--------+--------------------------------------------+
| slice  |           file data + EA                   |
| header |                                            |
+--------+--------------------------------------------+

+--------+---------------------+-----------+------+
| slice  |   file data + EA    | catalogue | term |
| header |                     |           |      |
+--------+---------------------+-----------+------+
the last slice is smaller because there was not enough data to make it full.

The archive is written quite sequentially, except when creating a new
slice, the flag in the slice header has to be changed to non terminal.

Else, Dar read first the slice header of the first slice, then the version
header, then the terminator and catalogue (located on the last slice) and then
proceed to the operation. If it is extracting the whole archive, dar goes
back to the first slice and asks for all slices one by one.

QUESTION:
why not having put the slices information at the end, for dar not to ask
the first then the last slice ?

slicing and archiving are approached in two independent ways. For slicing,
making header at the end of the file would require much more complicated and
heavy code, because, slice have variable size, and header too. Then it would
have cost more memory and process to manage the end of slice, when reading
data you must not provide the header as data.

keeping slicing and archiving as two independent classes, is really necessary
for dar to evaluate. Without this, it would not have been so easy to create
dar_xform, which is only concerned by "sar" class (the C++ class that
implements slicing). So putting the slicing information in the catalogue or
terminator is really a bad idea for long term evolution and maintenance point
of view.

maybe the sar class will receive a new implementation one day in that way that
header get stored at the end of slices, and overall slicing information would
be stored at the end of the last slice. This way, dar would not ask the first
slice before asking for the last one, in the meanwhile, you will have to first
provide the first slice in any case.



   V - EA Support & Compilation Problems
=========================================

if you just want to compile DAR with EA support you just need
the attr-x.x.x.src.tar.gz package. (I've been using attr-2.0.5.src.tar.gz).
if you want to use EA, then you need to change your kernel upgrade a few
commands. I will just detail the DAR compilation here.

I personally got some problem to compile dar with EA support, due to EA package
installation problem:

when installing EA package, the /usr/include/attr directory is not created
nor the xattr.h file put in it. To solve this problem, create it manually,
and copy the xattr.h (and also attributes.h even if it is not required by dar)
to it, giving it proper permission (world readable). These include files can be
found in the "include" subdir of the xattr package:

as root type the following replacing <package> by the path where your package
has been compiled:

cd /usr/include
mkdir attr
chmod 0755 attr
cp <package>/include/xattr.h <package>/include/attributes.h attr
cd attr
chmod 0644 *

the second problem is while linking, the static library version does not exist.
you can built it using the following command (after package compilation):

as previously as root type:

chdir <package>/libattr
ar r libattr.a syscalls.o libattr.o
mv libattr.a /usr/lib
chmod 0644 /usr/lib/libattr.a

dar should now be able to compile with EA_SUPPORT activated in its makefile.


  VI - run DAR in background
==============================

DAR can be run in background:

        dar [command-line arguments] < /dev/null &


 VII - run command or scripts from DAR
=======================================

this concerns options -E and -F. They both receive a string as argument. Thus
if the argument must be a command with its own arguments, you have to put
theses between quotes for they appear as a single string to the shell that
interprets the dar command-line. For example:

you want to call

        df .

this is two worlds df (the command) and '.' its argument. Then you have to use
the following on DAR command-line:

        -E "df ."
or
        -E 'df .'


DAR provides several substitution strings:

        %% is replaced by a single % Thus if you need a % in you command line
           you MUST replace %% in the argument string of -E
        %p is replaced by the path to the slices
        %b is replaced by the basename of the slices
        %n is replaced by the number of the slice
        %c is replaced by the context (OPERATION or INIT)

The number of the slice (%n) is either the just written slice or the next
slice to be read. For example if you make a backup (-c or -C), this is the
number of the last slice completed. Else (using -t, -d, -A (with -c or -C), -l
 or -x), this is the number of the slice that will be required very soon.
While %c (the context) is substitued by INIT or OPERATION. INIT when the slice
is asked before the catalogue is read and OPERATION once it has been read.

What the use of this feature ?

For example you want to burn the band-new slices on CD as soon as they are
available

let's build a little script for that:

        %cat burner
        #!/bin/tcsh -f

        if("$1" == "" || "$2" == "") then
           echo "usage: $0 <filename> <number>"
           exit 1
        endif

        mkdir T
        mv $1 T
        mkisofs -o /tmp/image.iso -r -J -V "archive_$2" T
        cdrecord dev=0,0 speed=8 -data /tmp/image.iso
        rm /tmp/image.iso
        if(! diff /mnt/cdrom/$1 T/$1) then
          exit 2
        else
          rm -rf T
        endif
        %

This little script, receive the slice filename, and its number as argument,
what it does is to burn a CD with it, and compare the resulting CD with the
original slice. Upon failure, the script return 2 (or 1 if syntax is not
correct on the command-line)

on could then use it this way:

        -E "./burner %p/%b.%n.dar %n"

which can make the following DAR command-line:

        dar -c ~/tmp/example -z -R / usr/local -s 650M -E "./burner %p/%b.%n.dar %n" -p

First note that as our script does not change CD from the device, we need to
pause between slices (-p option). The pause take place after the execution of
the command (-E option). Thus we could add in the script a command to send a
mail or play a music to inform us that the slice is burned. The advantage,
here is that we don't have to come twice by slices, once when the slice is
ready, once when the slice is burnt.


Another example:

you want to send a huge file by email. (OK that's better to use FTP,
but sometimes, people think than the less you can do the more they
control you, and thus they disable many services, either by fear of
the unknown, either by stupidity). So you only have mail available
to transfer your data:

        dar -c toto -s 2M my_huge_file -E "uuencode %b.%n.dar %b.%n.dar | mail -s 'slice %n' your@email.address ; rm %b.%n.dar ; sleep 300"

Here we make an archive with slices of 2 Megabytes, because our mail system
does not allow larger emails. We save only one file: "my_huge_file" (but we
could even save the whole filesystem it would also work). The command we
execute each time a slice is ready is:

  uuencode the file and send the output my email to our address.
  remove the slice
  and wait 5 minutes, to no overload too much the mail system, This is also
  useful, if you have a small mailbox, from which it takes time to retrieve
  mail.

Note that we did not used the %p substitution string, as the slices are saved
in the current directory.

Last example, is while extracting, in the case the slices cannot all be
present in the filesystem, you need a script or a command to fetch the soon
to be requested slice. It could be using ftp, lynx, ssh, etc. I let you do
the script as an exercise. :-)


VIII - Makefile targets (does only concern versions 1.x.x)
========================

Here follows user available targets and macros to be used with make.
- A target is a word that can be put as argument of make like "all" or
  "clean".
- A macro is a variable that can be changed in the Makefile or changed on the
  make command-line, this way :

        make INSTALL_ROOT_DIR="/some/where/else"

targets:
---------

default : builds dar dar_xform and dar_slave (used if not target is given)
all     : builds dar dar_xform dar_slave and test programs
depend  : rebuild file dependency. This modifies the Makefile
install : install dar software and manual pages as described by
          INSTALL_ROOT_DIR BIN_DIR and MAN_DIR macros
install-doc : install documentation (tutorial notes, etc.) as described by
          the INSTALL_ROOT_DIR and INSTALL_DOC_DIR macros
uninstall : remove dar software, man pages and documentation if present
          as described by macro used for installation
test    : only builds test programs
clean_all : remove all generated files (temporary and final files)
clean   : remove all files except the C++ generated files with the "usage"
          extension
usage   : builds the dar-help program, and generates "*.usage" files that
          contains generated the C++ code  corresponding to the help text
          displayed with option -h.
clean_usage : remove C++ generated files


Macros:
--------
DAR_VERSION       : DO NOT CHANGE IT ! : get dar version from source
INSTALL_ROOT_DIR  : can be changed     : base directory for installation
BIN_DIR           : can be changed     : subdir where to store binaries
MAN_DIR           : can be changed     : subdir where to store man pages
DOC_DIR           : can be changed     : subdir where to store doc files
EA_SUPPORT        : set or unset it    : if set add support for EA
FILEOFFSET        : set or unset it    : if set, support for large files
USE_SYS_SIGLIST   : set or unset it    : if set, uses the sys_siglist vector
OS_BITS           : set or unset it    : if set, change int macros for alpha OS
CXX               : can be changed     : point to your C++ compiler
CC                : can be changed     : point to your C compiler





  IX - Scrambling
==================

How it works ?

take the pass phrase. It is a string, thus a sequence of bytes, thus a
sequence of integer each one between 0 and 255 (including 0 and 255).
The data to "scramble" is also a sequence of byte, usually much longer
than the pass phrase. The principle is to add byte by byte the pass
phrase to the data, modulo 256. The pass phrase is repeated all along
the archive. Let's take an example:

the pass phrase is "he\220lo" (where \220 is the character which value
is 220).
the data is "example"

taken from ASCII standard:
h = 104
l = 108
o = 111
e = 101
x = 120
a = 97
m = 109
p = 112

        e       x       a       m       p       l       e
        101     120     97      109     112     108     101

+       h       e       \220    l       o       h       e
        104     101     220     108     111     104     101

---------------------------------------------------------------

        205     221     317     217     223     212     202

---------------------------------------------------------------
modulo
256 :   205     221     61      217     223     212     202
        \205    \201    =       \217    \223    \212    \202

thus the data "example" will be written in the archive
"\205\201=\217\223\212\202"

This method is the only I know that allows to decode any portion
without knowing the rest of the data. It does not consume much
resources to compute, but it is probably terribly weak and easy
to crack. Of course, the data is more difficult to retrieve
without the key when the key is long. if you have any
information about the protection level of this algorithm, or know
another better algorithm that allows decryption of any part of
the data without knowing the rest of the data, let me know ! :-)

Some asked me to put PGP inside DAR. But some expert in security
disagreed: It must stay independent. In fact the only  way to not
loose the possibility to restore a few files without having to
read all the archive (as tar does), would be to encrypt data file
by file. Which makes the protection to the data very weak compared
to a global encryption of the archive or just slice by slice.


   X - dar_manager
===================

dar_manager is the last member of the dar suite. Its role is to
gather information about several backups to easily and
automatically restore the last versions of a given set of file
over many different backups (up to 65534).

A first thing to do is to build a "database" from archives or
their extracted catalogues (-C then -B plus -A options). You may
have several databases, they will be independent of each others.
Each database is stored in a single (compressed) file.

When you will need a particular file to be restored, using the
collected information, dar_manager will call dar with proper
options to restore the file(s) from the correct archive. This
is particularly useful when making differential backups, because
all files are not saved at each backup (those that did not
changed) the last version of a given file may be located in
an archive done many time ago.

As dar_manager calls dar, it must know the path to each archive.
By default, it uses the path given using -A option, as well as
the basename given. But you might be feeding the database with
an extracted catalogue (dar's -C option), while the real archive
is stored on a CD with a different basename. You may either
use the -b and -p option to change the basename and path at
any time, or adding a optional argument to -A option.

Next point, you may need some special options to always be passed to
dar, this is the use of the -o option.

last point dar_manager looks for dar in the PATH variable. But
you can also specify which dar command you want to be used (-d
option), if it is not in the PATH or for security reasons.

Normal operation is to update your database(s) after each new
archive has been created. And when you need to restore a particular file or set
of files, you will just have to call dar with -r option:

dar -r file1 home/my_directory tmp/file2 ...


  XI - file extension used
===========================

dar suite programs use several type of files:

- slices              (dar, dar_xform, dar_slave, dar_manager)
- configuration files (dar, dar_xform, dar_slave)
- databases           (dar_manager)
- user command        (dar, dar_xform, dar_slave)

if for slice the extension and even the filename format cannot be customized,
(basename.slicenumber.dar) there is not mandatory rule for the other type of
files.

In the case you have no idea how to name theses, here is what I use:
configuration files receive ".dcf" as extension, (Dar Configuration file)
while database receive      ".dmd" as extension, (Dar Manager Database)
for user command I propose  ".duc" as extension. (Dar User Command)

but, you are totally free to use the filename you want !   ;-)




 XII - dar and ssh (see also chapter II)
========================================

As reported "DrMcCoy" in the historical forum "Dar Technical Questions", the
netcat program can be very helpful if you plane to backup over the network.

The context in which will take place the following examples are a "local" host
named "flower" has to be backup or restored form/to a remote host called
"honey" (OK, the name of the machines are silly...)

example of use with netcat. Note that netcat command name is "nc"

1 - full backup of "flower" saved on "honey"

        on honey:
                nc -l -p 5000 > backup.1.dar

        then on flower:
                dar -c - -R / -z | nc -w 3 honey 5000

    but this will produce only one slice, instead you could
    use the following to have several slices on honey:

        on honey:
                nc -l -p 5000 | dar_xform -s 10M -S 5M -p - backup

        on flower:
                dar -c - -R / -z | nc -w 3 honey 5000

   by the way note that dar_xform can also launch a user script between
   slices exactly the same way as dar does, thanks to the -E and -F options.


2 - testing the archive

   testing the archive can be done on honey but you could also do it remotely
   even if it is not very useful !

        on honey:
                nc -l -p 5000 | dar_slave backup | nc -l -p 5001

        on flower:
                nc -w 3 honey 5001 | dar -t - | nc -w 3 honey 5000

   note also that dar_slave can run a script between slices, if for example
   you need to load slices from a robot, this can be done automatically, or if
   you just want to mount/unmount a removable media eject or load it and ask
   the user to change it ...

3 - comparing with original filesystem

        on honey:
                nc -l -p 5000 | dar_slave backup | nc -l -p 5001

        on flower:
                nc -w 3 honey 5001 | dar -d - -R / | nc -w 3 honey 5000

4 - making a differential backup

    Here the problem is that dar needs two pipes to send orders and read data
    coming from dar_slave, and a third pipe to write out the new archive. This
    cannot be realized only with stdin and stdout as previously. Thus we will
    need a named pipe (created by the mkfifo command).

        on honey:
                nc -l -p 5000 | dar_slave backup | nc -l -p 5001

                nc -l -p 5002 | dar_xform -s 10M -p - diff_backup

        on flower:
                mkfifo toslave
                nc -w 3 honey 5000 < toslave &
                nc -w 3 honey 5001 | dar -A - -o toslave -c - -R / -z | nc -w 3 honey 5002


with netcat the data goes in clear over the network. You could use ssh instead
if you want to have encryption over the network. The principle are the same.

1 - full backup of "flower" saved on "honey"
    we assume you have a sshd daemon on flower
        on honey:
                ssh flower dar -c - -R / -z > backup.1.dar

        or still on honey:
                ssh flower dar -c - -R / -z | dar_xform -s 10M -S 5M -p - backup

2 - testing the archive
        on honey:
                dar -t backup

        or from flower: (assuming you have a sshd daemon on honey)

                ssh honey dar -t backup

3 - comparing with original filesystem

        on flower:
                mkfifo todar toslave
                ssh honey dar_slave backup > todar < toslave &
                dar -d - -R / -i todar -o toslave

        or on honey:
                mkfifo todar toslave
                ssh flower dar -d - -R / > toslave < todar &
                dar_slave backup > todar < toslave

4 - making a differential backup

        on flower:
                mkfifo todar toslave
                ssh honey dar_slave backup > todar < toslave &
        and on honey:
                ssh flower dar -c - -A - -i todar -o toslave > diff_linux.1.dar
                or
                ssh flower dar -c - -A - -i todar -o toslave | dar_xform -s 10M -S 5M -p - diff_linux



XIII - overflow in arithmetic integer operations
==================================================

Some code explanation about the detection of integer arithmetic operation
overflows. We speak about *unsigned* integers, and we have only portable
standard ways to detect overflows, when using 32 bits or 64 bits integer in
place of infinint.

developed in binary a number is a finite suite of digits (0 or 1). To obtain
the original number from the binary representation, we must multiply each digit
by a power of two.
example the binary representation "101101" designs the number N where:

        N = 2^5 + 2^3 + 2^2 + 1*2^0

in that context I will say that 5 is the maximum power of N (the power of the
higher non null binary digit).

for the addition "+" operation, if an overflow occurs, the result is less
than one or both operands

for substraction "-" operation, if the second operand is greater than the first
there will be an overflow (result must be unsigned thus positive) else there
will not be any overflow.

for division "/" and modulo "%" operations, there is never overflow (there is
just illicit division by zero case).

for multiplication "*" operation, I've chosen an heuristic to quickly detect
overflow:

given :
        A and B two integers, which max powers are m and n respectively, we
have

        A < 2^(m+1)
and
        B < 2^(n+1)

thus we also have:

        A.B < 2^(m+1).2^(n+1)

which is

        A.B < 2^(m+n+2)

by consequences we know that the maximum power of the product of A by B is at
most m+n+1 and while m+n+1 is less than or equal to the maximum power of the
integer field there will not be overflow else we consider there will be
an overflow even if it may not be always the case (this is an heuristic
algorithm).


 XIV - using data protection with DAR & PAR
=============================================

Parchive (PAR in the following) is a very nice program that makes possible to
recover a file which has been corrupted. It creates redundancy data stored in
a separated file (or set of files), which can be used to repair the original
file. This additional data may also be damaged, PAR will be able to repair
the original file as well as the redundancy files, up to a certain point, of
course. This point is defined by the percentage of redundancy you defined for
a given file. But,... check the official PAR site here:

         http://parchive.sourceforge.net

First to remind you the chapter XI above, dar can use two types of files.
- DUC files (Dar User Scripts), are any script or command. They are bind to
dar through the -E option, and are launch after slice creation or before
slice reading.
- DCF files (Dar Configuration Files), are text files, that extend the command
line arguments. They must respect a well defined syntax (see man page
'conditional syntax' for details). Theses configuration files are bind to dar
through the -B option.

In the following we will present two DUC files, and one DCF file.  All theses
files are distributed and normally installed under $prefix/shared/dar
directory where $prefix is /usr/local by default, or else the path given to
--prefix=... configure option.

The discussion here is to show how to use PAR with DAR. Two DUC files
are available for that. They are intended to be called between
slice by dar thanks to dar's -E option. the first "dar_par_create.duc"
generates redundancy files for each slice. Theses files will be required
later if corruption occurred on a slice. This script is expected to be
used when creating an archive (-c option or -C option).

        dar -c some_archive -E "/usr/local/share/dar/dar_par_create.duc %p %b %n %e %c 20" ... and other options to dar

The second "dar_par_test.duc", tests the coherence of the redundancy files
together with the slice they protect. If a corruption is detected, the
scripts asks Parchive to repair the slice (thus this will fail on CD-ROM,
as the filesystem is Read-Only, but we will see further how to fix that). This
script is also expected to be used as argument to -E option, but this time,
when testing an archive (dar's -t option), or "diffing" (-d option) an archive
with a filesystem.

        dar -t some_archive -E "/usr/local/share/dar/dar_par_test.duc %p %b %n %e %c" ... and other options to dar

In both previous examples, the %p %b %n %e %c are macro that dar replaces
respectively by the path, basename, slice number, extension, context of the
last slice created or the next slice to read depending on the operation asked
(backup or restore for example). See dar's man page and chapter XV of this file
below for more.

Now, to avoid you having to type all theses -E options, a DCF file named
"dar_par.dcf" is provided. You can thus replace the -E options and arguments
by
        -B /usr/local/share/dar/dar_par.dcf

you could thus type

        dar -c some_archive -B /usr/local/share/dar/dar_par.dcf ...
        dar -t some_archive -B /usr/local/share/dar/dar_par.dcf ...
        dar -l some_archive -B /usr/local/share/dar/dar_par.dcf ...
        etc.

If you plane to always use Parchive with dar, you can even add
"-B /usr/local/share/dar/dar_par.dcf" in your $HOME/.darrc file ! This way,
dar would always use Parchive (unless -N option is given on command-line).

here is an example:
#cat ~/.darrc
all:
        -B /usr/local/share/dar/dar_par.dcf

 ... and so on.


What to do of this extra data files generated by PAR ?

You can put them after each slice on a CD-R, which requires you to adjust the
slice size a few megabytes below the size of the CD-R to have enough room left
to add the extra PAR files. Note that the amount of data generated by par
depends on the redundancy rate specified on command-line (PAR -r option).

You can also gather the PAR data of all the slices of your archive and put then
on a separated disk. That's up to you.

The problem is when the corrupted slice is on a CD-R and thus cannot be
repaired in place. You then need to copy it on your hard disk and run PAR to
repair it. The problem is that with the standard 'cp' command to copy the
corrupted slice to disk, you will not be able to access the data present after
the corruption. Thus, you may miss too much data for it may be repaired by PAR.
To solve this problem, the command 'dar_cp' (which is a replacement for 'cp'
and which is installed with dar) can be used. It will skip over the I/O error
and continue the copy with good readable data. This way you will only miss the
corrupted data, which should be most of the time possible to recover thanks to
PAR redundancy files.

Once repaired, you can burn it again, but you may rather put symbolic links in
the directory where resided your repaired slice, each symbolic link pointing
to the name of the *other* slices on CD-R which are not corrupted. Dar must
be given a single directory where all slice will be fetched.

Let's take an example. You have an archive which basename is 'coucou' and which
is made of 183 slices of 650 MB each except the last one which is only 459 MB.

Calling

        dar -B /usr/local/share/dar/dar_par.dcf -t /mnt/cdrom/coucou

shows that Parchive detected an error on slice 80 but could not repair it, as
the filesystem (on CD-R) is read-only. Thus, you need to copy it on you hard
disk for reparation:

        dar_cp /mnt/cdrom/coucou.80.dar /tmp

we need also to copy the redundancy files

        dar_cp /mnt/cdrom/coucou.80.dar.par2 /tmp
        dar_cp /mnt/cdrom/coucou.80.dar.vol000+992.par2 /tmp

then we repair it thanks to Parchive:

        cd /tmp
        par2repair coucou.80.dar

if this succeeded, then you can burn it back on a new CD-R with all its
parity files. But you can also, add, beside the coucou.80.dar slice, as many
symbolic link as there is other slices located on CD:

        cd /tmp
        ln -s coucou.1.dar /mnt/cdrom
        ln -s coucou.2.dar /mnt/cdrom
        etc... up to (but except slice 80)
        ln -s coucou.183.dar /mnt/cdrom

then you can restore your archive giving /tmp in place of /mnt/cdrom to dar:

        dar -x /tmp/coucou -R ... etc.

dar will find all slice on CD-R (thanks to the symbolic links we created),
except for slice 80 which is the repaired one and which dar will find on hard
disk.


  XV - Dar User Command
============================

Since version 1.2.0 dar's user can make dar calling their commands or scripts
between slices, thanks to the -E and -F options. To be able to easily share
your commands or scripts, I propose you the following convention:

  - use the ".duc" extension to show anyone the script/command respect the
    following
  - must be called from dar with the following arguments:

        example.duc %p %b %n %e %c [other optional arguments]

  - must provide brief help on what it does and what are the expected
    arguments, when called without argument. This is the standard
    "usage:" convention.

Then, any user, could share their "dar user commands" (i.e.: DUC files) and
don't bother much about how to use them. Moreover it would be easy to chain
them:

if for example two persons created their own script, one "burn.duc" which
burns a slice on CD-R(W) and "par.duc" which makes a Parchive redundancy file
from a slice, anybody could use both at a time giving the following argument
to dar:

        -E "par.duc %p %b %n %e %c 1 ; burn.duc %p %b %n %e %c"

or since version 2.1.0 with the following argument:

        -E "par.duc %p %b %n %e %c 1" -E "burn.duc %p %b %n %e %c"

of course a script has not to use all its arguments, in the case of burn.duc
for example, the %c (context) is probably useless, and not used inside the
script, while it is still possible to give it all the "normal" arguments of a
DUC file, extra not used argument are simply ignored.

If you have interesting DUC scripts, you are welcome to contact me by email,
for I add them on the web site and in the following releases. For now, check
doc/samples directory for a few examples of DUC files.

Note that starting version 2.1.0, several -B options will be possible,
each given command will be called in the order of their appearance of the
corresponding -B option.


 XVI - Examples of file filtering
==================================

file filtering is what defines which files are saved, listed, restored,
compared, tested, and so on, in brief, I will say which file are elected for
the operated in the following, meaning by "operation, either a backup, a
restoration, a archive contents listing, a archive comparison, etc.

file filtering is done using the following options -X, -I, -P, -R and with
the any non options argument given on the command-line (which is the man page
is referred as [list of path]). A non option argument is one that does not
start by a dash ('-' character). If however you need to give in the [list of
path] an argument that starts by a dash, you need to add the '--' argument
after which no search is done for command line options.

OK, Let's start with some concretes examples:

	dar -c toto

this will backup the current directory and all what is located into it to
build the toto archive, also located in the current directory. Usually you
should get a warning telling you that you are about to backup the archive
itself

Now let's see something less obvious:

	dar -c toto -R / home/ftp

the -R option tell dar to consider all file under the / root directory, while
the "home/ftp" argument is not an option argument, and thus is a member of
[list of path]. This tells dar to restrict the operation only on the
home/ftp subdirectory of the given root directory thus here /home/ftp.

But this is a little bit different than the following:

	dar -c toto -R /home/ftp

here dar will save any file under /home/ftp without any restriction. So what
is the difference ? Yes, exactly the same files will be saved as just above,
but the file /home/ftp/welcome.msg for example, will be stored as
<ROOT>/welcome.msg . Where <ROOT> will be replaced by the argument given to
-R option (which defaults to "."), at restoration or comparison time. While in
the previous example the same file would have been stored with the following
path <ROOT>/home/ftp/welcome.msg .


	dar -c toto -R / -P home/ftp/pub  home/ftp etc

as previously, but the -P option make all files under the /home/ftp/pub not to
be considered for the operation. Additionally the /etc directory and its
subdirectories are saved, here [list of path] contains two members.

	dar -c toto -R / -P etc/password etc

here we save all the /etc except the /etc/password file. Arguments given to -P
can be plain files also. But when they are directory this exclusion applies to
the directory itself and its contents. Note that using -X to exclude "password"
does have the same effect:

	dar -c toto -R / -X "password" etc

will save all the /etc directory except any file with name equal to "password".
thus of course /etc/password will no be saved, but if it exists,
/etc/rc.d/password will not be saved neither if it is not a directory. Yes,
if a directory /etc/rc.d/password exist, it will not be affected by the -X
option. As well as -I, -X option do not apply to directories. The reason is
to be able to filter some kind of file without excluding a particular directory
for example you want to save all mp3 files and only MP3 files,

	dar -c toto -R / -I "*.mp3" -I "*.MP3" home/ftp

will save any mp3 o MP3 ending files under the /home/ftp directories and
subdirectories. If instead -I (or -X) applied to directories, we would only be
able to recurse in subdirectories ending by ".mp3" or ".MP3". If you had a
directory named "/home/ftp/Music" for example, full of mp3, you would not have
been able to save it.

Note that the glob expressions (where comes the wild-card '*' '?' and so on),
can do much more complicated things like "*.[mM][pP]3" you could replace the
previous example by

	dar -c toto -R / -I "*.[mM][pP]3" home/ftp

this would cover all .mp3 .mP3 .Mp3 and .MP3 files.

Last an very complete example:

	dar -c toto -R / -P "*/.mozilla/*/[Cc]ache" -X ".*~" -X ".*~" -I "*.[Mm][pP][123]" home/ftp -- -fake

so what ?

OK, here we save all under /home/ftp and /-fake but we do not save the contents
of "*/.mozilla/*/[Cc]ache" like for example
"/home/ftp/.mozilla/ftp/abcd.slt/Cache" directory and its contents. In theses
directories we save any file matching "*.[Mm][pP][123]" files except those
ending by a tilde (~ character), Thus for example file which name is
"toto.mp3" or ".bloup.Mp2"

Now the inside algorithm:

	a file is elected for operation if
 1 - its name does not match any -X option or it is a directory
*and*
 2 - if some -I is given, file is either a directory or match at least one
     of the -I option given.
*and*
 3 - path and filename do not match any -P option
*and*
 4 - if some non option are given (building a [list of path]) the path to the
     file is one of the member of [list of path] or a subdirectory of one of
     the members [list of path] .

