


MM_DATA(5)                                             MM_DATA(5)


NNAAMMEE
       MM data - NetWorker media multiplexor data (tape and disk)
       format

DDEESSCCRRIIPPTTIIOONN
       This documents the data format that  the  NetWorker  media
       multiplexor daemon, nnssrrmmmmdd(8), writes to long term storage
       media such as tapes and optical disks.  See  nnssrr__ddeevviiccee(5)
       and nnssrrmmmm(8) for a discussion of supported device families
       and types.  The format described here applies to any fixed
       record  device,  such  as  raw disks, or fixed record tape
       devices with file marks.  NetWorker uses the _e_X_t_e_r_n_a_l _D_a_t_a
       _R_e_p_r_e_s_e_n_t_a_t_i_o_n  (XDR) standard to write media which can be
       interchanged among a wide variety of machines.   Only  the
       mechanism  used  to  multiplex  save  set streams onto the
       storage media is described here; the formats of  save  set
       streams  depend  on  the type of NetWorker client, and are
       described in nnssrr__ddaattaa(5).

       A _v_o_l_u_m_e is one physical piece of media  such  as  a  tape
       reel  or disk cartridge.  A tape volume is made up of mul-
       tiple media _f_i_l_es, and each media file may contain several
       media  _r_e_c_o_r_ds.   These media files and records should not
       be confused with a client's (for example UNIX or DOS) user
       files  or  records; the two do not necessarily correspond.
       For example, a given media file or  even  a  single  media
       record  may  contain  many small client user files. On the
       other hand, a single large client file may be split across
       several  media  files,  and  even  across several volumes.
       Media files do not span volume boundaries.  Save sets  may
       span media files and even volumes.

       On  most tapes, media files can be skipped very quickly by
       the device's hardware or associated  device  driver  soft-
       ware,  and  the  hardware can detect when an end of a file
       has been reached.  On some  tapes,  records  can  also  be
       quickly  skipped  forward.  Otherwise, access to the media
       is sequential.

       Media records are  described  by  the  mmrreeccoorrdd  structure.
       Label  records are fixed in size, MMIINNMMRREECCSSIIZZEE bytes. Other
       records can potentially be a larger size that must be some
       constant  for  the  rest  of the volume.  NetWorker always
       writes, reads and skips data in units of full-sized  media
       records.   Each  mmrreeccoorrdd  contains zero or more mmcchhuunnkks of
       data from either one or more client save sessions, or used
       internally for synchronization and labels.  The XDR format
       of a media file's mmrreeccoorrdds and mmcchhuunnkks are as follows:


       const MINMRECSIZE = 32768;      /* minimum media record size */
       const MMAXCHK = 2048;           /* max number of chunks in record */
       const MHNDLEN = 124;            /* private area len for handlers */




NetWorker rt_1998_4Q.Build.MMaJrFcehra0l8_,0021000                        1





MM_DATA(5)                                             MM_DATA(5)


       typedef unsigned long ssid_t;   /* save set id */
       typedef unsigned long ssoff_t;  /* an offset in a save set */
       typedef unsigned long volid_t;  /* key for the volume data base */

       struct  mchunk {
           ssid_t mc_ssid;             /* owning save set id */
           ssoff_t mc_low;             /* 1st byte, relative to save stream */
           opaque mc_data<MINMRECSIZE>;/* chunk's data */
       };

       struct  mrecord {
           opaque mr_handler[MHNDLEN]; /* private to media handler */
           u_long mr_orec;             /* record size */
           volid_t mr_volid;           /* encompassing volume's id */
           u_long mr_fn;               /* encompassing file number */
           u_long mr_rn;               /* record number within the file */
           u_long mr_len;              /* record byte length */
           mchunk mr_chunk<MMAXCHK>;   /* chunks of save streams */
       };


The first field of an mmrreeccoorrdd, mmrr__hhaannddlleerr, is reserved for media-
specific  data  (currently it is not used by any implementation).
The mmrr__oorreecc field is the size of the  current  record.   A  media
record's  header  fields, mmrr__vvoolliidd, mmrr__ffnn, and mmrr__rrnn, are used to
check the tape position and the data read from the  record.   The
file  numbers  and  record  numbers  start  at zero and increment
sequentially.  The record number is reset each time the file num-
ber is incremented.  On disks, file numbers are always zero.  The
mmrr__lleenn field is the actual number of valid bytes in this  record,
as opposed to the size of the device's read or write request.

If  file  or  record  skipping is unreliable, NetWorker can still
recover from isolated errors, at worst by rewinding  and  reading
the tape from the start.  If a volume can be physically unmounted
or mounted without notice to the media  management  daemon,  then
the volume identifier in each record provides a quick way of ver-
ifying when this happens, without the need for a full rewind  and
reading of the label in most cases.

The  mmcchhuunnkks  within  an  mmrreeccoorrdd contain client data from one or
more save sessions.  The mmcc__ssssiidd and mmcc__llooww values  are  used  to
reconstruct  the save streams from the chunks within the records.
The mmcc__ddaattaa field holds the actual data of  each  chunk.   For  a
given  save  set,  mmcc__llooww plus the length of mmcc__ddaattaa should equal
the following chunk's value for mmcc__llooww..  Save sets may by  inter-
mingled arbitrarily within media records.

The  first  chunk  of the first record of the first media file on
the volume encapsulates the vvoolluummee llaabbeell  information;  for  some
media,  the  second chunk contains additional volume information,
for example the media pool the volume belongs to; subsequent data
in  the  first  file is reserved for future expansion.  The label
may be duplicated in a second file for redundancy,  in  case  the



NetWorker rt_1998_4Q.Build.MMaJrFcehra0l8_,0021000                        2





MM_DATA(5)                                             MM_DATA(5)


first  copy of the label gets accidentally overwritten.  The for-
mats of the volume label and  additional  label  information  are
described by the following XDR data structures:


const MVOLMAGIC = 0x070460;     /* volume magic number */
const NSR_LENGTH = 64;          /* length of several strings */
const RAP_MAXVALLEN = 1024;     /* max length of EACH value */
const RAP_MAXNAMELEN = 64;      /* max length of attribute name */

struct mvollabel {
    u_long  mvl_magic;          /* medium volume verification number */
    u_long  mvl_createtime;     /* time at which volume labeled */
    u_long  mvl_expiretime;     /* time for volume to expire */
    u_long  mvl_recsize;        /* expected size of mrecords */
    volid_t mvl_volid;          /* medium volume id */
    string  mvl_volname<NSR_LENGTH>;/* medium volume name */
};

struct vallist {
    vallist *next;
    string value<RAP_MAXVALLEN>;/* attribute value */
};

struct attrlist {
    attrlist *next;
    string name<RAP_MAXNAMELEN>;/* attribute name */
    vallist *values;            /* attribute values */
};

/*
 * Additional information may includes the following attributes
 * (listed by the name they are stored with):
 * "volume pool" : the media pool
 */
struct mvolinfo {
    struct attrlist *mvi_attributes;/* any other information */
};


The  mmvvll__mmaaggiicc  field must be equal to MMVVOOLLMMAAGGIICC in order for the
chunk to represent a valid volume label.   If  the  volume  label
changes in the future, the new format will have another ``magic''
number, but the format described here must still be allowed.  The
mmvvll__vvoolliidd  is  an internal identifier assigned and managed by the
media manager.  The  mmvvll__vvoollnnaammee  is  the  volume  name  that  is
assigned when the media is first labeled.  The time fields are in
UST format - the number of seconds elapsed since 00:00 GMT,  Jan-
uary  1,  1970.   The  mmvvll__rreeccssiizzee  is the size of all subsequent
media records found on the tape.

The mmvvpp__ppooooll is the pool name that is assigned when the media  is
first  labeled.   Different  media  pools allow administrators to
segregate their data onto  sets  of  volumes.   Media  cannot  be



NetWorker rt_1998_4Q.Build.MMaJrFcehra0l8_,0021000                        3





MM_DATA(5)                                             MM_DATA(5)


reassigned from one media pool to another.  Pool names are a max-
imum of NNSSRR__LLEENNGGTTHH characters long.

Synchronization marks, called sscchhuunnkkss, are also written  periodi-
cally to the media for each save set.  Synchronization chunks are
used by ssccaannnneerr(8) when verifying or extracting directly  from  a
volume.  They are also used by nnssrrmmmmdd when trying to recover from
media errors during file recovery.  The following XDR data struc-
ture describes a synchronization chunk:


struct schunk {
    opaque ssi_host[NSR_LENGTH];   /* save set host */
    opaque ssi_name[NSR_LENGTH];   /* symbolic name */
    u_long ssi_time;               /* save time */
    u_long ssi_expiry;             /* expiration date */
    u_long ssi_size;               /* actual size saved */
    u_long ssi_nfiles;             /* number of files */
    ssid_t ssi_ssid;               /* ssid for this save set */
    u_long ssi_flag;               /* various flags, see below */
    u_long ssi_info;               /* volid or ssid, see below */
};

#define SSI_START       1          /* start of a save set */
#define SSI_SYNC        2          /* synchronization point */
#define SSI_CONT        3          /* continued from another volume */
#define SSI_END         4          /* end of this save set */
#define SSI_SSMASK      0x0000000f /* save set sync chunk type */
#define SSI_LBIAS       0x10000000 /* the level is included in the flags */
#define SSI_LMASK       0xff000000 /* mask to cover bits for level */
#define SSI_LSHIFT      24         /* shift amount for the level */
#define SSI_INCOMPLETE  0x00010000 /* not finished (aborted) */
#define SSI_CONTINUED   0x00800000 /* continued save set series */

The  ssssii__hhoosstt  is  the name of the index which contains this save
set.  Traditionally this is the host name of the client where the
save  set  originated.   The  ssssii__nnaammee is the save set name to be
presented to the user.  These are both  null-terminated  strings,
even though the fields are fixed length.  The ssssii__ttiimmee field con-
tains the create time of the save set  in  UST.   The  ssssii__eexxppiirryy
field  is  the  expiration  date, in UST for this save set.  This
field is zero if an explicit save set  expiration  time  was  not
specified  when  the  save  set  was  created.   The ssssii__ssiizzee and
ssssii__nnffiilleess are the number of bytes and number of files  saved  so
far  for  this save set.  The ssssii__ssssiidd is the save set identifier
of this save set.

The ssssii__ffllaagg indicates the type of  this  synchronization  chunk,
the  level of save set, and other information about the save set.
There are four basic types of synchronization marks that  can  be
found from examining ssssii__ffllaagg && SSSSII__SSSSMMAASSKK.  SSSSII__SSTTAARRTT is used to
mark the beginning of a save set.  SSSSII__SSYYNNCC marks a periodic syn-
chronization  point and is only written at an exact file boundary
in  the  save  set.   SSSSII__CCOONNTT  indicates  that   this   is   the



NetWorker rt_1998_4Q.Build.MMaJrFcehra0l8_,0021000                        4





MM_DATA(5)                                             MM_DATA(5)


continuation  of  a  save set that started on a different volume.
When ssssii__ffllaagg && SSSSII__SSSSMMAASSKK is  SSSSII__CCOONNTT,  ssssii__iinnffoo  contains  the
volume  identifier  for  the  save set's preceding volume.  These
synchronization chunks are used when a save set  spans  a  volume
boundary.  SSSSII__EENNDD marks the end of a save set.

If  the  SSSSII__LLBBIIAASS  bit  is set then ssssii__ffllaagg & SSSSII__LLMMAASSKK shifted
right by SSSSII__LLSSHHIIFFTT specifies the level of  the  save  set.   The
SSSSII__IINNCCOOMMPPLLEETTEE  bit  indicates  that this save set did not finish
properly.  This could be caused by  a  user  interrupting  an  in
progress save.

The  SSSSII__CCOONNTTIINNUUEEDD  bit indicates that this save set is logically
continued to or from another save set.  These continued save sets
are  used  to  handle very large save sets.  If the SSSSII__CCOONNTTIINNUUEEDD
bit is set and ssssii__ffllaagg && SSSSII__SSSSMMAASSKK is SSSSII__SSTTAARRTT, then  ssssii__iinnffoo
gives  the  previous save set id that this save set was continued
from.  If the SSSSII__CCOONNTTIINNUUEEDD bit is set and ssssii__ffllaagg &&  SSSSII__SSSSMMAASSKK
is  SSSSII__EENNDD,  then  ssssii__iinnffoo gives the next save set id that this
save set is continued to.

SSEEEE AALLSSOO
       nnssrr__ddeevviiccee(5),  nnssrr__ddaattaa(5),  nnssrrmmmm(8),  nnssrrmmmmdd(8),  nnssrrmm--
       mmddbbdd(8), nnssrr(8), ssccaannnneerr(8).

       RRFFCC 11001144 XXDDRR:: EExxtteerrnnaall DDaattaa RReepprreesseennttaattiioonn SSppeecciiffiiccaattiioonn































NetWorker rt_1998_4Q.Build.MMaJrFcehra0l8_,0021000                        5


