
#include "bacula_template.inc" title="Storage Daemon Design"

<H1>Storage Daemon Design</H1>

<h2>General</h2>
This chapter is intended to be a technical discussion of the
Storage daemon services and as such is not targeted at end users but
rather at developers and system administrators that want or
need to know more of the working details of <b>Bacula</b>.

<h2>Introduction</h2>
The Bacula Storage daemon provides storage resources to a 
Bacula installation. An individual Storage daemon is associated with a physical 
permanent storage device (for example, a tape drive, CD writer, tape changer or 
jukebox, etc.), and may employ auxiliary storage resources (such as space on a 
hard disk file system) to increase performance and/or optimize use of the 
permanent storage medium. 
<p>Any number of storage daemons may be run on a given machine; each associated 
with an individual storage device connected to it, and BACULA operations may 
employ storage daemons on any number of hosts connected by a network, local or 
remote. The ability to employ remote storage daemons (with appropriate security 
measures) permits automatic off-site backup, possibly to publicly available 
backup repositories. 

<h2>Development Outline</h2>
In order to provide a high performance backup and 
restore solution that scales to very large capacity devices and networks, the 
storage daemon must be able to extract as much performance from the storage device 
and network with which it interacts. In order to accomplish this, storage daemons 
will eventually have to sacrifice simplicity and painless portability in favor 
of techniques which improve performance. My goal in designing the storage daemon 
protocol and developing the initial prototype storage daemon is to provide for 
these additions in the future, while implementing an initial storage daemon which 
is very simple and portable to almost any POSIX-like environment. This original 
storage daemon (and its evolved descendants) can serve as a portable solution for 
non-demanding backup requirements (such as single servers of modest size, 
individual machines, or small local networks), while serving as the starting 
point for development of higher performance configurable derivatives which use 
techniques such as POSIX threads, shared memory, asynchronous I/O, buffering to 
high-speed intermediate media, and support for tape changers and jukeboxes. 


<h2>Connections and Sessions</h2>A client connects to a storage server by 
initiating a conventional TCP connection. The storage server accepts the 
connection unless its maximum number of connections has been reached or the 
specified host is not granted access to the storage server. Once a connection has 
been opened, the client may make any number of Query requests, and/or initiate 
(if permitted), one or more Append sessions (which transmit data to be stored by 
the storage daemon) and/or Read sessions (which retrieve data from the storage 
daemon). 
<p>Most requests and replies sent across the connection are simple ASCII
strings, with status replies prefixed by a four digit status code for
easier parsing.  Binary data appear in blocks stored and retrieved from
the storage.  Any request may result in a single-line status reply of
&quot;<TT>3201&nbsp;Notification&nbsp;pending</TT>&quot;, which
indicates the client must send a &quot;Query notification&quot; request
to retrieve one or more notifications posted to it.  Once the
notifications have been returned, the client may then resubmit the
request which resulted in the 3201 status.
<p>The following descriptions omit common error codes, yet to be defined, which 
can occur from most or many requests due to events like media errors, restarting 
of the storage daemon, etc. These details will be filled in, along with a 
comprehensive list of status codes along with which requests can produce them in 
an update to this document. 


<h3>Append Requests</h3>
<DL>
  <DT><SPAN class=ascii>append open session</SPAN> = &lt;JobId&gt; [ 
  &lt;Password&gt; ] 
  <DD>A data append session is opened with the Job ID given by 
  <EM>JobId</EM> with client password (if required) given by <EM>Password</EM>. 
  If the session is successfully opened, a status of <TT>3000&nbsp;OK</TT> is 
  returned with a &quot;<TT>ticket&nbsp;=&nbsp;</TT><EM>number</EM>&quot; reply used to 
  identify subsequent messages in the session. If too many sessions are open, or 
  a conflicting session (for example, a read in progress when simultaneous read 
  and append sessions are not permitted), a status of 
  &quot;<TT>3502&nbsp;Volume&nbsp;busy</TT>&quot; is returned. If no volume is mounted, or 
  the volume mounted cannot be appended to, a status of 
  &quot;<TT>3503&nbsp;Volume&nbsp;not&nbsp;mounted</TT>&quot; is returned. 
  <p></p>
  <DT><SPAN class=ascii>append data</SPAN> = &lt;ticket-number&gt;
  <DD>If the append data is accepted, a 
  status of <TT>3000&nbsp;OK data address = &lt;IPaddress&gt; port = &lt;port&gt;</TT> is returned,
  where the <tt>IPaddress</tt> and <tt>port</tt> specify the IP address and
  port number of the data channel. Error status codes are 
  <TT>3504&nbsp;Invalid&nbsp;ticket&nbsp;number</TT> and 
  <TT>3505&nbsp;Session&nbsp;aborted</TT>, the latter of which indicates the 
  entire append session has failed due to a daemon or media error. 
  <p>Once the File daemon has established the connection to the data channel
  opened by the Storage daemon, it will transfer a header packet followed
  by any number of data packets. The header packet is of the form:
  <p><tt>&lt;file-index&gt; &lt;stream-id&gt; &lt;info&gt;</tt></p>
  <p>The details are specified in the <a href="daemonprotocol.html">Daemon Protocol</a>
  section of this document.
  <p></p>
  <DT><SPAN class=ascii>*append abort session</SPAN> = &lt;ticket-number&gt;
  <DD>The open append session with ticket <EM>ticket-number</EM> is aborted; any blocks 
  not yet written to permanent media are discarded. Subsequent attempts to 
  append data to the session will receive an error status of 
  <TT>3505&nbsp;Session&nbsp;aborted</TT>. 
  <p></p>
  <DT><SPAN class=ascii>append end session</SPAN> = &lt;ticket-number&gt; 
  <DD>The open append session with ticket <EM>ticket-number</EM> is marked complete; no 
  further blocks may be appended. The storage daemon will give priority to saving 
  any buffered blocks from this session to permanent media as soon as possible. 
  <p></p>
  <DT><SPAN class=ascii>append close session</SPAN> = &lt;ticket-number&gt; 
  <DD>The append session with ticket <EM>ticket</EM> is closed. This message 
  does not receive an <TT>3000&nbsp;OK</TT> reply until all of the content of the 
  session are stored on permanent media, at which time said reply is given, 
  followed by a list of volumes, from first to last, which contain blocks from 
  the session, along with the first and last file and block on each containing 
  session data and the volume session key identifying data from that session in 
  lines with the following format: 
  <p><code><TT>Volume = </TT>&lt;Volume-id&gt; &lt;start-file&gt; &lt;start-block&gt;
  &lt;end-file&gt; &lt;end-block&gt; &lt;volume-session-id&gt;</code>where <EM>Volume-id</EM> is the volume label, 
  <EM>start-file</EM> and <EM>start-block</EM> are the file and block containing the 
  first data from that session on the volume, <EM>end-file</EM> and 
  <EM>end-block</EM> are the file and block with the last data from the session on 
  the volume and <EM>volume-session-id</EM> is the volume session ID for blocks from the 
  session stored on that volume. </DD></DL>
  
<h3>Read Requests</h3>
<DL>
  <DT><SPAN class=ascii>Read open session</SPAN> = &lt;JobId&gt; &lt;Volume-id&gt; &lt;start-file&gt; &lt;start-block&gt;
  &lt;end-file&gt; &lt;end-block&gt; &lt;volume-session-id&gt; &lt;password&gt; 
  <dd>where <EM>Volume-id</EM> is the volume label, 
  <EM>start-file</EM> and <EM>start-block</EM> are the file and block containing the 
  first data from that session on the volume, <EM>end-file</EM> and 
  <EM>end-block</EM> are the file and block with the last data from the session on 
  the volume and <EM>volume-session-id</EM> is the volume session ID for blocks from the 
  session stored on that volume.
  
  <p>If the session is successfully opened, a status of 
  <p><code>
  <TT>3100&nbsp;OK Ticket&nbsp;=&nbsp;</TT><EM>number</EM>&quot;</code>
  <p>is returned with a reply used to identify 
  subsequent messages in the session. If too many sessions are open, or a 
  conflicting session (for example, an append in progress when simultaneous read 
  and append sessions are not permitted), a status of 
  &quot;<TT>3502&nbsp;Volume&nbsp;busy</TT>&quot; is returned. If no volume is mounted, or 
  the volume mounted cannot be appended to, a status of 
  &quot;<TT>3503&nbsp;Volume&nbsp;not&nbsp;mounted</TT>&quot; is returned. If no block with 
  the given volume session ID and the correct client ID number appears in the 
  given first file and block for the volume, a status of 
  &quot;<TT>3505&nbsp;Session&nbsp;not&nbsp;found</TT>&quot; is returned. 
  <p></p>
  
  <DT><SPAN class=ascii>Read data</SPAN> = &lt;Ticket&gt; &gt; &lt;Block&gt; 
  <DD>The specified Block of data from open read session with the specified Ticket number 
  is returned, with a status of <TT>3000&nbsp;OK</TT> followed 
  by a &quot;<TT>Length&nbsp;=&nbsp;</TT><EM>size</EM>&quot; line giving the length in 
  bytes of the block data which immediately follows. Blocks must be retrieved in 
  ascending order, but blocks may be skipped. If a block number greater than the 
  largest stored on the volume is requested, a status of 
  &quot;<TT>3201&nbsp;End&nbsp;of&nbsp;volume</TT>&quot; is returned. If a block number 
  greater than the largest in the file is requested, a status of 
  &quot;<TT>3401&nbsp;End&nbsp;of&nbsp;file</TT>&quot; is returned. 
  <p></p>
  
  <DT><SPAN class=ascii>Read close session</SPAN> = &lt;Ticket&gt; 
  <DD>The read session with Ticket number is closed. A read session 
  may be closed at any time; you needn't read all its blocks before closing it. 
  </DD></DL>

<!--  commented out -------------------

<h3>Query Requests</h3>
<DL>
  <DT><SPAN class=ascii>*Query configuration</SPAN> [ = <EM>string</EM> ] 
  <DD>A status line of &quot;<TT>3000 OK</TT>&quot; is returned, followed by the server's 
  configuration in &quot;name = value&quot; format as given in the configuration file. If 
  a <EM>string</EM> is specified, only configuration records with that type are 
  returned. Security-related information (such as Permit and Deny statements) 
  may not be retrieved by a configuration query. Items returned are in a 
  canonical form and may not be identical to the entries in the server's 
  configuration file, nor in the same order. 
  <p></p>
  <DT><SPAN class=ascii>*Query volume</SPAN> 
  <DD>A status line of &quot;<TT>3000 OK</TT>&quot; is returned, followed by (at least) the 
  following information regarding the current volume mounted by the storage 
  daemon. If no volume is mounted, the volume label will be blank. 
  <p><code><TT>Media type = </TT><EM>string</EM><BR><TT>Media capacity = 
    </TT><EM>size</EM><BR><TT>Volume label = </TT><EM>string</EM><BR><TT>Volume 
    capacity used = </TT><EM>size</EM><BR><TT>File number = 
    </TT><EM>number</EM><BR><TT>Block number = 
    </TT><EM>number</EM><BR><TT>Access mode = 
    </TT><EM>Read/Append|Read|Append</EM><BR></code>
  <p></p>
  <DT><SPAN class=ascii>*Query sessions</SPAN> 
  <DD>A status line of &quot;<TT>3000 OK</TT>&quot; is returned, followed by (at least) the 
  following information regarding the current activity of the daemon. 
  <p><code><tt>Maximum connections = </TT><EM>number</EM><BR><TT>Open 
    connections = </TT><EM>number</EM><BR><TT>Maximum append sessions = 
    </TT><EM>number</EM><BR><TT>Open append sessions = 
    </TT><EM>number</EM><BR><TT>Maximum read sessions = 
    </TT><EM>number</EM><BR><TT>Open read sessions = 
    </TT><EM>number</EM><BR><TT>Last cleaning = 
</TT><EM>time</EM><BR></code>
  <p></p>
  <DT><SPAN class=ascii>*Query notifications</SPAN> 
  <DD>A status line of &quot;<TT>3000 OK</TT>&quot; is returned, followed by one or more 
  lines containing notifications queued for this client. Each notification 
  consists of a line which begins with the type of the notification followed by 
  an equal sign, the <EM>time</EM> it was posted, and a <EM>string</EM> 
  containing the text of the notification. If no notifications are posted to 
  this client, a status of &quot;<TT>3501&nbsp;No&nbsp;notifications&nbsp;posted</TT>&quot; 
  is returned. </DD></DL>

  
<h3>Volume Requests</h3>
<DL>
  <DT><SPAN class=ascii>*Volume mount</SPAN> [ = <EM>string</EM> ] 
  <DD>If the <EM>string</EM> is the same as the currently mounted volume, 
  nothing occurs. If one or more sessions are active on the currently mounted 
  volume a status of &quot;<TT>3502&nbsp;Volume&nbsp;busy</TT>&quot; is returned. 
  Otherwise, any currently mounted volume is unmounted and the requested volume 
  given by <EM>string</EM> is mounted (or whatever volume is loaded into the 
  device, is no <EM>string</EM> is given and anonymous mounts are permitted), 
  then a status of <TT>3000&nbsp;OK</TT> is returned. 
  
  <DT><SPAN class=ascii>*Volume unmount</SPAN> = <EM>string</EM> 
  <DD>If one or more sessions are active on the currently mounted volume a 
  status of &quot;<TT>3502&nbsp;Volume&nbsp;busy</TT>&quot; is returned. Otherwise, if 
  <EM>string</EM> is equal to the currently mounted volume's label, it is 
  unmounted and a status of <TT>3000&nbsp;OK</TT> is returned. If the currently 
  mounted volume does not match the <EM>string</EM>, no action is taken and a 
  status of &quot;<TT>3503&nbsp;Volume&nbsp;not&nbsp;mounted</TT>&quot; is returned. 
  <DT><SPAN class=ascii>Volume suspend</SPAN> = <EM>string</EM> 
  <DD>If one or more sessions are active on the currently mounted volume a 
  status of &quot;<TT>3502&nbsp;Volume&nbsp;busy</TT>&quot; is returned. Otherwise, if 
  <EM>string</EM> is equal to the currently mounted volume's label, it is 
  unmounted and a status of <TT>3000&nbsp;OK</TT> is returned, and the storage 
  daemon will open no further sessions on the volume; this allows shutting down 
  access to a volume prior to mounting another. If the currently mounted volume 
  does not match the <EM>string</EM>, no action is taken and a status of 
  &quot;<TT>3503&nbsp;Volume&nbsp;not&nbsp;mounted</TT>&quot; is returned. </DD></DL>
  
 -------  end comment out  -->


<p>
<HR>

<ADDRESS>by <A href="http://www.fourmilab.ch/">John Walker</A><BR>January 30th, 
MM </ADDRESS>

<page_prev name="Bacula Director Services" url="director.html">
<page_next name="Bacula File Services" url="file.html">
