
#include "devel_template.inc" title="Bacula Daemon Protocol"

<H1>Daemon Protocol</H1>

<h2>General</h2>
This document describes the protocols used between the
various daemons.  As Bacula has developed, it has become
quite out of date. The general idea still holds true, but the
details of the fields for each command, and indeed the commands
themselves have changed considerably.
<p>It is intended to be a technical discussion of the
general daemon protocols and as such is not targeted at end users but
rather at developers and system administrators that want or
need to know more of the working details of <b>Bacula</b>.

<h2>Low Level Network Protocol</h2>
At the lowest level, the network protocol is handled by <b>BSOCK</b>
packets which contain a lot of information about the status of
the network connection: who is at the other end, etc. Each basic
<b>Bacula</b> network read or write actually consists of two
low level network read/writes. The first write always sends
four bytes of data in machine independent byte order.  If data is
to follow, the first four bytes are a positive non-zero integer indicating
the length of the data that follow in the subsequent write.  If the
four byte integer is zero or negative, it indicates a special request, a
sort of network signaling capability.  In this case, no data packet will
follow. The low level BSOCK routines expect that only a single thread
is accessing the socket at a time. It is advised that multiple threads
do not read/write the same socket. If you must do this, you must provide
some sort of locking mechanism. I would not be appropriate for efficiency
reasons to make every call to the BSOCK routines lock and unlock the
packet.

<h2>General Daemon Protocol</h2>
In general, all the daemons follow the following global rules.
There may be exceptions depending on the specific case. Normally,
one daemon will be sending commands to another daemon (specifically,
the Director to the Storage daemon and the Director to the 
File daemon). 
<ul type="disc">
  <li>Commands are always ASCII commands that are 
      upper/lower case dependent as well as space sensitive. 
  <li>All binary data is converted into ASCII (either with printf statements or
      using base64 encoding).  
  <li>All responses to commands sent are always
      prefixed with a return numeric code where codes in the 1000's are
      reserved for the Director, the 2000's are reserved for the 
      File daemon, and the 3000's are reserved for the Storage daemon.
   <li>Any response that is not prefixed with a numeric
       code is a command (or subcommand if you like) coming
       from the other end. For example, while the Director is
       corresponding with the Storage daemon, the Storage daemon
       can request Catalog services from the Director.  This
       convention permits each side to send commands to the
       other daemon while simultaneously responding to commands.</li>
   <li>Any response that is of zero length, depending on the context,
       either terminates the data stream being sent or terminates command mode
       prior to closing the connection. </li>
   <li>Any response that is of negative length is a special sign that
       normally requires a response. For example, during data transfer from the
       File daemon to the Storage daemon, normally the File daemon
       sends continuously without intervening reads. However, periodically,
       the File daemon will send a packet of length -1 indicating that
       the current data stream is complete and that the Storage 
       daemon should respond to the packet with an OK, ABORT JOB,
       PAUSE, etc.  This permits the File daemon to efficiently send
       data while at the same time occasionally &quot;polling&quot;
       the Storage daemon for his status or any special requests.
       <p></p>
       Currently, these negative lengths are specific to the daemon, but
       shortly, the range 0 to -999 will be standard daemon wide signals,
       while -1000 to -1999 will be for Director user, -2000 to -2999
       for the File daemon, and -3000 to -3999 for the Storage 
       daemon.</li>
</ul>

<h2>The Protocol Used Between the Director and the Storage Daemon</h2>
Before sending commands to the File daemon, the Director opens a 
Message channel with the Storage daemon, identifies itself and presents
its password. If the password check is OK, the Storage daemon
accepts the Director. The Director then passes the Storage daemon,
the JobId to be run as well as the File daemon authorization
(append, read all, or read for a specific session). The Storage daemon
will then pass back to the Director a enabling key for this JobId
that must be presented by the File daemon when opening the job.
Until this process is complete, the Storage daemon is
not available for use by File daemons.
<pre>
SD: listens
DR: makes connection
DR: Hello &lt;Director-name&gt; calling &lt;password&gt;
SD: 3000 OK Hello
DR: JobId=nnn Allow=(append, read) Session=(*, SessionId) 
                    (Session not implemented yet)
SD: 3000 OK Job Authorization=&lt;password&gt;
DR: use device=&lt;device-name&gt; media_type=&lt;media-type&gt 
        pool_name=&lt;pool-name&gt; pool_type=&lt;pool_type&gt;
SD: 3000 OK use device
</pre>
For the Director to be authorized, the &lt;Director-name&gt; and the 
&lt;password&gt; must match the values in one of the Storage daemon's
Director resources (there may be several Directors that can access a 
single Storage daemon).
<h2>The Protocol Used Between the Director and the File Daemon</h2>
A typical conversation might look like the following:
<pre>
FD: listens
DR: makes connection
DR: Hello &lt;Director-name&gt; calling &lt;password&gt;
FD: 2000 OK Hello
DR: JobId=nnn Authorization=&lt;password&gt;
FD: 2000 OK Job
DR: storage address = &lt;Storage daemon address&gt; port = &lt;port-number&gt;
          name = &lt;DeviceName&gt; mediatype = &lt;MediaType&gt;
FD: 2000 OK storage
DR: include
DR: &lt;directory1&gt;
DR: &lt;directory2&gt;
    ...
DR: Null packet
FD: 2000 OK include
DR: exclude
DR: &lt;directory1&gt;
DR: &lt;directory2&gt;
    ...
DR: Null packet
FD: 2000 OK exclude
DR: full
FD: 2000 OK full
DR: save
FD: 2000 OK save
FD: Attribute record for each file as sent to the
    Storage daemon (described above).
FD: Null packet
FD: &lt;append close responses from Storage daemon&gt;
    e.g. 
    3000 OK Volumes = &lt;number of volumes&gt;
    3001 Volume = &lt;volume-id&gt; &lt;start file&gt; &lt;start block&gt; 
         &lt;end file&gt; &lt;end block&gt; &lt;volume session-id&gt; 
    3002 Volume data = &lt;date/time of last write&gt; &lt;Number bytes written&gt; 
         &lt;number errors&gt;
    ... additional Volume / Volume data pairs for volumes 2 .. n
FD: Null packet

FD: close socket

</pre>


<h2>The Save Protocol Between the File Daemon and the Storage Daemon</h2>
Once the Storage daemon has issued the <b>save</b> command, the File 
daemon will contact the Storage daemon to begin the save.
<p></p>
<p>
In what follows: FD: refers to information set via the network
from the File daemon to the Storage daemon, and SD: refers to
information set from the Storage daemon to the File daemon.

<h3>Command and Control Information</h3>
Command and control information is exchanged in 
human readable ASCII commands.
<pre>
FD: listens
SD: makes connection
FD: append open session = &lt;JobId&gt; [&lt;password&gt;]
SD: 3000 OK ticket = &lt;number&gt;
FD: append data &lt;ticket-number&gt;
SD: 3000 OK data address = &lt;IPaddress&gt; port = &lt;port&gt;

</pre>
<h3>Data Information</h3>
The Data information consists of
the file attributes and data to the Storage daemon.
For the most part, the data information is sent one way: from
the File daemon to the Storage daemon. This allows the
File daemon to transfer information as fast as possible
without a lot of handshaking and network overhead.
<p>However, from time to time, the File daemon needs
to do a sort of checkpoint of the situation to ensure
that everything is going well with the Storage daemon.
To do so, the File daemon sends a packet with a negative
length indicating that he wishes the Storage daemon to
respond by sending a packet of information to the File
daemon. The File daemon then waits to receive a packet from
the Storage daemon before continuing.
<p>
All data sent are in binary format except for the
header packet, which is in ASCII.
There are two packet types used data transfer mode: a header packet, the 
contents of which are known to the Storage daemon, and a
data packet, the contents of which are never examined by the
Storage daemon.
<p>


The first data packet to the Storage daemon  
will be an ASCII header packet consisting of the following
data. 
<p class="tty"> &lt;File-Index&gt; &lt;Stream-Id&gt; &lt;Info&gt;</p>

where <b>&lt;File-Index&gt;</b> is a sequential number beginning from one that 
increments with each file (or directory) sent. 
<p></p>
where <b>&lt;Stream-Id&gt;</b> will be 1 for the Attributes record and
2 for uncompressed File data. 3 is reserved for the MD5 signature for the file.
<p></p>
where <b>&lt;Info&gt;</b> transmit information about the
Stream to the Storage Daemon. It is a character string field where
each character has a meaning. The only character currently defined
is 0 (zero), which is simply a place holder (a no op). In the future,
there will be codes indicating compressed data, encrypted data, etc.
<p></p>
Immediately following the header packet, the Storage daemon will
expect any number of data packets. The series of data packets
is terminated by a zero length packet, which indicates to the Storage
daemon that the next packet will be another header
packet. As previously mentioned, a negative length packet is a
request for the Storage daemon to temporarily enter command mode
and send a reply to the File daemon.

Thus an actual conversation might contain the following exchanges:
<pre>
FD: &lt;1 1 0&gt; (header packet)
FD: &lt;data packet containing file-attributes&gt;
FD: Null packet
FD: &lt;1 2 0&gt;
FD: &lt;multiple data packets containing the file data&gt;
FD: Packet length = -1
SD: 3000 OK
FD: &lt;2 1 0&gt;
FD: &lt;data packet containing file-attributes&gt;
FD: Null packet
FD: &lt;2 2 0&gt;
FD: &lt;multiple data packets containing the file data&gt;
FD: Null packet
FD: Null packet


FD: append end session &lt;ticket-number&gt;
SD: 3000 OK end
FD: append close session &lt;ticket-number&gt;
SD: 3000 OK Volumes = &lt;number of volumes&gt;
SD: 3001 Volume = &lt;volumeid&gt; &lt;start file&gt; &lt;start block&gt; 
     &lt;end file&gt; &lt;end block&gt; &lt;volume session-id&gt; 
SD: 3002 Volume data = &lt;date/time of last write&gt; &lt;Number bytes written&gt; 
     &lt;number errors&gt;
SD: ... additional Volume / Volume data pairs for 
     volumes 2 .. n
FD: close socket
</pre>
The information returned to the File daemon by the Storage daemon in
response to the <b>append close session</b> is transmit in turn to the Director.


<!--    begin comment out ---------------------
<h3>Misc</h3>
<p>
NOTE, THIS NEEDS SOME REDOING, PLEASE IGNORE ...
<p></p>
Assume you're creating a tar-like archive once again.  
You're copying successive files into client blocks of any 
size you like and sending them to the storage daemon when 
each fills up.  At the start of each file, you note the 
client block number and offset within that block where the 
file starts - this constitutes the &quot;storage coordinates&quot; of 
the file, relative to the session.  At the end of the session, 
you obtain the list of volume names, positions, and block 
ranges used by the storage daemon for that session and store 
that as the &quot;session coordinates&quot;.
Now suppose we want to restore a file from the session.
The catalog entry tells us:
<pre>
Client session ID
Client session start block
Client session start offset
</pre>
we then look up the Client session ID and obtain the list of volumes and ranges.  
We next determine from this the volume which contains the first block of the file 
we want to restore.  This allows us to open a read session on that volume starting 
at the given file, media block, and client session start block.  
The first block the storage daemon returns will be the one containing 
the start of the requested file, and the client session start offset tells 
us the byte offset within that block to start reading.  
From there we just read successive blocks until the end of the 
requested file is reached.

 end comment out ------------ -->

<page_prev name="Porting Notes" url="porting.html">
<page_next name="Storage Media Output Format" url="mediaformat.html">
