
#include "devel_template.inc" title="Bacula Developer Notes"

<H1>Bacula Developer Notes</H1>

<h2>General</h2>
This document is intended mostly for developers and describes the 
the general framework of making Bacula source changes.
<h3>Contributions</h3>  
Contributions from other programmers will be broken into
two groups. The first are contributions that are aids and
not essential to Bacula. In general, these will be scripts or
will go into and examples or contributions directory. 
<p>The second class of contributions are those which will
be integrated with Bacula and become an essential part.
Within this class of contributions, there are two hurdles
to surmount. One is getting your patch accepted, and two is
dealing with copyright issues.
The rest of this document describes some of the requirements
for such code.

<h3>Patches</h3>
Subject to the copyright hurdle described below, your patches should be
sent in <b>diff -u</b> format relative to the current contents
of the Source Forge CVS, which is the easiest for me to
understand.  If you plan on doing significant development work over a
period of time, after having your first patch reviewed and approved, you
will be eligible for having CVS access so that you can commit your
changes directly to the CVS repository. To do so, you will need a
userid on Source Forge.

<h3>Copyrights</h3>
To avoid future problems concerning changing licensing or copyrights,
all code contributions more than a hand full of lines
must be in the Public Domain or have the
copyright assigned to Kern Sibbald and John Walker as in the current code.
This double copyright ownership ensures that no single person controls
the software.
<p>
Your name should be clearly indicated as the author of the code, and you
must be extremely careful not to violate any copyrights or use
other people's code without acknowledging it. The purpose
of this requirement is to avoid future problems copyright, patent,
or intellectual property  problems.
To understand on possible source of future problems, please
examine the difficulties Mozilla is (was?) having finding previous
contributors at <a href="http://www.mozilla.org/MPL/missing.html">
http://www.mozilla.org/MPL/missing.html</a>.  The other important
issue is to avoid copyright, patent, or intellectual property
violations as are currently (May 2003) being claimed by SCO against
IBM.
<p>Although the copyright will be held by Kern and John, each
developer is expected to indicate that he wrote and/or modified
a particular module (or file) and any other sources.
<p>Although we have absolutely no plans for any commercial
venture, it would be a shame to totally rule it out.
Our desire is that if one day
something commercial is developed around Bacula (as 
is the case with MySQL, RedHat, Ximian, and other organizations),
each developer will have an opportunity to participate,
at a minimum, proportional to his prior contributions. 
Again, we have no plans for creating any such commercial 
enterprise.
<p>
If you have any doubts about this, please don't hesitate to ask.  Our
(John and my) track records with Autodesk are easily available; early
programmers/founders/contributors and later employees had substantial
shares of the company, and no one founder had a controlling part of the
company.  Even though Microsoft created many millionaires among early
employees, the politics of Autodesk (during our time at the helm) is in
stark contrast to Microsoft where the majority of the company is still
tightly held among a few.
<p>
Items not needing a copyright assignment are: most small changes,
enhancements, or bug fixes of 5-10 lines of code, and documentation.

<h3>Copyright Assignment</h3>
Since this is not a commercial enterprise, and I prefer to believe
in everyone's good faith, developers can assign the copyright 
by explicitly acknowledging that they do so in their first
submission.

<h3>Developing Bacula</h3>
Typically the simplest way to develop Bacula is to open one
xterm window pointing to the source directory you wish to update;
a second xterm window at the top source directory level, and a
third xterm window at the bacula directory &lt;top&gt;/src/bacula.
After making source changes in one of the directories, in the
top source directory xterm, build the source, and start the
daemons by entering:
<p class=tty>make</p>
and
<p class=tty>./startit</p>
then in the enter:
<p class=tty>./console</p> or
<p class=tty>./gnome-console</p>

to start the Console program.  Enter any commands for testing.  For
example: run kernsverify full.
<p>Note, the instructions here to use <b>./startit</b>
are different from using a production system where the
administrator starts Bacula by entering
<b>./bacula start</b>.  This difference allows a development version of
<b>Bacula</b> to be run on a computer at the same time that a production
system is running.  The <b>./startit</b> strip starts <b>Bacula</b>
using a different set of configuration files, and thus permits avoiding
conflicts with any production system.

<p>To make additional source changes,
exit from the Console program, and in the top source directory,
stop the daemons by entering:
<p class=tty>./stopit</p>
then repeat the process.

<h3>Debugging</h3>
Probably the first thing to do is to turn on debug output.
<p>A good place to start is with a debug level of
20 as in <b>./startit -d20</b>. The startit command starts
all the daemons with the same debug level. Alternatively, 
you can start the appropriate
daemon with the debug level you want.  If you really need more info,
a debug level of 60 is not bad, and for just about everything a level
of 200.

<h3>Using a Debugger</h3>
If you have a serious problem such as a segmentation
fault, it can usually be found quickly using a good
multiple thread debugger such as <b>gdb</b>.  For
example, suppose you get a segmentation violation in
<b>bacula-dir</b>.  You might use the following to find
the problem:
<p class="tty">
&lt;start the Storage and File daemons&gt;<br>
cd dird<br>
gdb ./bacula-dir<br>
run -f -s -c ./dird.conf<br>
&lt;it dies with a segmentation fault&gt;<br>
where<br>
</p>
The <b>-f</b> option is specified on the <b>run</b>
command to inhibit <b>dird</b> from going into the
background. You may also want to add the <b>-s</b>
option to the run command to disable signals which
can potentially interfere with the debugging.
<p>As an alternative to using the debugger, each
<b>Bacula</b> daemon has a built in back trace
feature when a serious error is encountered. It
calls the debugger on itself, produces a back trace,
and emails the report to the developer. For
more details on this, please see the chapter in this
manual entitled <a href="kaboom.html">What To Do When
Bacula Crashes (Kaboom)</a>.

<h3>Memory Leaks</h3>
Because Bacula runs routinely and unattended on
client and server machines, it may run for a long time.  As a
consequence, from the very beginning, Bacula uses SmartAlloc
to ensure that there are no memory leaks. To make detection
of memory leaks effective, all Bacula code that dynamically
allocates memory MUST have a way to release it.  In general when the
memory is no longer needed, it should be immediately released, but in
some cases, the memory will be held during the entire time that Bacula
is executing.  In that case, there MUST be a routine that can be called
at termination time that releases the memory.  In this way, we will be
able to detect memory leaks.  Be sure to immediately correct any and all
memory leaks that are printed at the termination of the daemons.

<h3>Special Files</h3>
Kern uses files named 1, 2, ... 9 with any extension as scratch
files. Thus any files with these names are subject to being rudely
deleted at any time.


<h3>When Implementing Incomplete Code</h3>
Please identify all incomplete code with a comment that contains
<b>***FIXME***</b>, where there are three asterisks (*) before and after
the word FIXME (in capitals) and no intervening spaces.  This is
important as it allows new programmers to easily recognize where things
are partially implemented.

<h3>Bacula Source File Structure</h3>
<p>The distribution generally comes as a tar file of the form
<b>bacula.x.y.z.tar.gz</b> where x, y, and z are the version,
release, and update numbers respectively.
<p>Once you detar this file, you will have a directory structure
as follows:
<pre>
|
|- depkgs                  
   |- mtx              (autochanger control program + tape drive info)
   |- sqlite           (SQLite database program)
|- depkgs-win32   
   |- pthreads         (Native win32 pthreads library -- dll)
   |- zlib             (Native win32 zlib library)
   |- wx               (wxWidgets source code)
|- bacula              (main source directory containing configuration
   |                    and installation files)
   |- autoconf         (automatic configuration files, not normally used
   |                    by users)
   |- doc              (documentation directory)
      |- home-page     (Bacula's home page source)
      |- html-manual   (html document directory)
      |- techlogs      (Technical development notes);
   |- intl             (programs used to translate)
   |- platforms        (OS specific installation files)
      |- redhat        (Red Hat installation)
      |- solaris       (Sun installation)
      |- freebsd       (FreeBSD installation)
      |- irix          (Irix installation -- not tested)
      |- unknown       (Default if system not identified)
   |- po               (translations of source strings)
   |- src              (source directory; contains global header files)
      |- cats          (SQL catalog database interface directory)
      |- console       (bacula user agent directory)
      |- dird          (Director daemon)
      |- filed         (Unix File daemon)
         |- win32      (Win32 files to make bacula-fd be a service)

      |- findlib       (Unix file find library for File daemon)
      |- gnome-console (GNOME version of console program)
      |- lib           (General Bacula library)
      |- stored        (Storage daemon)
      |- tconsole      (Tcl/tk console program -- not yet working)
      |- testprogs     (test programs -- normally only in Kern's tree)
      |- tools         (Various tool programs)
      |- win32         (Native Win32 File daemon)
         |- baculafd   (Visual Studio project file)
         |- compat     (compatibility interface library)
         |- filed      (links to src/filed)
         |- findlib    (links to src/findlib)
         |- lib        (links to src/lib)
         |- console    (beginning of native console program)
         |- wx-console (wxWidget console Win32 specific parts)
     |- wx-console     (wxWidgets console main source program)

|- regress             (Regression scripts)
   |- bin              (temporary directory to hold Bacula installed binaries)
   |- build            (temporary directory to hold Bacula source)
   |- scripts          (scripts and .conf files)
   |- tests            (test scripts)
   |- tmp              (temporary directory for temp files)
</pre>

<h3>Header Files</h3>
Please carefully  follow the scheme defined below as it permits in
general only two header file includes per C file, and thus 
vastly simplifies programming. With a large complex project like
Bacula, it isn't always easy to ensure that the right headers are
invoked in the right order (there are a few kludges to make this
happen -- i.e. in a few include files because of the chicken
and egg problem, certain references to typedefs had to be
replaced with <b>void</b> ).
<p>
Every file should include <b>bacula.h</b>. It pulls in just about
everything, with very few exceptions.  If you have system dependent
ifdefing, please do it in <b>baconfig.h</b>. The version number and
date are kept in <b>version.h</b>. 
<p>Each of the subdirectories (console, cats, dird, filed, findlib, lib,
stored, ...) contains a single directory dependent include file generally the name of the 
directory, which should be included just after the include of <b>bacula.h</b>.
This file (for example, for the dird directory, it is <b>dird.h</b>) contains
either definitions of things generally needed in this directory, or it
includes the appropriate header files. It always includes <b>protos.h</b>. 
See below.
<p>Each subdirectory contains a header file named <b>protos.h</b>, which 
contains the prototypes for subroutines exported by files in that directory.
<b>protos.h</b> is always included by the main directory dependent include
file.

<h3>Programming Standards</h3>
For the most part, all code should be written in C unless there
is a burning reason to use C++, and then only the simplest C++
constructs will be used.  Note, Bacula is slowly evolving to use
more and more C++.

<p>Code should have some documentation -- not a lot, but enough
so that I can understand it. Look at the current code, and you
will see that I document more than most, but am definitely not
a fanatic.
<p>I prefer simple linear code where possible. Gotos are strongly
discouraged except for handling an error to either bail out 
or to retry some code, and such use of gotos can vastly simplify
the program.
<p>
Remember this is a C program that is migrating to a <b>tiny</b>
subset of C++, so be conservative in your use of C++ features.

<h3>Do Not Use</h3>
<ul>
 <li>STL -- it is totally incomprehensible.</li>
</ul>

<h3>Avoid if Possible</h3>
<ul>
 <li>Returning a malloc'ed buffer from a subroutine --
     someone will forget to release it.</li>
 <li>Using reference variables -- it allows subroutines to
     create side effects.</li>
 <li>Heap allocation (malloc) unless needed -- it is expensive.</li>
 <li>Templates -- they can create portability problems.</li>
 <li>Fancy or tricky C or C++ code, unless you give a
     good explanation of why you used it.</li>
 <li>Too much inheritance -- it can complicate the code,
   and make reading it difficult (unless you are in love
   with colons)</li>
</ul>

<h3>Do Use Whenever Possible</h3>
<ul>
  <li>Locking and unlocking within a single subroutine.</li>
  <li>Malloc and free within a single subroutine.</li>
  <li>Comments and global explanations on what your code or
     algorithm does.</li>
</ul>

<h3>Indenting Standards</h3>
I cannot stand code indented 8 columns at a time.  This makes the code
unreadable.  Even 4 at a time uses a lot of space, so I have adopted
indenting 3 spaces at every level.  Note, indention is the visual
appearance of the source on the page, while tabbing is replacing a
series of up to 8 spaces from a tab character.
<p>
The closest set of parameters for the Linux <b>indent</b> program
that will produce reasonably indented code are:
<pre>
-nbad -bap -bbo -nbc -br -brs -c36 -cd36 -ncdb -ce -ci3 -cli0
-cp36 -d0 -di1 -ndj -nfc1 -nfca -hnl -i3 -ip0 -l85 -lp -npcs
-nprs -npsl -saf -sai -saw -nsob -nss -nbc -ncs -nbfda
</pre>
You can put the above in your .indent.pro file, and then
just invoke indent on your file.  However, be warned. This
does not produce perfect indenting, and it will mess up C++
class statements pretty badly.
<p>
Braces are required in all
if statements (missing in some very old code). To avoid generating
too many lines, the first brace appears on the first line (e.g. of
an if), and the closing brace is on a line by itself. E.g.
<pre>
   if (abc) {
      some_code;
   }
</pre>
Just follow the convention in the code.  Originally I indented case
clauses under a switch(), but now I prefer non-indented cases. 
<pre>
   switch (code) {
   case 'A':
      do something
      break;
   case 'B':
      again();
      break;
   default:
      break;
   }
</pre>
Avoid using  // style comments except for temporary code or 
turning off debug code. Standard C
comments are preferred (this also keeps the code closer to C).
<p>
Attempt to keep all lines less than 85 characters long so
that the whole line of code is readable at one time.  This
is not a rigid requirement.
<p>
Always put a brief description at the top of any new file
created describing what it does and including your name
and the date it was first written. Please don't forget any
Copyrights and acknowledgments if it isn't 100% your
code. Also, include the Bacula copyright notice that
is in <b>src/c</b>.
<p>
In general you should have two includes at the top of the
file. One is <b>#include "bacula.h"</b> and the second is
an include for the particular directory the code is in, for
example, <b>#include "dird.h"</b>.  Sometimes additional
includes are needed, but this should be rare.  
<p>
In general (except for self-contained packages), prototypes
should all be put in <b>protos.h</b> in each directory.
<p>
Always put space around assignment and comparison operators.
<pre>
   a = 1;
   if (b &gt;= 2) {
     cleanup();
   }
</pre>
but your can compress things in a <b>for</b> statement:
<pre>
   for (i=0; i < del.num_ids; i++) {
    ...
</pre>
<p>
Don't overuse the inline if (?:). A full <b>if</b> is preferred,
except in a print statement, e.g.:
<pre>
   if (ua->verbose && del.num_del != 0) {
      bsendmsg(ua, _("Pruned %d %s on Volume %s from catalog.\n"), del.num_del,
         del.num_del == 1 ? "Job" : "Jobs", mr->VolumeName);
   }
</pre>
<p>
Leave a certain amount of debug code (Dmsg) in code you submit,
so that future problems can be identified. This is particularly
true for complicated code likely to break. However, try to
keep the debug code to a minimum to avoid bloating the program
and above all to keep the code readable.
<p>
Please keep the same style in all new code you develop.  If
you include code previously written, you have the option of
leaving it with the old indenting or re-indenting it.  If the
old code is indented with 8 spaces, then please re-indent it
to Bacula standards.
<p>
If you are using <b>vim</b>, simply set your tabstop to 8 and
your shiftwidth to 3.

<h3>Tabbing</h3>
Tabbing (inserting the tab character in place of spaces) is as
normal on all Unix systems -- a tab is converted space up to
the next column multiple of 8.
My editor converts strings of spaces to tabs automatically -- this
results in significant compression of the files.  Thus, you can
remove tabs by replacing them with spaces if you wish.  Please
don't confuse tabbing (use of tab characters) with indenting  
(visual alignment of the code).

<h3>Don'ts</h3>
Please don't use:
<pre>
strcpy()
strcat()
strncpy()
strncat();
sprintf()
snprintf()
</pre>
They are system dependent and un-safe.
These should be replaced by the Bacula safe equivalents:
<pre>
char *bstrncpy(char *dest, char *source, int dest_size);
char *bstrncat(char *dest, char *source, int dest_size);
int bsnprintf(char *buf, int32_t buf_len, const char *fmt, ...);
int bvsnprintf(char *str, int32_t size, const char  *format, va_list ap);
</pre>
See src/lib/bsys.c for more details on these routines.  
<p>
Don't use the <b>%lld</b> or the <b>%q</b> printf format editing types to edit
64 bit integers -- they are not portable.  Instead, use <b>%s</b> with
<b>edit_uint64()</b>. For example:
<pre>
   char buf[100];
   uint64_t num = something;
   char ed1[50];

   bsnprintf(buf, sizeof(buf), &quot;Num=%s\n&quot;, edit_uint64(num, ed1));
</pre>
The edit buffer <b>ed1</b> must be at least 27 bytes long to avoid
overflow.  See src/lib/edit.c for more details. If you look at the code,
don't start screaming that I use <b>lld</b>.  I actually use subtle trick taught to
me by John Walker. The <b>lld</b> that 
appears in the editing routine is actually <b>#define</b> to a what is
needed on your OS (usually &quot;lld&quot; or &quot;q&quot;) and is defined in
autoconf/configure.in for each OS.  C string concatenation causes
the appropriate string to be concatenated to the &quot;%&quot;.
<p>
Also please don't use the STL or Templates or any complicated C++ code.

<h3>Message Classes</h3>
Currently, there are four classes of messages: Debug, Error, Job, and
Memory.

<h3>Debug Messages</h3>
Debug messages are designed to be turned on at a specified debug
level and are always sent to STDOUT. There are designed to only be
used in the development debug process. They are coded as:
<p class="tty">DmsgN(level, message, arg1, ...)</p>
where the N is a number indicating how many arguments are to be
substituted into the message (i.e. it is a count of the number
arguments you have in your message -- generally the number of percent signs (%)).
<b>level</b> is the debug level at which you wish
the message to be printed. message is the debug message to be printed,
and arg1, ... are the arguments to be substituted. Since not all
compilers support #defines with varargs, you must explicitly specify
how many arguments you have. 
<p>When the debug message is printed, it will automatically be prefixed
by the name of the daemon which is running, the filename where the
Dmsg is, and the line number within the file. 
<p>Some actual examples are:
<p class="tty">Dmsg2(20, &quot;MD5len=%d MD5=%s\n&quot;, strlen(buf), buf);</p>
<p class="tty">Dmsg1(9, &quot;Created client %s record\n&quot;, client->hdr.name);</p>

<h3>Error Messages</h3>
Error messages are messages that are related to the daemon as
a whole rather than a particular job. For example, an out of
memory condition my generate an error message. They are coded as:
<p class="tty">EmsgN(error-code, level, message, arg1, ...)</p>
As with debug messages, you must explicitly code the of arguments
to be substituted in the message. error-code indicates the severity
or class of error, and it may be one of the following:
<p>
<table cellspacing="2" cellpadding="4" border="0">
<tr>
    <td><b>M_ABORT</b></td>    <td>Causes the daemon to immediately abort. This should be used only 
                                       in extreme cases. It attempts to produce a
                                       traceback.
</tr>
<tr>
    <td><b>M_ERROR_TERM</b></td>    <td>Causes the daemon to immediately terminate. This should be used only 
                                       in extreme cases. It does not produce a
                                       traceback.
</tr>

<tr>
    <td><b>M_FATAL</b></td>    <td>Causes the daemon to terminate the current job, but the daemon keeps running</td>
</tr>
<tr>
    <td><b>M_ERROR</b></td>     <td>Reports the error. The daemon and the job continue running</td>
</tr>
<tr>
    <td><b>M_WARNING</b></td>     <td>Reports an warning message. The daemon and the job continue running</td>
</tr>
<tr>
    <td><b>M_INFO</b></td>     <td>Reports an informational message.</td>
</tr>
</table>

<p>There are other error message classes, but they are in a
state of being redesigned or deprecated, so please do not use them.
Some actual examples are:
<p class="tty">Emsg1(M_ABORT, 0, &quot;Cannot create message thread: %s\n&quot;, strerror(status));</p>
<p class="tty">Emsg3(M_WARNING, 0, &quot;Connect to File daemon %s at %s:%d failed. Retrying ...\n&quot;,
            client-&gt;hdr.name, client-&gt;address, client-&gt;port);</p>
<p class="tty">  Emsg3(M_FATAL, 0, &quot;bdird&lt;filed: bad response from Filed to %s command: %d %s\n&quot;,
      cmd, n, strerror(errno));</p>

<h3>Job Messages</h3>
Job messages are messages that pertain to a particular job such as a file that
could not be saved, or the number of files and bytes that were saved.  

<h3>Memory Messages</h3>
Memory messages are messages that are edited into a memory buffer. Generally
they are used in low level routines such as the low level device file dev.c
in the Storage daemon or in the low level Catalog routines. These routines
do not generally have access to the Job Control Record and so they
return error messages reformatted in a memory buffer. Mmsg() is the way
to do this.

<page_prev name="Internal Component Designs" url="internals.html">
<page_next name="Porting Notes" url="porting.html">
