You want to help?  Excellent!
-----------------------------
Barry currently only supports about 3 database record formats.  A
database dump reveals over 50 different databases types, and you can help
code these.

Use the following command to dump the Database Database to stdout:

	./btool -t

Pick a database you are interested in, and run the following command to
dump the protocol to stdout:

	./btool -v -d "Memos"


Protocol Documentation:
-----------------------
Once you have the raw protocol dump, you need to reverse engineer it.
There are a number of places you can find documentation on this.

The original source that I started with was the Cassis project docs.
You can find them here:

	http://off.net/cassis/protocol-description.html

While these docs are quite helpful, they only cover the serial protocol.

The USB protocol is more data-block based.  There are no checksums, since
USB provides a reliable data transfer.  The common protocol headers are
struct based, until you get to the variable sized data.  (This is also
struct based, but requires pointer arithmetic.)

See the following Barry source file for protocol structure information:

	src/protostructs.h


Protostructs.h
--------------
These structures are all low level in nature, and MUST be packed.  You
will notice that every struct in this header is flagged with the gcc
extension "__attribute__ ((packed))", which makes it useful to apply
structure to raw protocol data.

You can build / deconstruct a protocol packet in this manner:

Imagine you have a packet of raw data in the following buffer:

	char buff[4096];

You can apply the Packet struct to it directly:

	struct Packet *packet = (struct Packet *) buff;

Looking at the structure, this gives you a lot of fixed-position data:



///////////////////////////////////////////////////////////////////////////////
// Main packet struct

struct Packet
{
        uint16_t        socket;         // socket ID... 0 is always there
        uint16_t        size;           // total size of data packet
        uint8_t         command;

        union PacketData
        {

                SocketCommand           socket;
                SequenceCommand         sequence;
                ModeSelectCommand       mode;
                DBAccess                db;
                uint8_t                 raw[1];

        } __attribute__ ((packed)) u;
} __attribute__ ((packed));



In the case of database records, you are interested in the DBAccess union
member:




///////////////////////////////////////////////////////////////////////////////
// Database access command structure

// even fragmented packets have a tableCmd
struct DBAccess
{
        uint8_t         tableCmd;

        union DBData
        {
                DBCommand               command;
                DBResponse              response;
                CommandTableField       table[1];
                uint8_t                 return_code;
                uint8_t                 fragment[1];

        } __attribute__ ((packed)) u;
} __attribute__ ((packed));



When downloading database records, they all arrive in a DBResponse packet.

struct DBResponse
{
        uint8_t         operation;

        union Parameters
        {

                DBR_OldTaggedRecord     tagged;
                DBR_OldDBDBRecord       old_dbdb;
                DBR_DBDBRecord          dbdb;

        } __attribute__ ((packed)) u;

} __attribute__ ((packed));


Depending on the command you use, the format of the record will change.
Also, newer versions of Blackberries have new "GET" commands that
return a slightly different record header.  Barry currenly uses
the "old" command, which is supported by all USB Blackberries tested
so far, so it has the greatest compatibility.

Some database records are tagged with a unique ID.  These databases
include the Contact records, Service Book records, and Calendar
records.  This allows for intelligent syncing, as Barry can reference,
update, and delete records individually instead of a group.

struct DBR_OldTaggedRecord
{
        uint8_t         unknown;
        uint16_t        index;
        uint32_t        uniqueId;
        uint8_t         unknown2;

        union TaggedData
        {
                CommonField     field[1];
        } __attribute__ ((packed)) u;
} __attribute__ ((packed));



This is enough information to get you to the data you need to reverse
engineer.  That data will be located at:

	packet->u.db.u.response.u.tagged.u.field[0]


So, taking an Address Book record as an example:

    00000000: 06 00 37 00 40 03 44 00 08 00 80 82 b1 22 00 06  ..7.@.D......"..
    00000010: 00 20 43 68 72 69 73 00 05 00 20 46 72 65 79 00  . Chris... Frey.
    00000020: 14 00 01 63 64 66 72 65 79 40 6e 65 74 64 69 72  ...cdfrey@netdir
    00000030: 65 63 74 2e 63 61 00                             ect.ca.

We now know:

	06 00		- packet->socket
	37 00		- packet->size
	40		- packet->command
	03		- packet->u.db.tableCmd
	44		- packet->u.db.u.response.operation
	00		- packet->u.db.u.response.u.tagged.unknown
	08 00		- packet->u.db.u.response.u.tagged.index
	80 82 b1 22	- packet->u.db.u.response.u.tagged.uniqueId
	00		- packet->u.db.u.response.u.tagged.unknown2
	06 00		- packet->u.db.u.response.u.tagged.u.field[0].size
	20		- packet->u.db.u.response.u.tagged.u.field[0].type
	...             - continued data from sequential CommonField packets

Note that CommonField packets are variable size, so you can only use
field[0] the first time.  You'll need to do pointer arithmetic to
calculate the start of the next CommonField.

This is a common theme in the database records.  There is a static header
(eg. DBR_OldTaggedRecord) followed by a number of fields, which each contains
a common field header (eg. CommonField), followed by a variable sized block
of data (CommonField's u.raw field).  The next field starts size bytes later,
and so on.



This should give you enough information to begin hacking on additional
database records.  Once you have a "Record" header struct, you are ready
to begin parsing into an API class.




Record API Classes
------------------
Record API Classes are intended to be consistent, C++-friendly structures
for storing the parsed data.  You can find them in:

	src/record.h

These classes may be broken out into multiple files someday.

The requirements of these classes are as follows:

	- no virtual functions
	- must be valid STL container items
	- generally contain all public data
	- provide member functions for:
		- dumping to ostream
		- parsing downloaded data blocks from the BlackBerry
		- building data blocks for the BlackBerry for uploads
	- must have absolutely no low level protocol contamination

The final point deserves some commentary.  When using gcc-specific extensions
like __attribute__, and when using pointer arithmetic and pointer casts
across types, as is necessary in some of the low level parsing code,
special compiler flags are required.

Specifically, g++ requires -fno-strict-aliasing to stop it from optimizing
away some of the pointer casts and data accesses.  This is not desireable
in an end-user application, so care must be taken to hide all low level
parsing in the library, while exposing the record class structures and
member function APIs to the application.

Therefore src/record.h (or any new record header you add) must not include
src/protostructs.h.  Your .cc code may include protostructs.h, but not
the header, which will be used by the application.


Good luck, and happy hacking!

October 2006

