<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN">
<article lang="en">
  <articleinfo>
    <title>OBDA Access HOWTO</title>
   <author>
        <firstname>Aaron</firstname>
        <surname>Mackey</surname>
        <affiliation>
                <orgname>
                        <ulink url="http://www.virginia.edu">University of Virginia</ulink>
                </orgname>
                <address>
                        <email>amackey@virginia.edu</email>
                </address>
        </affiliation>
    </author>
    <author>
      <!-- one of (CONTRIB AUTHORBLURB AFFILIATION OTHERNAME LINEAGE SURNAME FIRSTNAME HONORIFIC) -->
      <firstname>Brian</firstname>
      <surname>Osborne</surname>
      <authorblurb>
	<para>Bioperl contributor</para>
      </authorblurb>
      <affiliation>
	<orgname><ulink url="http://www.cognia.com">Cognia Corporation</ulink></orgname>
	<address>
	  <email>brian-at-cognia.com</email>
        </address>
      </affiliation>
    </author>
    <author>
      <!-- one of (CONTRIB AUTHORBLURB AFFILIATION OTHERNAME LINEAGE SURNAME FIRSTNAME HONORIFIC) -->
      <firstname>Peter</firstname>
      <surname>Schattner</surname>
      <authorblurb>
	<para>Bioperl contributor</para>
      </authorblurb>
      <affiliation>
	<orgname><ulink url="http://www.ucsc.edu">UCSC</ulink></orgname>
	<address>
	  <email>schattner@alum.mit.edu</email>
        </address>
      </affiliation>
    </author>    
    <author>
      <!-- one of (CONTRIB AUTHORBLURB AFFILIATION OTHERNAME LINEAGE SURNAME FIRSTNAME HONORIFIC) -->
      <firstname>Heikki</firstname>
      <surname>Lehvaslaiho</surname>
      <authorblurb>
	<para>Bioperl contributor</para>
      </authorblurb>
      <affiliation>
	<orgname><ulink url="http://ebi.ac.uk">EBI</ulink></orgname>
	<address>
	  <email>heikki@ebi.ac.uk</email>
        </address>
      </affiliation>
    </author>
    <author>
        <firstname>Lincoln</firstname>
        <surname>Stein</surname>
        <affiliation>
                <orgname>
                        <ulink url="http://www.cshl.org">Cold Spring Harbor Laboratory</ulink>
                </orgname>
                <address>
                        <email>lstein@cshl.org</email>
                </address>
        </affiliation>
</author>
 <pubdate>2002-07-14</pubdate>
    <revhistory>
      <revision>
	<revnumber>1.1</revnumber>
	<date>2003/02/20</date>
	<authorinitials>lstein</authorinitials>
	<revremark>added some OBDA documentation plus flat file index load script</revremark>
      </revision>
      <revision>
	<revnumber>1.2</revnumber>
	<date>2003/02/20</date>
	<authorinitials>lstein</authorinitials>
	<revremark>check in reasonable howtos for database access and flat databases</revremark>
      </revision>
      <revision>
	<revnumber>1.3</revnumber>
	<date>2003/02/20</date>
	<authorinitials>lstein</authorinitials>
	<revremark>added new heading to biodatabase access doc for Heikki to fill in</revremark>
      </revision>
      <revision>
	<revnumber>1.4</revnumber>
	<date>2003/02/20</date>
	<authorinitials>heikki</authorinitials>
	<revremark>added explanation for biogetseq</revremark>
      </revision>
      <revision>
	<revnumber>1.5</revnumber>
	<date>2003/03/01</date>
	<authorinitials>peters</authorinitials>
	<revremark>documentation updates</revremark>
      </revision>
      <revision>
	<revnumber>1.6</revnumber>
	<date>2003/03/12</date>
	<authorinitials>amackey</authorinitials>
	<revremark>documentation updates</revremark>
      </revision>
      <revision>
	<revnumber>1.7</revnumber>
	<date>2003/06/10</date>
	<authorinitials>bosborne</authorinitials>
	<revremark>convert to XML/SGML, corrections</revremark>
      </revision>
    </revhistory>
    <legalnotice>
      <para>This document is copyright Lincoln Stein, 2002.  For
	reproduction other than personal use please contact lstein at cshl.org
      </para>      
    </legalnotice>
    
    <abstract>
      <para>
	This is a HOWTO written in DocBook (SGML) that explains how to set up 
      and use the Open Biological Database Access system. 
      </para>
    </abstract>
  </articleinfo>
  
  <section id="introduction">
    <title>Introduction</title>
    <para>
Importing sequences with annotations is a central part of
most bioinformatics tasks.  Bioperl supports importing sequences from
indexed flat-files, local relational databases and remote (internet)
databases. Previously, separate programming syntax was required for
each of these types of data access (see for example Section III.1 of
the Bioperl tutorial). In addition, if one wanted to change one's mode
of sequence-data acquisition (for example, by implementing a local
relational database version of Genbank when previously the data had
been stored in an indexed flat-file) one had to rewrite all of the
data-access subroutines in one's application code.
    </para>
    <para>
The Open Biological Database Access (OBDA) System was designed so that
one could use the same application code to access data from all three
of the database types by simply changing a few lines in a
configuration file. This makes application code more portable and
easier to maintain. This document shows how to set up the required
OBDA-registry configuration file and how to access data from the
databases referred to in the configuration file using a perl script as
well as from the command line.
    </para>
    <para>
Note: accessing data via the OBDA system is optional in Bioperl.  It
is still possible (though probably not advantageous) to access
sequence data via the old database-format-specific modules such as 
Bio::Index::Fasta.
    </para>
  </section>
  <section id="registry">
    <title>Using the OBDA Registry System</title>
    <para>
The OBDA BioDirectory Registry is a platform-independent system for 
specifying how BioPerl programs find sequence databases.  It uses a 
site-wide configuration file, known as the registry, which defines 
one or more databases and the access methods to use to access them.
    </para>
    <para>
For instance, you might start out by accessing sequence data over the
web, and later decide to install a locally mirrored copy of GenBank.
By changing one line in the registry file, all
Bio{Perl,Java,Python,Ruby} applications will start using the mirrored
local copy automatically - no source code changes are necessary.
    </para>
  </section>

  <section id="installation">
    <title>Installing the Registry File</title>
    <para>
The registry file should be named seqdatabase.ini.  By default, it
should be installed in one of the following locations:
    <programlisting>
   $HOME/.bioinformatics/seqdatabase.ini
   /etc/bioinformatics/seqdatabase.ini
    </programlisting>
The Bio{Perl,Java,Python,Ruby} registry-handling code will initialize
itself from the registry file located in the home directory first,
followed by the system-wide default in /etc. ActiveState Perl users
should make sure to set the $HOME variable, otherwise the
seqdatabase.ini file will not be found. Unix users need not do this
since the code will use the getpwuid() method.    
    </para>
    <para>
If a local registry file cannot be found, the registry-handling code
will take its configuration from the file located at this URL:
    <programlisting>
   http://www.open-bio.org/registry/seqdatabase.ini
    </programlisting>    
    </para>
  </section>
  
  <section id="modifying">
    <title>Modifying the Search Path</title>
    <para>
The registry file search path can be modified by setting the
environment variable OBDA_SEARCH_PATH.  This variable is a
      semicolon-delimited string of directories and URLs, for example:
    <programlisting>
 OBDA_SEARCH_PATH=/home/lstein/;http://foo.org/
    </programlisting>
The search order proceeds from left to right.  The first file or URL
that is found ends the search.
    </para>    
    <para>
Warning! Note that the fact that the search path is for an entire file
(seqdatabase.ini) rather than for single entry (e.g. 'genbank') means
that you have to copy any default values you want to keep from the
(old) default configuration file to your new configuration file. 
    </para>
    <para>
For example, say you have been using biofetch with the default
configuration file http://www.open-bio.org/registry/seqdatabase.ini
for all your sequence-data retrieval.  If you now install a local copy
of genbank, your local seqdatabase.ini must not only have a section
indicating that the genbank copy is local but it must have sections
configuring the web access for all the other databases you use, since 
http://www.open-bio.org/registry/seqdatabase.ini will no longer be
found in a registry-file search.
    </para>
  </section>
  <section id="format">
    <title>Format of the Registry File</title>
    <para>
The registry file is a simple text file, as shown in the following
example:
     <programlisting>
 VERSION=1.00

 [embl]
 protocol=biofetch
 location=http://www.ebi.ac.uk/cgi-bin/dbfetch
 dbname=embl

 [swissprot]
 protocol=biofetch
 location=http://www.ebi.ac.uk/cgi-bin/dbfetch
 dbname=swall
    </programlisting>
The first line is the registry format version number in the format
VERSION=X.XX.  The current version is 1.00.
    </para>
    <para>
The rest of the file is composed of simple sections, formatted as:
    <programlisting>
  [database-name]
  tag=value
  tag=value

  [database-name]
  tag=value
  tag=value
    </programlisting>
Each section starts with a symbolic database service name enclosed in
square brackets.  Service names are case insensitive.  The remainder
of the section is followed by a series of tag=value pairs that
configure access to the service.
    </para>
    <para>
Database-name sections can be repeated, in which case the client should
try each service in turn from top to bottom.
    </para>
    <para>
The options under each section must have two non-optional tag=value
lines being
     <programlisting>
  protocol="protocol-type"
  location="location-string"
     </programlisting>
    </para>
  </section>
  <section id="protocol">
    <title>The Protocol Tag</title>
    <para>
The protocol tag species what access mode to use.  Currently it can be
one of:
    <programlisting>
  flat
  biofetch
  biosql
    </programlisting>
"flat" is used to fetch sequences from local flat files that have been
indexed using BerkeleyDB or binary search indexing.
    </para>
    <para>
"biofetch" is used to fetch sequences from web-based databses.  Due to
restrictions on the use of these databases, this is recommended only
for lightweight applications.
    </para>
    <para>
"biosql" fetches sequences from BioSQL databases.  To use this module
you will need to have an instantiated relational database conforming
to the BioSQL schema, and install the bioperl-db distribution.
    </para>
  </section>
  <section id="location">
  <title>The Location Tag</title>
  <para>
The location tag tells the bioperl sequence fetching code where the
database is located.  Its interpretation depends on the protocol
chosen.  For example, it might be a directory on the local file
system, or a remote URL.  See below for protocol-specific details.
  </para>
  </section>
  <section id="others">
  <title>Other Tags</title>
    <para>
Any number of additional tag values are allowed.  The number and
nature of these tags depends on the access protocol selected.  Some
protocols require no additional tags, whereas others will require
several.
    </para>
    <table>
      <title>OBDA Protocols</title>
      <tgroup cols="3">
        <thead>
          <row>
            <entry>Protocol</entry>
            <entry>Tag</entry>
            <entry>Description</entry>
          </row>
        </thead>
        <tbody>
        <row>
 <entry>flat</entry><entry>location</entry><entry>Directory in which
 the index is stored. The "config.dat" file generated during indexing must be found in this location</entry>
 </row>
 <row>
 <entry>flat</entry><entry>dbname</entry><entry>Name of database</entry>
 </row>
 <row>
 <entry>biofetch</entry><entry>location</entry><entry>Base URL for the
 web service. Currently the only compatible biofetch service is http://www.ebi.ac.uk/cgi-bin/dbfetch</entry>
 </row>
 <row>>
 <entry>biofetch</entry><entry>dbname</entry><entry>Name of the
 database.  Currently recognized values are "embl" (sequence and
 protein), "swall" (SwissProt + TREMBL), and "refseq" (NCBI RefSeq entries)</entry>
 </row>
 <row>
 <entry>biosql</entry><entry>location</entry><entry>host:port</entry>
 </row>
 <row>
 <entry>biosql</entry><entry>dbname</entry><entry>database name</entry>
 </row>
 <row>
 <entry>biosql</entry><entry>driver</entry><entry>mysql|postgres|oracle|sybase|sqlserver|access|csv|informix|odbc|rdb</entry>
 </row>
 <row>
 <entry>biosql</entry><entry>user</entry><entry>username</entry>
 </row>
 <row>
 <entry>biosql</entry><entry>passwd</entry><entry>password</entry>
 </row>
 <row>
 <entry>biosql</entry><entry>biodbname</entry><entry>biodatabase name</entry>
      </row>
      </tbody>
     </tgroup>
    </table>
  </section>
  <section id="local">
  <title>Installing Local Databases</title>
  <para>
If you are using the biofetch protocol, you're all set.  You can start
reading sequences immediately.  For the flat and biosql protocols, you
will need to create and initialize local databases.  See the following
documentation on how to do this:
  </para>	
  <para>
   flat protocol:    See FLAT-DATABASES-HOWTO.txt (in the doc/howto subdirectory)
  </para>
  <para>
   biosql protocol:  See BIOSQL-HOWTO.txt (this doc is still being developed)
  </para>
  </section>
  <section id="code">
  <title>Writing code to use the Registry</title>
  <para>
Once you've set up the OBDA registry file, accessing sequence data
from within a perl script is simple. The following example shows how;
note that nowhere in the script do you explicitly specify whether the
data is stored in a flat file, a local relational database or a
database on the internet.
  </para>
  <para>
To use the registry from a Perl script, use the following idiom:
  <programlisting>
    1 use Bio::DB::Registry;
    2 $registry = Bio::DB::Registry->new;
    3 $db = $registry->get_database('embl');
    4 $seq = $db->get_Seq_by_acc("J02231");
    5 print $seq->seq,"\n";
  </programlisting>
  </para>
  <para>
In lines 1 and 2, we bring in the Bio::DB::Registry module and create
a new Bio::DB::Registry object.  We then ask the registry to return a
database accessor for the symbolic data source "embl", which must be
defined in an [embl] section in the seqdatabase.ini registry file.
  </para>
  <para>
The returned accessor is a Bio::DB::RandomAccessI object (see the
appropriate manual page), which has just three methods:
  <programlisting>
   $db->get_Seq_by_id($id);
   $db->get_Seq_by_acc($acc);
   $db->get_Seq_by_version($versioned_acc);
  </programlisting>
  </para>
  <para>
These methods return Bio::Seq objects by searching for their primary
IDs, accession numbers and accession.version numbers respectively.
The returned objects have all the methods defined by Bio::Seq (see
the appropriate manual page).  In line 5, we call the sequence
object's seq() method to fetch and print out the DNA or protein
string.
  </para>
  </section>
  <section id="biogetseq">
  <title>Using biogetseq to Access Registry Databases</title>
  <para>
As a convenience, the Bioperl distribution includes a script
      'biogetseq' that enables one to have OBDA access to sequence
      data from the command line.
  </para>
  <para>
The script 'biogetseq' is located at the scripts/DB directory of the
bioperl distribution. Move or add it into your path to run it. You can get to help by running it with no arguments:
  <programlisting>
Usage: biogetseq --dbname embl --format embl --namespace acc id [ id ... ]*
       dbname defaults to embl
       format defaults to embl
       namespace defaults to 'acc' ['id', 'acc', 'version']
       rest of the arguments is a list of ids in the given namespace
  </programlisting>
  </para>
  <para>
If you have a set of ids you want to fetch from EMBL database, you
just give them as space separated parameters:
  <programlisting>
  % biogetseq J02231 A21530 A10516
  </programlisting>
  </para>
  <para>
The output is directed to STDOUT, so it can be redirected to a
file.  The options can be given in long "double hyphen" format or
abbreviated to one letter format:
  <programlisting>
  % biogetseq -f fasta -n acc J02231 A21530 A10516 > filed.seq
  </programlisting>
  </para>
  </section>
</article>
