\input texinfo   @c -*-texinfo-*-
@c
@c $Id: semantic.texi,v 1.68 2002/06/13 18:14:00 ponced Exp $
@c
@setfilename semantic.info
@settitle Semantic Bovinator: Parser generation for Emacs

@ifinfo
@format
START-INFO-DIR-ENTRY
* semantic: (semantic).       Semantic Parsing for Emacs
END-INFO-DIR-ENTRY
@end format
@end ifinfo

@titlepage
@sp 10
@center @titlefont{Semantic}
@vskip 0pt plus 1 fill
Copyright @copyright{} 1999, 2000, 2001, 2002 Eric M. Ludlam
@end titlepage

@node Top, Install, (dir), (dir)
@comment  node-name,  next,  previous,  up

Semantic is a program for Emacs which includes, at its core, a lexer,
and a compiler compiler (bovinator).  Additional tools include a
bnf->semantic table converter, example tables, and a speedbar tool.

The core utility is the ``semantic bovinator'' which has similar
behaviors as yacc or bison.  Since it is not designed to be as feature
rich as these tools, it uses the term ``bovine'' for cow, a lesser
cousin of the yak and bison.

To send bug reports, or participate in discussions about semantic,
use the mailing list cedet-semantic@@sourceforge.net via the URL:
@url{http://lists.sourceforge.net/lists/listinfo/cedet-semantic}

@menu
* Install::                     Installing semantic.
* Overview::                    Introduce basic concepts.
* Semantic Components::         Enumerate all semantic modules.
* Lexing::                      Setting up the lexer for your language.
* Bovinating::                  Setting up the parser for your language.
* BNF conversion::              Using the BNF converter to make tables.
* Compiling::                   Running the bovinator on a source file.
* Debugging::                   Debugging bovine tables
* Programming::                 How to program with a nonterminal stream.
* Current Context::             How to get the current code context.
* Tools::                       User Tools which use semantic.
* Index::
@end menu

@node Install, Overview, Top, Top
@chapter Installation

To install semantic, untar the distribution into a subdirectory, such as
@file{/usr/share/emacs/site-lisp/semantic-#.#}.  Next, add the following
lines into your individual @file{.emacs} file, or into
@file{site-lisp/site-start.el}.

@example
(setq semantic-load-turn-everything-on t)
(load-file "/path/to/semantic/semantic-load.el")
@end example

If you would like to turn individual tools on or off in your init
file, skip the first line.

@node Overview, Semantic Components, Install, Top
@chapter Overview

Semantic is a tool primarily for the Emacs-Lisp programmer.
However, it comes with ``applications'' that non-programmer might
find useful.
This chapter is mostly for the benefit of these non-programmers
as it gives brief descriptions of basic concepts such as
grammars, parsers, compiler-compilers, parse-tree, etc.

@cindex grammar
The grammar of a natural language defines rules by which valid phrases
and sentences can be composed using words, the fundamental units with
which all sentences are created.
@cindex context-free grammar
In a similar fashion, a ``context-free grammar'' defines the rules by which
programs can be composed using the fundamental units of the language,
i.e., numbers, symbols, punctuations, etc.
Context-free grammars are often specified in a well-known form called
@cindex Backus-Naur Form
@cindex BNF
Backus-Naur Form, BNF for short.
This is a systematic way of representing context-free grammars
such that programs can read files with grammars written in BNF
and generate code for ``parser'' of that language.
@cindex yacc
@cindex compiler-compiler
YACC (Yet Another Compiler Compiler) is one such program that has been
part of UNIX operating systems since the 1970's.
YACC is pronounced the same as ``yak'', the long-haired ox found in Asia.
The parser generated by YACC is usually a C program.
@cindex bison
@uref{http://www.gnu.org/software/bison/bison.html , Bison}
is also a ``compiler compiler'' that takes BNF grammars and produces
parsers in C language.
The difference between YACC and Bison is that Bison is
@cindex free software
@uref{http://www.gnu.org/philosophy/free-sw.html , free software}
and upward-compatible with YACC.
It also comes with an excellent manual.

Semantic is similar in spirit to YACC and Bison.
@cindex bovinator
Semantic, however, is referred to as a @dfn{bovinator} rather than
as a parser, because it is a lesser cousin of YACC and Bison.
It is lesser in that it does not perform a full parse
like YACC or Bison.
@cindex bovination
Instead, it @dfn{bovinates}.
``Bovination'' refers to partial parsing which
@cindex parse tree
creates @dfn{parse trees} of only the top most
expressions rather than parsing every nested expression.
This is sufficient for the purposes for which semantic was designed.
Semantic is meant to be used within Emacs for providing
editor-related features such as code browsers and translators rather
than for compiling which requires far more complex and complete parsers.
Semantic is not designed to be able to create full parse trees.

@cindex parser
One key benefit of semantic is that it creates parse trees
@cindex bovine tree
(perhaps the term @dfn{bovine tree} may be more accurate)
with the same structure regardless of the type of language involved.
Higher level applications written to work with bovine trees
will then work with any language for which the grammar is available.
For example, a code browser written today that supports C, C++, and
Java may work without any change on other languages that do not even
exist yet.
All one has to do is to write the BNF specification for the new language.
The rest of the work is done by semantic.
For certain languages, it is hard if not impossible to specify the syntax
of the language in BNF form, e.g.,
@uref{http://www.texinfo.org ,texinfo}
and other document oriented languages.
Semantic provides a parser for texinfo nevertheless.
Instead of BNF grammar, texinfo files are ``parsed'' using
@ref{Regexps,regular-expressions,regular-expressions,emacs}.

Semantic comes with grammars for these languages:

@itemize @bullet
@item C
@item Emacs-Lisp
@item java
@item makefile
@item scheme
@end itemize

Several tools employing semantic that provide user observable features
are listed in @ref{Tools} section.

@node Semantic Components, Lexing, Overview, Top
@chapter Semantic Components

This chapter gives an overview of major components
of semantic and how they interact with each other to perform its job.

The first step of parsing is to break up the input file into
its fundamental components.
This step is called lexing.
The output of the lexer is a list of tokens that make up the file.

@example
        syntax table, keywords list, and options
                         |
                         |
                         v
    input file  ---->  Lexer   ----> token stream
@end example

The next step is the parsing shown below.

@example
                    bovine table
                         |
                         v
    token stream --->  Parser  ----> parse tree
@end example

The end result, the parse tree, is created based on the ``bovine table'',
which is the internal representation of the BNF language grammar used by
semantic.

Semantic database provides caching of the parse trees by saving them
into files named @file{semantic.cache} automatically then loading them
when appropriate instead of re-parsing.  The reason for this is to save the
time it takes to parse a file which could take several seconds or more
for large files.

Finally, semantic provides an API for the Emacs-Lisp programmer to access the
information in the parse tree.

@node Lexing, Bovinating, Semantic Components, Top
@chapter Preparing your language for Lexing

In order to reduce a source file into a token list, it must first be
converted into a token stream.  Tokens are syntactic elements such as
whitespace, symbols, strings, lists, and punctuation.

The lexer uses the major-mode's syntax table for conversion.
@xref{Syntax Tables,,,elisp}.
As long as that is set up correctly (along with the important
@code{comment-start} and @code{comment-start-skip} variable) the lexer
should already work for your language.

The primary entry point of the lexer is the @dfn{semantic-flex} function
shown below.
Normally, you do not need to call this function.
It is usually called by @emph{semantic-bovinate-toplevel} for you.

@anchor{semantic-flex}
@defun semantic-flex start end &optional depth length
Using the syntax table, do something roughly equivalent to flex.
Semantically check between @var{START} and @var{END}.  Optional
argument @var{DEPTH} indicates at what level to scan over entire
lists.  The return value is a token stream.  Each element is a list
of the form (symbol start-expression .  end-expresssion). @var{END} does
not mark the end of the text scanned, only the end of the beginning of
text scanned.  Thus, if a string extends past @var{END}, the end of
the return token will be larger than @var{END}.  To truly restrict
scanning, use `narrow-to-region'.  The last argument, @var{LENGTH}
specifies that @dfn{semantic-flex} should only return
@var{LENGTH} tokens.
@end defun

@menu
* Lexer Overview::
* Lexer Output::
* Lexer Options::
* Keywords::
* Keyword Properties::
@end menu

@node Lexer Overview, Lexer Output, Lexing, Lexing
@section Lexer Overview

Semantic lexer breaks up the content of an Emacs buffer into a list of
tokens.  This process is based mostly on regular expressions which in
turn depend on the syntax table of the buffer's major mode being setup
properly.
@xref{Major Modes,,,emacs}.
@xref{Syntax Tables,,,elisp}.
@xref{Regexps,,,emacs}.

Specifically, the following regular expressions which rely on syntax
tables are used:

@table @code
@item @code{\\s-}
whitespace characters
@item @code{\\sw}
word constituent
@item @code{\\s_}
symbol constituent
@item @code{\\s.}
punctuation character
@item @code{\\s<}
comment starter
@item @code{\\s>}
comment ender
@item @code{\\s\\}
escape character
@item @code{\\s)}
close parenthesis character
@item @code{\\s$}
paired delimiter
@item @code{\\s\"}
string quote
@item @code{\\s\'}
expression prefix
@end table

In addition, Emacs' built-in features such as
@code{comment-start-skip},
@code{forward-comment},
@code{forward-list},
and
@code{forward-sexp}
are employed.

@node Lexer Output, Lexer Options, Lexer Overview, Lexing
@section Lexer Output

The lexer, @ref{semantic-flex}, scans the content of a buffer and
returns a token list.
Let's illustrate this using this simple example.

@example
00: /*
01:  * Simple program to demonstrate semantic.
02:  */
03:
04: #include <stdio.h>
05:
06: int i_1;
07:
08: int
09: main(int argc, char** argv)
10: @{
11:     printf("Hello world.\n");
12: @}
@end example

Evaluating @code{(semantic-flex (point-min) (point-max))}
within the buffer with the code above returns the following token list.
The input line and string that produced each token is shown after
each semi-colon.

@example
((punctuation     52 .  53)     ; 04: #
 (INCLUDE         53 .  60)     ; 04: include
 (punctuation     61 .  62)     ; 04: <
 (symbol          62 .  67)     ; 04: stdio
 (punctuation     67 .  68)     ; 04: .
 (symbol          68 .  69)     ; 04: h
 (punctuation     69 .  70)     ; 04: >
 (INT             72 .  75)     ; 06: int
 (symbol          76 .  79)     ; 06: i_1
 (punctuation     79 .  80)     ; 06: ;
 (INT             82 .  85)     ; 08: int
 (symbol          86 .  90)     ; 08: main
 (semantic-list   90 . 113)     ; 08: (int argc, char** argv)
 (semantic-list  114 . 147)     ; 09-12: body of main function
 )
@end example

As shown above, the token list is a list of ``tokens''.
Each token in turn is a list of the form

@example
(TOKEN-TYPE BEGINNING-POSITION . ENDING-POSITION)
@end example

@noindent
where TOKEN-TYPE is a symbol, and the other two are integers indicating
the buffer position that delimit the token such that

@lisp
(buffer-substring BEGINNING-POSITION ENDING-POSITION)
@end lisp

@noindent
would return the string form of the token.

Note that one line (line 4 above) can produce seven tokens while
the whole body of the function produces a single token.
This is because the @var{depth} parameter of @code{semantic-flex} was
not specified.
Let's see the output when @var{depth} is set to 1.
Evaluate @code{(semantic-flex (point-min) (point-max) 1)} in the same buffer.
Note the third argument of @code{1}.

@example
((punctuation    52 .  53)     ; 04: #
 (INCLUDE        53 .  60)     ; 04: include
 (punctuation    61 .  62)     ; 04: <
 (symbol         62 .  67)     ; 04: stdio
 (punctuation    67 .  68)     ; 04: .
 (symbol         68 .  69)     ; 04: h
 (punctuation    69 .  70)     ; 04: >
 (INT            72 .  75)     ; 06: int
 (symbol         76 .  79)     ; 06: i_1
 (punctuation    79 .  80)     ; 06: ;
 (INT            82 .  85)     ; 08: int
 (symbol         86 .  90)     ; 08: main

 (open-paren     90 .  91)     ; 08: (
 (INT            91 .  94)     ; 08: int
 (symbol         95 .  99)     ; 08: argc
 (punctuation    99 . 100)     ; 08: ,
 (CHAR          101 . 105)     ; 08: char
 (punctuation   105 . 106)     ; 08: *
 (punctuation   106 . 107)     ; 08: *
 (symbol        108 . 112)     ; 08: argv
 (close-paren   112 . 113)     ; 08: )

 (open-paren    114 . 115)     ; 10: @{
 (symbol        120 . 126)     ; 11: printf
 (semantic-list 126 . 144)     ; 11: ("Hello world.\n")
 (punctuation   144 . 145)     ; 11: ;
 (close-paren   146 . 147)     ; 12: @}
 )
@end example

The @var{depth} parameter ``peeled away'' one more level of ``list''
delimited by matching parenthesis or braces.
The depth parameter can be specified to be any number.
However, the parser needs to be able to handle the extra tokens.

This is an interesting benefit of the lexer having the full
resources of Emacs at its disposal.
Skipping over matched parenthesis is achieved by simply calling
the built-in functions @code{forward-list} and @code{forward-sexp}.

All common token symbols are enumerated below.  Additional token
symbols aside from these can be generated by the lexer if user option
@var{semantic-flex-extensions} is set.  It is up to the user to add
matching extensions to the parser to deal with the lexer
extensions.  An example use of @var{semantic-flex-extensions} is in
@file{semantic-make.el} where @var{semantic-flex-extensions} is set to
the value of @var{semantic-flex-make-extensions} which may generate
@code{shell-command} tokens.

@anchor{Default syntactic tokens}
@subsection Default syntactic tokens if the lexer is not extended.
@table @code
@item bol
Empty string matching a beginning of line.
This token is produced only if the user set
@var{semantic-flex-enable-bol} to non-@code{nil}.
@item charquote
String sequences that match @code{\\s\\+}.
@item close-paren
Characters that match @code{\\s)}.
These are typically @code{)}, @code{@}}, @code{]}, etc.
@item comment
A comment chunk.
These token types are not produced by default.
They are produced only if the user set
@var{semantic-ignore-comments} to @code{nil}.
@item newline
Characters matching @code{\\s-*\\(\n\\|\\s>\\)}.
This token is produced only if the user set
@var{semantic-flex-enable-newlines} to
non-@code{nil}.
@item open-paren
Characters that match @code{\\s(}.
These are typically @code{(}, @code{@{}, @code{[}, etc.
Note that these are not usually generated unless the @var{depth}
argument to @ref{semantic-flex} is greater than 0.
@item punctuation
Characters matching @code{\\(\\s.\\|\\s$\\|\\s'\\)}.
@item semantic-list
String delimited by matching parenthesis, braces, etc. that
the lexer skipped over, because the @var{depth} parameter
to @ref{semantic-flex} was not high enough.
@item string
Quoted strings, i.e., string sequences that start and end with
characters matching @code{\\s\"}.
The lexer relies on @code{forward-sexp} to find the
matching end.
@item symbol
String sequences that match @code{\\(\\sw\\|\\s_\\)+}.
@item whitespace
Characters that match `\\s-+' regexp. 
This token is produced only if the user set
@var{semantic-flex-enable-whitespace} to non-@code{nil}.  If
@var{semantic-ignore-comments} is non-@code{nil} too comments are
considered as whitespaces.
@end table

@node Lexer Options, Keywords, Lexer Output, Lexing
@section Lexer Options

Although most lexer functions are called for you by other semantic
functions, there are ways for you to extend or customize the lexer.
Three variables shown below serve this purpose.

@defvar semantic-flex-unterminated-syntax-end-function
Function called when unterminated syntax is encountered.
This should be set to one function.  That function should take three
parameters.  The @var{SYNTAX}, or type of syntax which is unterminated.
@var{SYNTAX-START} where the broken syntax begins.
@var{FLEX-END} is where the lexical analysis was asked to end.
This function can be used for languages that can intelligently fix up
broken syntax, or the exit lexical analysis via @dfn{throw} or @dfn{signal}
when finding unterminated syntax.
@end defvar

@defvar semantic-flex-extensions
Buffer local extensions to the lexical analyzer.
This should contain an alist with a key of a regex and a data element of
a function.  The function should both move point, and return a lexical
token of the form:

@example
( TYPE START . END)
@end example

@code{nil} is a valid return value.
@var{TYPE} can be any type of symbol, as long as it doesn't occur as a
nonterminal in the language definition.
@end defvar

@defvar semantic-flex-syntax-modifications
Changes the syntax table for a given buffer.
These changes are active only while the buffer is being flexed.
This is a list where each element has the form
@example
(CHAR CLASS)
@end example

CHAR is the char passed to `modify-syntax-entry',
and CLASS is the string also passed to `modify-syntax-entry' to define
what syntax class CHAR has.

@example
(setq semantic-flex-syntax-modifications '((?. "_"))
@end example

This makes the period @asis{.} a symbol constituent.  This may be necessary if
filenames are prevalent, such as in Makefiles.
@end defvar

@defvar semantic-flex-enable-newlines
When flexing, report @code{'newlines} as syntactic elements.
Useful for languages where the newline is a special case terminator.
Only set this on a per mode basis, not globally.
@end defvar

@defvar semantic-flex-enable-whitespace
When flexing, report @code{'whitespace} as syntactic elements.
Useful for languages where the syntax is whitespace dependent.
Only set this on a per mode basis, not globally.
@end defvar

@defvar semantic-flex-enable-bol
When flexing, report beginning of lines as syntactic elements.
Useful for languages like python which are indentation sensitive.
Only set this on a per mode basis, not globally.
@end defvar

@defvar semantic-number-expression
Regular expression for matching a number.
If this value is @code{nil}, no number extraction is done during lex.
This expression tries to match C and Java like numbers.

@example
DECIMAL_LITERAL:
    [1-9][0-9]*
  ;
HEX_LITERAL:
    0[xX][0-9a-fA-F]+
  ;
OCTAL_LITERAL:
    0[0-7]*
  ;
INTEGER_LITERAL:
    <DECIMAL_LITERAL>[lL]?
  | <HEX_LITERAL>[lL]?
  | <OCTAL_LITERAL>[lL]?
  ;
EXPONENT:
    [eE][+-]?[09]+
  ;
FLOATING_POINT_LITERAL:
    [0-9]+[.][0-9]*<EXPONENT>?[fFdD]?
  | [.][0-9]+<EXPONENT>?[fFdD]?
  | [0-9]+<EXPONENT>[fFdD]?
  | [0-9]+<EXPONENT>?[fFdD]
  ;
@end example
@end defvar

@node Keywords, Keyword Properties, Lexer Options, Lexing
@section Keywords

Another important piece of the lexer is the keyword table (see
@ref{Settings}).  You language will want to set up a keyword table for
fast conversion of symbol strings to language terminals.

The keywords table can also be used to store additional information
about those keywords.  The following programming functions can be useful
when examining text in a language buffer.

@defun semantic-flex-keyword-p text
Return non-@code{nil} if @var{TEXT} is a keyword in the keyword table.
@end defun

@defun semantic-flex-keyword-put text property value
For keyword @var{TEXT}, set @var{PROPERTY} to @var{VALUE}.
@end defun

@defun semantic-token-put-no-side-effect token key value
For @var{TOKEN}, put the property @var{KEY} on it with @var{VALUE} without side effects.
If @var{VALUE} is @code{nil}, then remove the property from @var{TOKEN}.
All cons cells in the property list are replicated so that there
are no side effects if @var{TOKEN} is in shared lists.
@end defun

@defun semantic-flex-keyword-get text property
For keyword @var{TEXT}, get the value of @var{PROPERTY}.
@end defun

@defun semantic-flex-map-keywords fun &optional property
Call function @var{FUN} on every semantic keyword.
If optional @var{PROPERTY} is non-nil, call @var{FUN} only on every
keyword which has a @var{PROPERTY} value.
@var{FUN} receives a semantic keyword as argument.
@end defun

@defun semantic-flex-keywords &optional property
Return a list of semantic keywords.
If optional @var{PROPERTY} is non-nil, return only keywords which have
@var{PROPERTY} set.
@end defun

Keyword properties can be set up in a BNF file for ease of maintenance.
While examining the text in a language buffer, this can provide an easy
and quick way of storing details about text in the buffer.

@node Keyword Properties,  , Keywords, Lexing
@section Standard Keyword Properties

Add known properties here when they are known.

@node Bovinating, BNF conversion, Lexing, Top
@chapter Preparing a bovine table for your language

When converting a source file into a nonterminal token stream
(parse-tree) it is important to specify rules to accomplish this.  The
rules are stored in the buffer local variable
@code{semantic-toplevel-bovine-table}.

While it is certainly possible to write this table yourself, it is most
likely you will want to use the BNF converter (see @xref{BNF conversion}.)
This is an easier method for specifying your rules.  You will still need
to specify a variable in your language for the table, however.  A good
rule of thumb is to call it @code{language-toplevel-bovine-table} if it
part of the language, or @code{semantic-toplevel-language-bovine-table}
if you donate it to the semantic package.

When initializing a major-mode for your language, you will set the
variable @code{semantic-toplevel-bovine-table} to the contents of your
language table.  @code{semantic-toplevel-bovine-table} is always buffer
local.

Since it is important to know the format of the table when debugging ,
you should still attempt to understand the basics of the table.

Please see the documentation for the variable
@code{semantic-toplevel-bovine-table} for details on its format.

* add more doc here *

@node BNF conversion, Compiling, Bovinating, Top
@chapter Using the BNF converter to make bovine tables

The BNF converter takes a file in "Bovine Normal Form" which is similar
to "Backus-Naur Form".  If you have ever used yacc or bison, you will
find it similar.  The BNF form used by semantic, however, does not
include token precedence rules, and several other features needed to make
real parser generators.

It is important to have an Emacs Lisp file with a variable ready to take
the output of your table (see @xref{Bovinating}.)  Also, make sure that the
file @file{semantic-bnf.el} is loaded.  Give your language file the
extension @file{.bnf} and you are ready.

The comment character is @asis{#}.

When you want to test your file, use the keyboard shortcut @kbd{C-c C-c}
to parse the file, generate the variable, and load the new definition
in.  It will then use the settings specified above to determine what to
do.  Use the shortcut @kbd{C-c c} to do the same thing, but spend
extra time indenting the table nicely.

Make sure that you create the variable specified in the
@code{%parsetable} token before trying to convert the BNF file.  A
simple definition like this is sufficient.

@example
(defvar semantic-toplevel-lang-bovine-table
   nil
   "Table for use with semantic for parsing LANG.")
@end example

If you use tokens (created with the @code{%token} specifier), also
make sure you have a keyword table available, like this:

@example
(defvar semantic-lang-keyword-table
   nil
   "Table for use with semantic for keywords.")
@end example

Specify the name of the keyword table with the @code{%keywordtable}
specifier.

The BNF file has two sections.  The first is the settings section, and
the second is the language definition, or list of semantic rules.

@menu
* Settings::                    Setup for a language
* Rules::                       Create rules to parse a language
* Optional Lambda Expression::  Actions to take when a rule is matched
* Examples::                    Simple Samples
* Style Guide::                 What the tokens mean, and how to use them.

@end menu

@node Settings, Rules, BNF conversion, BNF conversion
@section Settings

A setting is a keyword starting with a @asis{%}.  (This syntax is taken
from yacc and bison. @xref{(bison)}.)

There are several settings that can be made in the settings section.
They are:

@deffn Setting %start <nonterminal>
Specify an alternative to @code{bovine-toplevel}.  (See below)
@end deffn

@deffn Setting %scopestart <nonterminal>
Specify an alternative to @code{bovine-inner-scope}.
@end deffn

@deffn Setting %outputfile <filename>
Required.  Specifies the file into which this files output is stored.
@end deffn

@deffn Setting %parsetable <lisp-variable-name>
Required.  Specifies a lisp variable into which the output is stored.
@end deffn

@deffn Setting %setupfunction <lisp-function-name>
Required.  Name of a function into which setup code is to be inserted.
@end deffn

@deffn Setting %keywordtable <lisp-variable-name>
Required if there are @code{%token} keywords.
Specifies a lisp variable into which the output of a keyword table is
stored.  This obarray is used to turn symbols into keywords when applicable.
@end deffn

@deffn Setting %token <name> "<text>"
Optional.  Specify a new token @var{NAME}.  This is added to a lexical
keyword list using @var{TEXT}.  The symbol is then converted into a new
lexical terminal.  This requires that the @code{%keywordtable} specified
variable is available in the file specified by @code{%outputfile}.
@end deffn

@deffn Setting %token <name> type "<text>"
Optional.  Specify a new token @var{NAME}.  It is made from an existing
lexical token of type @var{TYPE}.  @var{TEXT} is a string which will be
matched explicitly.  @var{NAME} can be used in match rules as though they were
flex tokens, but are converted back to @var{TYPE} "text" internally.
@end deffn

@deffn Setting %put <NAME> symbol <VALUE>
@deffnx Setting %put <NAME> ( symbol1 <VALUE1> symbol2 <VALUE2> ... )
@deffnx Setting %put ( <NAME1> <NAME2>...) symbol <VALUE>
Tokens created without a type are considered keywords, and placed in a
keyword table.  Use @code{%put} to apply properties to that keyword.
(see @ref{Lexing}).
@end deffn

@deffn Setting %languagemode <lisp-function-name>
@deffnx Setting %languagemode ( <lisp-function-name1> <lisp-function-name2> ... )
Optional.  Specifies the Emacs major mode associated with the language
being specified.  When the language is converted, all buffers of this
mode will get the new table installed.
@end deffn

@deffn Setting %quotemode backquote
Optional.  Specifies how symbol quoting is handled in the Optional
Lambda Expressions. (See below)
@end deffn

@deffn Setting {%( <lisp-expression> )%}
Specify setup code to be inserted into the @code{%setupfunction}.
It will be inserted between two specifier strings, or added to
the end of the function.
@end deffn

When working inside @code{%( ... )%} tokens, any lisp expression can be
entered which will be placed inside the setup function.  In general, you
probably want to set variables that tell Semantic and related tools how
the language works.

Here are some variables that control how different programs will work
with your language.

@defvar semantic-flex-depth
Default flexing depth.
This specifies how many lists to create tokens in.
@end defvar

@defvar semantic-number-expression
Regular expression for matching a number.
If this value is @code{nil}, no number extraction is done during lex.
Symbols which match this expression are returned as @code{number}
tokens instead of @code{symbol} tokens.

The default value for this variable should work in most languages.
@end defvar

@defvar semantic-flex-extensions
Buffer local extensions to the lexical analyzer.
This should contain an alist with a key of a regex and a data element of
a function.  The function should both move point, and return a lexical
token of the form:
@example
( TYPE START . END)
@end example

@code{nil} is also a valid return.
@var{TYPE} can be any type of symbol, as long as it doesn't occur as a
nonterminal in the language definition.
@end defvar

@defvar semantic-flex-syntax-modifications
Updates to the syntax table for this buffer.
These changes are active only while this file is being flexed.
This is a list where each element is of the form:
@example
(CHAR CLASS)
@end example
Where @var{CHAR} is the char passed to @dfn{modify-syntax-entry},
and @var{CLASS} is the string also passed to @dfn{modify-syntax-entry} to define
what class of syntax @var{CHAR} is.
@end defvar

@defvar semantic-flex-enable-newlines
When flexing, report @code{'newlines} as syntactic elements.
Useful for languages where the newline is a special case terminator.
Only set this on a per mode basis, not globally.
@end defvar

@defvar semantic-ignore-comments
Default comment handling.
@code{t} means to strip comments when flexing.  @code{Nil} means to keep comments
as part of the token stream.
@end defvar

@defvar semantic-symbol->name-assoc-list
Association between symbols returned, and a string.
The string is used to represent a group of objects of the given type.
It is sometimes useful for a language to use a different string
in place of the default, even though that language will still
return a symbol.  For example, Java return's includes, but the
string can be replaced with @code{Imports}.
@end defvar

@defvar semantic-case-fold
Value for @code{case-fold-search} when parsing.
@end defvar

@defvar semantic-expand-nonterminal
Function to call for each nonterminal production.
Return a list of non-terminals derived from the first argument, or @code{nil}
if it does not need to be expanded.
Languages with compound definitions should use this function to expand
from one compound symbol into several.  For example, in @var{C} the
definition
@example
int a, b;
@end example
is easily parsed into one token, but represents multiple variables.  A
functions should be written which takes this compound token and turns
it into two tokens, one for @var{A}, and the other for @var{B}.

Within the language definition (the @file{.bnf} sources), it is often
useful to set the NAME slot of a token with a list of items that
distinguish each element in the compound definition.

This list can then be detected by the function set in
@code{semantic-expand-nonterminal} to create multiple tokens.
This function has one additional duty of managing the overlays created
by semantic.  It is possible to use the single overlay in the compound
token for all your tokens, but this can pose problems identifying
all tokens covering a given definition.

Please see @file{semantic-java.el} for an example of managing overlays
when expanding a token into multiple definitions.
@end defvar

@defvar semantic-override-table
Buffer local semantic function overrides alist.
These overrides provide a hook for a `major-mode' to override specific
behaviors with respect to generated semantic toplevel nonterminals and
things that these non-terminals are useful for.
Each element must be of the form: (SYM . FUN)
where SYM is the symbol to override, and FUN is the function to
override it with.

Available override symbols:

@multitable {@code{abbreviate-nonterminal}} {(token & nosnarf)} {Find token in buffer}
@item SYMBOL                 @tab PARAMETERS        @tab DESCRIPTION
@item find-dependency        @tab (token)           @tab Find the dependency file
@item find-nonterminal       @tab (token & parent)  @tab Find token in buffer.
@item find-documentation     @tab (token & nosnarf) @tab Find doc comments.
@item abbreviate-nonterminal @tab (token & parent)  @tab Return summary string.
@item summarize-nonterminal  @tab (token & parent)  @tab Return summary string.
@item prototype-nonterminal  @tab (token)           @tab Return a prototype string.
@item concise-prototype-nonterminal'     @tab (tok & parent color) @tab Return a concise prototype string.
@item uml-abbreviate-nonterminal'        @tab (tok & parent color) @tab Return a UML standard abbreviation string.
@item uml-prototype-nonterminal'         @tab (tok & parent color) @tab Return a UML like prototype string.
@item uml-concise-prototype-nonterminal' @tab (tok & parent color) @tab Return a UML like concise prototype string.

@item prototype-file         @tab (buffer)          @tab Return a file in which prototypes are placed
@item nonterminal-children   @tab (token)           @tab Return first rate children. These are children which may contain overlays.
@item nonterminal-external-member-parent @tab (token) @tab Parent of TOKEN
@item nonterminal-external-member-p @tab (parent token) @tab Non nil if TOKEN has PARENT, but is not in PARENT.
@item nonterminal-external-member-children @tab (token & usedb) @tab Get all external children of TOKEN.
@item nonterminal-protection @tab (token & parent)  @tab Return protection as a symbol.
@item nonterminal-abstract   @tab (token & parent)  @tab Return if TOKEN is abstract.
@item nonterminal-leaf       @tab (token & parent)  @tab Return if TOKEN is leaf.
@item nonterminal-static     @tab (token & parent)  @tab Return if TOKEN is static.
@item beginning-of-context   @tab (& point)         @tab Move to the beginning of the
					                 current context.
@item end-of-context         @tab (& point)         @tab Move to the end of the
					                 current context.
@item up-context             @tab (& point)         @tab Move up one context level.
@item get-local-variables    @tab (& point)         @tab Get local variables.
@item get-all-local-variables@tab (& point)         @tab Get all local variables.
@item get-local-arguments    @tab (& point)         @tab Get arguments to this function.

@item end-of-command         @tab                   @tab Move to the end of the current
                                                         command
@item beginning-of-command   @tab                   @tab Move to the beginning of the
                                                         current command
@item ctxt-current-symbol    @tab (& point)         @tab List of related symbols.
@item ctxt-current-assignment@tab (& point)         @tab Variable being assigned to.
@item ctxt-current-function  @tab (& point)         @tab Function being called at point.
@item ctxt-current-argument  @tab (& point)         @tab The index to the argument of
                                                         the current function the cursor
                                                         is in.
@end multitable

Parameters mean:

@table @code
@item &
Following parameters are optional
@item buffer
The buffer in which a token was found.
@item token
The nonterminal token we are doing stuff with
@item parent
If a TOKEN is stripped (of positional information) then this will be the
parent token which should have positional information in it.
@end table

@end defvar

@defvar semantic-type-relation-separator-character
Character strings used to separation a parent/child relationship.
This list of strings are used for displaying or finding separators
in variable field dereferencing.  The first character will be used for
display.  In @var{C}, a type field is separated like this: ``type.field''
thus, the character is a ``.''.  In @var{C}, and additional value of ``->''
would be in the list, so that ``type->field'' could be found.
@end defvar


@defvar semantic-dependency-include-path
Defines the include path used when searching for files.
This should be a list of directories to search which is specific to
the file being included.
This variable can also be set to a single function.  If it is a
function, it will be called with one arguments, the file to find as a
string, and  it should return the full path to that file, or nil.
@end defvar

This configures Imenu to use semantic parsing.

@defvar imenu-create-index-function
The function to use for creating a buffer index.

It should be a function that takes no arguments and returns an index
of the current buffer as an alist.

Simple elements in the alist look like @samp{(INDEX-NAME . INDEX-POSITION)}.
Special elements look like @samp{(INDEX-NAME INDEX-POSITION FUNCTION ARGUMENTS...)}.
A nested sub-alist element looks like (INDEX-NAME SUB-ALIST).
The function @code{imenu--subalist-p} tests an element and returns t
if it is a sub-alist.

This function is called within a @code{save-excursion}.

The variable is buffer-local.
@end defvar


These are specific to the document tool.

@table @code
@item document-comment-start
Comment start string.
@item document-comment-line-prefix
Comment prefix string.  Used at the beginning of each line.
@item document-comment-end
Comment end string.
@end table

@node Rules, Optional Lambda Expression, Settings, BNF conversion
@section Rules

Writing the rules should be very similar to bison for basic syntax.
Each rule is of the form

@example
RESULT : MATCH1 (optional-lambda-expression)
       | MATCH2 (optional-lambda-expression)
       ;
@end example

@var{RESULT} is a non-terminal, or a token synthesized in your grammar.
@var{MATCH} is a list of elements that are to be matched if @var{RESULT}
is to be made.  The optional lambda expression is a list containing
simplified rules for concocting the parse tree.

In bison, each time an element of a @var{MATCH} is found, it is
"shifted" onto the parser stack.  (The stack of matched elements.)  When
all of @var{MATCH1}'s elements have been matched, it is "reduced" to
@var{RESULT}.  @xref{(bison)Algorithm}.

The first @var{RESULT} written into your language specification should
be @code{bovine-toplevel}, or the symbol specified with @code{%start}.
When starting a parse for a file, this is the default token iterated
over.  You can use any token you want in place of @code{bovine-toplevel}
if you specify what that nonterminal will be with a @code{%start} token
in the settings section.

@var{MATCH} is made up of symbols and strings.  A symbol such as
@code{foo} means that a syntactic token of type @code{foo} must be
matched.  A string in the mix means that the previous symbol must have
the additional constraint of exactly matching it.  Thus, the
combination:

@example
symbol "moose"
@end example

means that a symbol must first be encountered, and then it must
@code{string-match "moose"}.  Be especially careful to remember that the
string is a regular expression.  The code:

@example
punctuation "."
@end example

will match any punctuation.

For the above example in bison, a LEX rule would be used to create a new
token @var{MOOSE}.  In this case, the @var{MOOSE} token would appear.
For the bovinator, this task was mixed into the language definition to
simplify implementation, though Bison's technique is more efficient.

To make a symbol match explicitly for keywords, for example, you can use
the @code{%token} command in the settings section to create new symbols.

@example
%token MOOSE "moose"

find_a_moose: MOOSE
            ;
@end example

will match ``moose'' explicitly, unlike the previous example where moose
need only appear in the symbol.  This is because ``moose'' will be
converted to @var{MOOSE} in the lexical analysis stage.  Thus the symbol
@var{MOOSE} won't be available any other way.

If we specify our token in this way:

@example
%token MOOSE symbol "moose"

find_a_moose: MOOSE
            ;
@end example

then @code{MOOSE} will match the string "moose" explicitly, but it won't
do so at the lexical level, allowing use of the text "moose" in other
forms of regular expressions.

Non symbol tokens are also allowed.  For example:

@example
%token PERIOD punctuation "."

filename : symbol PERIOD symbol
         ;
@end example

will explicitly match one period when used in the above rule.

@xref{Default syntactic tokens}.


@node Optional Lambda Expression, Examples, Rules, BNF conversion
@section Optional Lambda Expressions

The OLE (Optional Lambda Expression) is converted into a bovine lambda
(see @xref{Bovinating}.) This lambda has special short-cuts to simplify
reading the Emacs BNF definition.  An OLE like this:

@example
( $1 )
@end example

results in a lambda return which consists entirely of the string
or object found by matching the first (zeroth) element of match.
An OLE like this:

@example
( ,(foo $1) )
@end example

executes `foo' on the first argument, and then splices its return
into the return list whereas:

@example
( (foo $1) )
@end example

executes foo, and that is placed in the return list.

Here are other things that can appear inline:
@table @code
@item $1
the first object matched.
@item ,$1
the first object spliced into the list (assuming it is a list from a
non-terminal)
@item '$1
the first object matched, placed in a list.  i.e. ( $1 )
@item foo
the symbol foo (exactly as displayed)
@item (foo)
a function call to foo which is stuck into the return list.
@item ,(foo)
a function call to foo which is spliced into the return list.
@item '(foo)
a function call to foo which is stuck into the return list in a list.
@item (EXPAND $1 nonterminal depth)
a list starting with EXPAND performs a recursive parse on the token
passed to it (represented by $1 above.)  The semantic list is a common
token to expand, as there are often interesting things in the list.
The @var{nonterminal} is a symbol in your table which the bovinator will
start with when parsing.  @var{nonterminal}'s definition is the same as
any other nonterminal.  @var{depth} should be at least 1 when
descending into a semantic list.
@item (EXPANDFULL $1 nonterminal depth)
is like EXPAND, except that the parser will iterate over
@var{nonterminal} until there are no more matches.  (The same way the
parser iterates over @code{bovine-toplevel}. This lets you have
much simpler rules in this specific case, and also lets you have
positional information in the returned tokens, and error skipping.
@item (ASSOC symbol1 value1 symbol2 value2 ... )
This is used for creating an association list.  Each @var{SYMBOL} is
included in the list if the associated @var{VALUE} is non-nil.  While
the items are all listed explicitly, the created structure is an
association list of the form:
@example
( ( symbol1 . value1) (symbol2 . value2) ... )
@end example
@end table

If the symbol @code{%quotemode backquote} is specified, then use
@code{,@@} to splice a list in, and @code{,} to evaluate the expression.
This lets you send @code{$1} as a symbol into a list instead of having
it expanded inline.

@node Examples, Style Guide , Optional Lambda Expression, BNF conversion
@section Examples

The rule:

@example
SYMBOL : symbol
@end example

is equivalent to

@example
SYMBOL : symbol
         ( $1 )
@end example

which, if it matched the string "A", would return

@example
( "A" )
@end example

If this rule were used like this:

@example
ASSIGN: SYMBOL punctuation "=" SYMBOL
        ( $1 $3 )
@end example

it would match "A=B", and return

@example
( ("A") ("B") )
@end example

The letters A and B come back in lists because SYMBOL is a nonterminal,
not an actual lexical element.

to get a better result with nonterminals, use @asis{,} to splice lists
in like this;

@example
ASSIGN: SYMBOL punctuation "=" SYMBOL
        ( ,$1 ,$3 )
@end example

which would return

@example
( "A" "B" )
@end example

@node Style Guide ,  , Examples, BNF conversion
@section Semantic Token Style Guide

In order for a generalized program using Semantic to work with
multiple languages, it is important to have a consistent meaning for
the contents of the tokens returned.  The variable
@code{semantic-toplevel-bovine-table} is documented with the complete
list of a tokens that a functional or OO language may use.  While any
given language is free to create their own tokens, such a language
definition would not produce a stream of tokens usable by a
generalized tool.

@section Minimum Requirements

In general, all tokens returned from a parser should be generated with
the following form:

@example
("NAME" type-symbol ... "DOCSTRING" PROPERTIES OVERLAY)
@end example

@var{NAME} and @var{type-symbol} are the only syntactic elements of a
nonterminal which are guaranteed to exist.  This means that a parser
which uses @code{nil} for either of these two slots, or some value
which is not type consistent is wrong.

@var{NAME} is also guaranteed to be a string.  This string represents
the name of the nonterminal, usually a named definition which the
language will use elsewhere as a reference to the syntactic element
found.

@var{type-symbol} is a symbol representing the type of the
nonterminal.  Valid @var{type-symbol}s can be anything, as long is it
is an Emacs Lisp symbol.

@var{DOCSTRING} is a required slot in the nonterminal, but can be
nil.  Some languages have the documentation saved as a comment
nearby.  In these cases, DOCSTRING is nil, and the function
`semantic-find-documentation'.

@var{PROPERTIES} is a slot generated by the semantic parser harness,
and need not be provided by a language author.  Programmatically access
nonterminal properties with @code{semantic-token-put} and
@code{semantic-token-get} to access properties.

@var{OVERLAY} represents positional information for this token.  It is
automatically generated by the semantic parser harness, and need not
be provided by the language author, unless they provide a nonterminal
expansion function via @code{semantic-expand-nonterminal}.

The @var{OVERLAY} property is accessed via several functions returning
the beginning, end, and buffer of a token.  Use these functions unless
the overlay is really needed (see @ref{Token Queries}).  Depending on the
overlay in a program can be dangerous because sometimes the overlay is
replaced with an integer pair
@example
[ START END ]
@end example
when the buffer the token belongs to is not in memory.  This happens
when a using has activated the Semantic Database @ref{semanticdb}.

@section Nonterminals for Functional Languages.

If a parser produces tokens for a functional language, then the
following token formats are available.

@table @asis
@item Variable
@itemx @code{("NAME" variable "TYPE" DEFAULT-VALUE EXTRA-SPEC}
@itemx @code{        "DOCSTRING" PROPERTIES OVERLAY)}
@var{TYPE} is a string representing the type of this variable.
@var{TYPE} can be @code{nil} for untyped languages.  Languages which
support variable declarations without a type (Such as C) should supply
a string representing the default type for that language.

@var{DEFAULT-VALUE} can be a string, or something pre-parsed and
language specific.  Hopefully this slot will be better defined in
future versions of Semantic.

@var{EXTRA-SPEC} are extra specifiers.  See below.


@item Function
@itemx @code{ ("NAME" function "TYPE" ( ARG-LIST ) EXTRA-SPEC}
@itemx @code{          "DOCSTRING" PROPERTIES OVERLAY)}
@var{TYPE} is a string representing the return type of this function
or method.  @var{type} can be @code{nil} for untyped languages, or for
procedures in languages which support functions with no return data.
See above for more.

@var{ARG-LIST} is a list of arguments passed to this function.
Each element in the arg list can be one of the following:
@table @asis
@item Semantic Token
A full semantic token with positional information.
@item A partial semantic token
Partial tokens may contain the @var{NAME} slot, @var{token-symbol},
and possibly a @var{TYPE}.
@item String
A string representing the name of the argument.  Common in untyped
languages.
@end table


@item Type Declaration
@itemx @code{ ("NAME" type "TYPE" ( PART-LIST ) ( PARENTS ) EXTRA-SPEC}
@itemx @code{          "DOCSTRING" PROPERTIES OVERLAY)}
@var{TYPE} a string representing the of the type, such as (in C)
``struct'', ``union'', ``enum'', ``typedef'', or ``class''.
The @var{TYPE} for a type token should not be nil, as even untyped
languages with structures have type types.

@var{PART-LIST} is the list of individual entries inside compound
types.  Structures, for example, can contain several fields which can
be represented as variables.  Valid entries in a @var{PART-LIST} are:
@table @asis
@item Semantic Token
A full semantic token with positional information.
@item A partial semantic token
Partial tokens may contain the @var{NAME} slot, @var{token-symbol},
and possibly a @var{TYPE}.
@item String
A string representing the name of the slot or field.  Common in untyped
languages.
@end table

@var{PARENTS} represents a list of parents of this type.  Parents are used
in two situations.
@table @asis
@item Inheritance
For types which inherit from other types of the same type-type (Such
as classes).
@item Aliases
For types which are aliases of other types, the parent type is the
type being aliased.  The Types' type is the command specifying that it
is an alias (Such as ``typedef'' in C or C++).
@end table

The structure of the @var{PARENTS} list is of this form:
@example
( EXPLICIT-PARENTS . INTERFACE-PARENTS)
@end example
@var{EXPLICIT-PARENTS} can be a single string (Just one parent) or a
list of parents (in a multiple inheritance situation.  It can also be
nil.

@var{INTERFACE-PARENTS} is a list of strings representing the names of
all INTERFACES, or abstract classes inherited from.  It can also be
nil.

This slot can be interesting because the form:
@example
( nil "string")
@end example
is a valid parent where there is no explicit parent, and only an
interface.


@item Include files
@itemx @code{("FILE" include SYSTEM "DOCSTRING" PROPERTIES OVERLAY)}
A statement which gets additional definitions from outside the current
file, such as an @code{#include} statement in C.
In this case, instead of @var{NAME}, a @var{FILE} is specified.
@var{FILE} can be a subset of the actual file to be loaded.

@var{SYSTEM} is true if this include is part of a set of system
includes.  This field isn't currently being used and may be
eliminated.


@item Package & Provide statements
@itemx @code{("NAME" package DETAIL "DOCSTRING" PROPERTIES OVERLAY)}
A statement which declares a given file is part of a package, such as
the Java @code{package} statement, or a @code{provide} in Emacs Lisp.

@var{DETAIL} might be an associated file name, or some other language
specific bit of information.
@end table

@section Extra Specifiers

Some default token types have a slot EXTRA-SPEC, for extra specifiers.
These specifiers provide additional details not commonly used, or not
available in all languages.  This list is an alist, and if a given key
is nil, it is not in the list, saving space.  Some valid extra
specifiers are:

@table @code
@item (parent .  "text")
Name of a parent type/class.  This is not the same as a parent for a
type.  In C++ and CLOS allow the creation of a function outside the
body of that class.  Such functions will set the @var{parent}
specifier to a plain text string which is the name of that parent.

@item (dereference .  INT)
Number of levels of dereference.
In C, the number of array dimensions.

@item (pointer . INT)
Number of levels of pointers.
In C, the number of @code{*} characters.

@item (typemodifiers .  ( "text" ... ))
Keyword modifiers for a type.  In C, such words would include
@code{register'} and @code{volatile'}

@item (suffix . "text")
Suffix information for a variable.  Not currently used.

@item (const .  t)
This exists if the variable or function return value is constant.

@item (throws .  ( "text" ... ))
For functions or methods in languages that support typed signal
throwing, this is a list of exceptions that can be thrown.

@item (destructor . t)
This exists for functions which are destructor methods in a class
definition.  In C++, a destructor's name excludes the ~ character.  When
producing the name of the function, the ~ is added back in.

@item (constructor . t)
This exists for functions which are constructors in a class
definition.  In C++ this is t when the name of this function is
the same as the name of the parent class.

@item (user-visible . t)
For functions in interpreted languages such as Emacs Lisp, this
signals that a function or variable is user visible.  In Emacs Lisp,
this means a function is @dfn{interactive}.

@item (prototype . t)
For functions or variables that are not declared locally, a prototype
is something that will define that function or variable for use.
In C, the term represents prototypes generally used in header files.
In Emacs Lisp, the @code{autoload} statement creates prototypes.
@end table

@node Compiling, Debugging, BNF conversion, Top
@chapter Compiling a language file with the bovinator

From a program you can use the function @code{semantic-bovinate-toplevel}.
This function takes one optional parameter specifying if the cache
should be refreshed.  By default, the cached results of the last parse
are always used.  Specifying that the cache should be checked will cause
it to be flushed if it is out of date.

Another function you can use is @code{semantic-bovinate-nonterminal}.
This command takes a token stream returned by the function
@code{semantic-flex} followed by a DEPTH (as above).  This takes an
additional optional argument of NONTERMINAL which is the nonterminal in
your table it is to start parsing with.

@deffn Command bovinate &optional clear
Bovinate the current buffer.  Show output in a temp buffer.
Optional argument @var{CLEAR} will clear the cache before bovinating.
@end deffn

@deffn Command semantic-clear-toplevel-cache
Clear the toplevel bovine cache for the current buffer.
Clearing the cache will force a complete reparse next time a token
stream is requested.
@end deffn

@defun semantic-bovinate-toplevel &optional checkcache
Bovinate the entire current buffer.
If the optional argument @var{CHECKCACHE} is non-@code{nil}, then flush the cache iff
there has been a size change.
@end defun

@node Debugging, Programming, Compiling, Top
@comment  node-name,  next,  previous,  up
@chapter Debugging

Writing language files using BNF is significantly easier than writing
then using regular expressions in a functional manner.  Debugging
them, however, can still prove challenging.

There are two ways to debug a language definition if it is not
behaving as expected.  One way is to debug against the source @file{.bnf}
file.  The second is to debug against the lisp table created from the
@file{.bnf} source, or perhaps written by hand.

If your language definition was written in BNF notation, debugging is
quite easy.  The command @code{bovinate-debug} will start you off.

@deffn Command bovinate-debug
Bovinate the current buffer and run in debug mode.
@end deffn

If you prefer debugging against the Lisp table, find the table in a
buffer, place the cursor in it, and use the command
@code{semantic-bovinate-debug-set-table} in it.

@deffn Command semantic-bovinate-debug-set-table &optional clear
Set the table for the next debug to be here.
Optional argument @var{CLEAR} to unset the debug table.
@end deffn

After the table is set, the @code{bovinate-debug} command can be run
at any time for the given language.

While debugging, two windows are visible.  One window shows the file
being parsed, and the syntactic token being tested is highlighted.  The
second window shows the table being used (either in the BNF source, or
the Lisp table) with the current rule highlighted.  The cursor will
sit on the specific match rule being tested against.

In the minibuffer, a brief summary of the current situation is
listed.  The first element is the syntactic token which is a list of
the form:

@example
(TYPE START . END)
@end example

The rest of the display is a list of all strings collected for the
currently tested rule.  Each time a new rule is entered, the list is
restarted.  Upon returning from a rule into a previous match list, the
previous match list is restored, with the production of the dependent
rule in the list.

Use @kbd{C-g} to stop debugging.  There are no commands for any
fancier types of debugging.

@node Programming, Current Context, Debugging, Top
@comment  node-name,  next,  previous,  up
@chapter Programming

Once a source file has been parsed, the following APIs can be used to
write programs that use the token stream most effectively.

@menu
* Token Queries::               Getting info about a parsed token (nonterminal).
* Nonterminal Streams::         Working with streams of nonterminals.
* Nonterminals at point::       Finding nonterminals at point.
* Nonterminal Sorting::         Reorganizing streams.
* Nonterminal Completion::      Completing read functions.
* Override Methods::            Language dependent functions covering
                                conversion to text strings, language dependent
                                queries and local context information
* Parser Hooks::                How to know when tags change.
* Example Programs::            Simple programming examples.
@end menu

@node Token Queries, Nonterminal Streams, Programming, Programming
@comment  node-name,  next,  previous,  up
@section Token Queries

When writing programs that use the bovinator, the following functions
are needed to find get details out of a nonterminal.

@defun semantic-equivalent-tokens-p token1 token2
Compare @var{TOKEN1} and @var{TOKEN2} and return non-@code{nil} if they are equivalent.
Use @dfn{eq} to test of two tokens are the same.  Use this function if tokens
are being copied and regrouped to test for if two tokens represent the same
thing, but may be constructed of different cons cells.
@end defun

@defun semantic-token-token token
Retrieve from @var{TOKEN} the token identifier.
i.e., the symbol @code{'variable}, @code{'function}, @code{'type}, or other.
@end defun

@defun semantic-token-name token
Retrieve the name of @var{TOKEN}.
@end defun

@defun semantic-token-docstring token &optional buffer
Retrieve the documentation of @var{TOKEN}.
Optional argument @var{BUFFER} indicates where to get the text from.
If not provided, then only the @var{POSITION} can be provided.
@end defun

@defun semantic-token-overlay token
Retrieve the @var{OVERLAY} part of @var{TOKEN}.
The returned item may be an overlay or an unloaded buffer representation.
@end defun

@defun semantic-token-extent token
Retrieve the extent (@var{START} @var{END}) of @var{TOKEN}.
@end defun

@defun semantic-token-start token
Retrieve the start location of @var{TOKEN}.
@end defun

@defun semantic-token-end token
Retrieve the end location of @var{TOKEN}.
@end defun

@defun semantic-token-type token
Retrieve the type of @var{TOKEN}.
@end defun

@defun semantic-token-put token property value
On @var{token}, set @var{property} to @var{value}.
@end defun

@defun semantic-token-get token property
For @var{token} get the value of @var{property}.
@end defun

@defun semantic-token-extra-spec token spec
Retrieve a specifier for the variable @var{TOKEN}.
@var{SPC} is the symbol whose modifier value to get.
This function can get specifiers from any type of @var{TOKEN}.
Do not use this function if you know what type of token you are dereferencing.
Instead, use the function specific to that token type.  It will be faster.
@end defun

@defun semantic-token-type-parts token
Retrieve the parts of the type @var{TOKEN}.
@end defun

@defun semantic-token-type-parent token
Retrieve the parent of the type @var{TOKEN}.
The return value is a list.  @var{A} value of @code{nil} means no parents.
The @dfn{car} of the list is either the parent class, or a list
of parent classes.  The @dfn{cdr} of the list is the list of
interfaces, or abstract classes which are parents of @var{TOKEN}.
@end defun

@defun semantic-token-type-parent-superclass token
Retrieve the parent super classes of type type @var{TOKEN}.
@end defun

@defun semantic-token-type-parent-implement token
Retrieve the parent interfaces of type type @var{TOKEN}.
@end defun

@defun semantic-token-type-modifiers token
Retrieve the type modifiers for the type @var{TOKEN}.
@end defun

@defun semantic-token-type-extra-specs token
Retrieve the extra specifiers for the type @var{TOKEN}.
@end defun

@defun semantic-token-type-extra-spec token spec
Retrieve a extra specifier for the type @var{TOKEN}.
@var{SPEC} is the symbol whose modifier value to get.
@end defun

@defun semantic-token-function-args token
Retrieve the arguments of the function @var{TOKEN}.
@end defun

@defun semantic-token-function-modifiers token
Retrieve the type modifiers of the function @var{TOKEN}.
@end defun

@defun semantic-token-function-destructor token
Non-@code{nil} if @var{TOKEN} is a destructor function.
@end defun

@defun semantic-token-function-extra-specs token
Retrieve the extra specifiers of the function @var{TOKEN}.
@end defun

@defun semantic-token-function-extra-spec token spec
Retrieve a specifier for the function @var{TOKEN}.
@var{SPEC} is a symbol whose specifier value to get.
@end defun

@defun semantic-token-function-throws token
Retrieve the throws signal of the function @var{TOKEN}.
This is an optional field, and returns @code{nil} if it doesn't exist.
@end defun

@defun semantic-token-function-parent token
The parent of the function @var{TOKEN}.
A function has a parent if it is a method of a class, and if the
function does not appear in body of its parent class.
@end defun

@defun semantic-token-variable-const token
Retrieve the status of constantness from the variable @var{TOKEN}.
@end defun

@defun semantic-token-variable-default token
Retrieve the default value of the variable @var{TOKEN}.
@end defun

@defun semantic-token-variable-modifiers token
Retrieve type modifiers for the variable @var{TOKEN}.
@end defun

@defun semantic-token-variable-extra-specs token
Retrieve extra specifiers for the variable @var{TOKEN}.
@end defun

@defun semantic-token-variable-extra-spec token spec
Retrieve a specifier value for the variable @var{TOKEN}.
@var{SPEC} is the symbol whose specifier value to get.
@end defun

@defun semantic-token-include-system token
Retrieve the flag indicating if the include @var{TOKEN} is a system include.
@end defun

For override methods that query a token, see @xref{Token Details}.

@node Nonterminal Streams, Nonterminals at point, Token Queries, Programming
@section Nonterminal streams

These functions take some key, and returns information found inside
the nonterminal stream.  Some will return one token (the first
matching item found.)  Others will return a list of all items matching
a given criterion.  All these functions work regardless of a buffer
being in memory or not.

@defun semantic-find-nonterminal-by-name name streamorbuffer &optional search-parts search-include
Find a nonterminal @var{NAME} within @var{STREAMORBUFFER}.  @var{NAME} is a string.
If @var{SEARCH-PARTS} is non-@code{nil}, search children of tokens.
If @var{SEARCH-INCLUDE} is non-@code{nil}, search include files.
@end defun

@defun semantic-find-nonterminal-by-property property value streamorbuffer &optional search-parts search-includes
Find all nonterminals with @var{PROPERTY} equal to @var{VALUE} in @var{STREAMORBUFFER}.
Properties can be added with @dfn{semantic-token-put}.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDES} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-by-extra-spec spec streamorbuffer &optional search-parts search-includes
Find all nonterminals with a given @var{SPEC} in @var{STREAMORBUFFER}.
@var{SPEC} is a symbol key into the modifiers association list.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDES} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-by-extra-spec-value spec value streamorbuffer &optional search-parts search-includes
Find all nonterminals with a given @var{SPEC} equal to @var{VALUE} in @var{STREAMORBUFFER}.
@var{SPEC} is a symbol key into the modifiers association list.
@var{VALUE} is the value that @var{SPEC} should match.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDES} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-by-position position streamorbuffer &optional nomedian
Find a nonterminal covering @var{POSITION} within @var{STREAMORBUFFER}.
@var{POSITION} is a number, or marker.  If @var{NOMEDIAN} is non-@code{nil}, don't do
the median calculation, and return nil.
@end defun

@defun semantic-find-innermost-nonterminal-by-position position streamorbuffer &optional nomedian
Find a list of nonterminals covering @var{POSITION} within @var{STREAMORBUFFER}.
@var{POSITION} is a number, or marker.  If @var{NOMEDIAN} is non-@code{nil}, don't do
the median calculation, and return nil.
This function will find the topmost item, and recurse until no more
details are available of findable.
@end defun

@defun semantic-find-nonterminal-by-token token streamorbuffer &optional search-parts search-includes
Find all nonterminals with a token @var{TOKEN} within @var{STREAMORBUFFER}.
@var{TOKEN} is a symbol representing the type of the tokens to find.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDE} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-standard streamorbuffer &optional search-parts search-includes
Find all nonterminals in @var{STREAMORBUFFER} which define simple token types.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDE} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-by-type type streamorbuffer &optional search-parts search-includes
Find all nonterminals with type @var{TYPE} within @var{STREAMORBUFFER}.
@var{TYPE} is a string which is the name of the type of the token returned.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDES} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-by-function function streamorbuffer &optional search-parts search-includes
Find all nonterminals in which @var{FUNCTION} match within @var{STREAMORBUFFER}.
@var{FUNCTION} must return non-@code{nil} if an element of @var{STREAM} will be included
in the new list.

If optional argument @var{SEARCH-PARTS} is non-@code{nil}, all sub-parts of tokens
are searched.  The over-loadable function @dfn{semantic-nonterminal-children} is
used for the searching child lists.  If @var{SEARCH-PARTS} is the symbol
@code{'positiononly}, then only children that have positional information are
searched.

If @var{SEARCH-INCLUDES} is non-@code{nil}, then all include files are also
searched for matches.
@end defun

@defun semantic-find-nonterminal-by-function-first-match function streamorbuffer &optional search-parts search-includes
Find the first nonterminal which @var{FUNCTION} match within @var{STREAMORBUFFER}.
@var{FUNCTION} must return non-@code{nil} if an element of @var{STREAM} will be included
in the new list.
If optional argument @var{SEARCH-PARTS}, all sub-parts of tokens are searched.
The over-loadable function @dfn{semantic-nonterminal-children} is used for
searching.
If @var{SEARCH-INCLUDES} is non-@code{nil}, then all include files are also
searched for matches.
@end defun

@defun semantic-recursive-find-nonterminal-by-name name buffer
Recursively find the first occurrence of @var{NAME}.
Start search with @var{BUFFER}.  Recurse through all dependencies till found.
The return item is of the form (@var{BUFFER} @var{TOKEN}) where @var{BUFFER} is the buffer
in which @var{TOKEN} (the token found to match @var{NAME}) was found.
@end defun

@node Nonterminals at point, Nonterminal Sorting, Nonterminal Streams, Programming
@comment  node-name,  next,  previous,  up
@section Nonterminals at point

When you just want to get at a nonterminal the cursor is on, there is
a more efficient mechanism than using
@code{semantic-find-nonterminal-by-position}.  This mechanism
directly queries the overlays the parsing step leaves in the buffer.
This provides for very rapid retrieval of what function or variable
the cursor is currently in.
@refill

These functions query the current buffer's overlay system for tokens.

@defun semantic-find-nonterminal-by-overlay &optional positionormarker buffer
Find all nonterminals covering @var{POSITIONORMARKER} by using overlays.
If @var{POSITIONORMARKER} is @code{nil}, use the current point.
Optional @var{BUFFER} is used if @var{POSITIONORMARKER} is a number, otherwise the current
buffer is used.  This finds all tokens covering the specified position
by checking for all overlays covering the current spot.  They are then sorted
from largest to smallest via the start location.
@end defun

@defun semantic-find-nonterminal-by-overlay-in-region start end &optional buffer
Find all nonterminals which exist in whole or in part between @var{START} and @var{END}.
Uses overlays to determine position.
Optional @var{BUFFER} argument specifies the buffer to use.
@end defun

@defun semantic-current-nonterminal
Return the current nonterminal in the current buffer.
If there are more than one in the same location, return the
smallest token.
@end defun

@defun semantic-current-nonterminal-parent
Return the current nonterminals parent in the current buffer.
@var{A} token's parent would be a containing structure, such as a type
containing a field.  Return @code{nil} if there is no parent.
@end defun

@node Nonterminal Sorting, Nonterminal Completion, Nonterminals at point, Programming
@section Nonterminal sorting

Sometimes it is important to reorganize a token stream into a form
that is better for display to a user.  It is important to not use
functions with side effects when doing this, and that could effect
the token cache.

There are some existing utility functions which will reorganize the
token list for you.

@defun semantic-bucketize tokens &optional parent filter
Sort @var{TOKENS} into a group of buckets based on token type.
Unknown types are placed in a Misc bucket.
Type bucket names are defined by either `semantic-symbol->name-assoc-list'.
If @var{PARENT} is specified, then @var{TOKENS} belong to this @var{PARENT} in some way.
This will use `semantic-symbol->name-assoc-list-for-type-parts' to
generate bucket names.
Optional argument @var{FILTER} is a filter function to be applied to each bucket.
The filter function will take one argument, which is a list of tokens, and
may re-organize the list with side-effects.
@end defun

@defvar semantic-bucketize-token-token
Function used to get a symbol describing the class of a token.
This function must take one argument of a semantic token.
It should return a symbol found in `semantic-symbol->name-assoc-list'
which @dfn{semantic-bucketize} uses to bin up tokens.
To create new bins for an application augment
`semantic-symbol->name-assoc-list', and
`semantic-symbol->name-assoc-list-for-type-parts' in addition
to setting this variable (locally in your function).
@end defvar

@defun semantic-adopt-external-members tokens
Rebuild @var{TOKENS} so that externally defined members are regrouped.
Some languages such as @var{C}++ and @var{CLOS} permit the declaration of member
functions outside the definition of the class.  It is easier to study
the structure of a program when such methods are grouped together
more logically.

This function uses @dfn{semantic-nonterminal-external-member-p} to
determine when a potential child is an externally defined member.

Note: Applications which use this function must account for token
types which do not have a position, but have children which *do*
have positions.

Applications should use @code{semantic-mark-external-member-function}
to modify all tokens which are found as externally defined to some
type.  For example, changing the token type for generating extra
buckets with the bucket function.
@end defun

@defvar semantic-orphaned-member-metaparent-type
In @dfn{semantic-adopt-external-members}, the type of @code{'type} for metaparents.
@var{A} metaparent is a made-up type semantic token used to hold the child list
of orphaned members of a named type.
@end defvar

@defvar semantic-mark-external-member-function
Function called when an externally defined orphan is found.
Be default, the token is always marked with the @code{adopted} property.
This function should be locally bound by a program that needs
to add additional behaviors into the token list.
This function is called with one argument which is a shallow copy
of the token to be modified.  This function should return the
token (or a copy of it) which is then integrated into the
revised token list.
@end defvar

@node Nonterminal Completion, Override Methods, Nonterminal Sorting, Programming
@section Nonterminal completion

These functions provide ways reading the names of items in a buffer with
completion.


@defun semantic-read-symbol prompt &optional default stream filter
Read a symbol name from the user for the current buffer.
@var{PROMPT} is the prompt to use.
Optional arguments:
@var{DEFAULT} is the default choice.  If no default is given, one is read
from under point.
@var{STREAM} is the list of tokens to complete from.
@var{FILTER} is provides a filter on the types of things to complete.
@var{FILTER} must be a function to call on each element.  (See !!!
@end defun

@defun semantic-read-variable prompt &optional default stream
Read a variable name from the user for the current buffer.
@var{PROMPT} is the prompt to use.
Optional arguments:
@var{DEFAULT} is the default choice.  If no default is given, one is read
from under point.
@var{STREAM} is the list of tokens to complete from.
@end defun

@defun semantic-read-function prompt &optional default stream
Read a function name from the user for the current buffer.
@var{PROMPT} is the prompt to use.
Optional arguments:
@var{DEFAULT} is the default choice.  If no default is given, one is read
from under point.
@var{STREAM} is the list of tokens to complete from.
@end defun

@defun semantic-read-type prompt &optional default stream
Read a type name from the user for the current buffer.
@var{PROMPT} is the prompt to use.
Optional arguments:
@var{DEFAULT} is the default choice.  If no default is given, one is read
from under point.
@var{STREAM} is the list of tokens to complete from.
@end defun


@node Override Methods, Parser Hooks, Nonterminal Completion, Programming
@section Override Methods

These functions are called `override methods' because they provide
generic behaviors, which a given language can override.  For example,
finding a dependency file in Emacs lisp can be done with the
`locate-library' command (which overrides the default behavior.)  In C,
a dependency can be found by searching a generic search path which can
be passed in via a variable.

@menu
* Token->Text:: Converting Tokens into text strings
* Token Details:: Arbitrary token detail fetching
* Local Context:: Deriving information about a language specific local context.
* Making New Methods:: How to add your own methods for a tool
@end menu

@node Token->Text, Token Details, Override Methods, Override Methods
@subsection Token->Text

Any given token consists of Meta information which is best viewed in
some textual form.  This could be as simple as the token's name, or as
a prototype to be added to header file in C.  Not only are there
several default converters from a Token into text, but there is also
some convenient variables that can be used with them.  Use these
variables to allow options on output forms when displaying tokens in
your programs.

@defvar semantic-token->text-functions
List of functions which convert a token to text.
Each function must take the parameters @var{TOKEN} &optional @var{PARENT} @var{COLOR}.
@var{TOKEN} is the token to convert.
@var{PARENT} is a parent token or name which refers to the structure
or class which contains @var{TOKEN}.  @var{PARENT} is @var{NOT} a class which a @var{TOKEN}
would claim as a parent.
@var{COLOR} indicates that the generated text should be colored using
@code{font-lock}.
@end defvar

@defvar semantic-token->text-custom-list
@var{A} List used by customizable variables to choose a token to text function.
Use this variable in the @code{:type} field of a customizable variable.
@end defvar

Every token to text conversion function must take the same parameters,
which are @var{TOKEN}, the token to be converted, @var{PARENT}, the
containing parent (like a structure which contains a variable), and
@var{COLOR}, which is a flag specifying that color should be applied
to the returned string.

When creating, or using these strings, particularly with color, use
@dfn{concat} to build up larger strings instead of @dfn{format}.  This
will preserve text properties.

@defun semantic-name-nonterminal token &optional parent color
Return the name string describing @var{TOKEN}.
The name is the shortest possible representation.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@defun semantic-summarize-nonterminal token &optional parent color
Summarize @var{TOKEN} in a reasonable way.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@defun semantic-prototype-nonterminal token &optional parent color
Return a prototype for @var{TOKEN}.
This function should be overloaded, though it need not be used.
This is because it can be used to create code by language independent
tools.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@defun semantic-prototype-file buffer
Return a file in which prototypes belonging to @var{BUFFER} should be placed.
Default behavior (if not overridden) looks for a token specifying the
prototype file, or the existence of an @var{EDE} variable indicating which
file prototypes belong in.
@end defun

@defun semantic-abbreviate-nonterminal token &optional parent color
Return an abbreviated string describing @var{TOKEN}.
The abbreviation is to be short, with possible symbols indicating
the type of token, or other information.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@defun semantic-concise-prototype-nonterminal token &optional parent color
Return a concise prototype for @var{TOKEN}.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@defun semantic-uml-abbreviate-nonterminal token &optional parent color
Return a @var{UML} style abbreviation for @var{TOKEN}.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@node Token Details, Local Context, Token->Text, Override Methods
@subsection Token Details

These functions help derive information about tokens that may not
be obvious for non-traditional languages with their own token types.

@defun semantic-nonterminal-children token &optional positionalonly
Return the list of top level children belonging to @var{TOKEN}.
Children are any sub-tokens which may contain overlays.
The default behavior (if not overridden with @code{nonterminal-children}
is to return type parts for a type, and arguments for a function.

If optional argument @var{POSITIONALONLY} is non-@code{nil}, then only return valid
children if they contain positions.  Some languages may choose to create
lists of children without position/overlay information.

If this function is overridden, use @dfn{semantic-nonterminal-children-default}
to also include the default behavior, and merely extend your own.

Note for language authors:
  If a mode defines a language that has tokens in it with overlays that
should not be considered children, you should still return them with
this function.  If you do not, then token re-parsing, and database
saving will fail.
@end defun

@defun semantic-nonterminal-external-member-parent token
Return a parent for @var{TOKEN} when @var{TOKEN} is an external member.
@var{TOKEN} is an external member if it is defined at a toplevel and
has some sort of label defining a parent.  The parent return will
be a string.

The default behavior, if not overridden with
@code{nonterminal-external-member-parent} is get the @code{'parent} extra
specifier of @var{TOKEN}.

If this function is overridden, use
@dfn{semantic-nonterminal-external-member-parent-default} to also
include the default behavior, and merely extend your own.
@end defun

@defun semantic-nonterminal-external-member-p parent token
Return non-@code{nil} if @var{PARENT} is the parent of @var{TOKEN}.
@var{TOKEN} is an external member of @var{PARENT} when it is somehow tagged
as having @var{PARENT} as it's parent.

The default behavior, if not overridden with
@code{nonterminal-external-member-p} is to match @code{'parent} extra specifier in
the name of @var{TOKEN}.

If this function is overridden, use
@code{semantic-nonterminal-external-member-children-p-default} to also
include the default behavior, and merely extend your own.
@end defun

@defun semantic-nonterminal-external-member-children token &optional usedb
Return the list of children which are not *in* @var{TOKEN}.
If optional argument @var{USEDB} is non-@code{nil}, then also search files in
the Semantic Database.  If @var{USEDB} is a list of databases, search those
databases.

Children in this case are functions or types which are members of
@var{TOKEN}, such as the parts of a type, but which are not defined inside
the class.  @var{C}++ and @var{CLOS} both permit methods of a class to be defined
outside the bounds of the class' definition.

The default behavior, if not overridden with
@code{nonterminal-external-member-children} is to search using
@dfn{semantic-nonterminal-external-member-p} in all top level definitions
with a parent of @var{TOKEN}.

If this function is overridden, use
@dfn{semantic-nonterminal-external-member-children-default} to also
include the default behavior, and merely extend your own.
@end defun

@defun semantic-nonterminal-protection token &optional parent
Return protection information about @var{TOKEN} with optional @var{PARENT}.
This function returns on of the following symbols:
   @code{nil}      - No special protection.  Language dependent.
   @code{'public}  - Anyone can access this @var{TOKEN}.
   @code{'private} - Only methods in the local scope can access @var{TOKEN}.
   @code{'friend}  - Like private, except some outer scopes are allowed
              access to token.
Some languages may choose to provide additional return symbols specific
to themselves.  Use of this function should allow for this.

The default behavior (if not overridden with @code{nonterminal-protection}
is to return a symbol based on type modifiers.
@end defun

@defun semantic-nonterminal-abstract token &optional parent
Return non @code{nil} if @var{TOKEN} is abstract.
Optional @var{PARENT} is the parent token of @var{TOKEN}.
In @var{UML}, abstract methods and classes have special meaning and behavior
in how methods are overridden.  In @var{UML}, abstract methods are italicized.

The default behavior (if not overridden with @code{nonterminal-abstract}
is to return true if @code{abstract} is in the type modifiers.
@end defun

@defun semantic-nonterminal-leaf token &optional parent
Return non @code{nil} if @var{TOKEN} is leaf.
Optional @var{PARENT} is the parent token of @var{TOKEN}.
In @var{UML}, leaf methods and classes have special meaning and behavior.

The default behavior (if not overridden with @code{nonterminal-leaf}
is to return true if @code{leaf} is in the type modifiers.
@end defun

@defun semantic-nonterminal-static token &optional parent
Return non @code{nil} if @var{TOKEN} is static.
Optional @var{PARENT} is the parent token of @var{TOKEN}.
In @var{UML}, static methods and attributes mean that they are allocated
in the parent class, and are not instance specific.
@var{UML} notation specifies that @var{STATIC} entries are underlined.

The default behavior (if not overridden with @code{nonterminal-static}
is to return true if @code{static} is in the type modifiers.
@end defun

@defun semantic-find-dependency token
Find the filename represented from @var{TOKEN}.
@var{TOKEN} may be a stripped element, in which case @var{PARENT} specifies a
parent token that has positional information.
Depends on @code{semantic-dependency-include-path} for searching.  Always searches
`.' first, then searches additional paths.


@defvar semantic-dependency-include-path
Defines the include path used when searching for files.
This should be a list of directories to search which is specific to
the file being included.
This variable can also be set to a single function.  If it is a
function, it will be called with one arguments, the file to find as a
string, and  it should return the full path to that file, or nil.
@end defvar


@end defun


@defun semantic-find-nonterminal token &optional parent
Find the location of @var{TOKEN}.
@var{TOKEN} may be a stripped element, in which case @var{PARENT} specifies a
parent token that has position information.
Different behaviors are provided depending on the type of token.
For example, dependencies (includes) will seek out the file that is
depended on, and functions will move to the specified definition.
@end defun

@defun semantic-find-documentation token
Find documentation from @var{TOKEN} and return it as a clean string.
@var{TOKEN} might have @var{DOCUMENTATION} set in it already.  If not, there may be
some documentation in a comment preceding TOKEN's definition which we
can look for.  When appropriate, this can be overridden by a language specific
enhancement.
@end defun

@node Local Context, Making New Methods, Token Details, Override Methods
@subsection Local Context

@defun semantic-up-context &optional point
Move point up one context from @var{POINT}.
Return non-@code{nil} if there are no more context levels.
Overloaded functions using @code{up-context} take no parameters.
@end defun

@defun semantic-beginning-of-context &optional point
Move @var{POINT} to the beginning of the current context.
Return non-@code{nil} if there is no upper context.
The default behavior uses @dfn{semantic-up-context}.  It can
be overridden with @code{beginning-of-context}.
@end defun

@defun semantic-end-of-context &optional point
Move @var{POINT} to the end of the current context.
Return non-@code{nil} if there is no upper context.
Be default, this uses @dfn{semantic-up-context}, and assumes parenthetical
block delimiters.  This can be overridden with @code{end-of-context}.
@end defun

@defun semantic-get-local-variables &optional point
Get the local variables based on POINT's context.
Local variables are returned in Semantic token format.
Be default, this calculates the current bounds using context blocks
navigation, then uses the parser with @code{bovine-inner-scope} to
parse tokens at the beginning of the context.
This can be overridden with @code{get-local-variables}.
@end defun

@defun semantic-get-local-arguments &optional point
Get arguments (variables) from the current context at @var{POINT}.
Parameters are available if the point is in a function or method.
This function returns a list of tokens.  If the local token returns
just a list of strings, then this function will convert them to tokens.
Part of this behavior can be overridden with @code{get-local-arguments}.
@end defun

@defun semantic-get-all-local-variables &optional point
Get all local variables for this context, and parent contexts.
Local variables are returned in Semantic token format.
Be default, this gets local variables, and local arguments.
This can be overridden with @code{get-all-local-variables}.
Optional argument @var{POINT} is the location to start getting the variables from.
@end defun

These next set of functions handle local context parsing.   This means
looking at the code (locally) and navigating, and fetching information
such as a the type of the parameter the cursor may be typing in.

@defun semantic-end-of-command
Move to the end of the current command.
Be default, uses @code{semantic-command-separation-character}.
Override with @code{end-of-command}.
@end defun

@defun semantic-beginning-of-command
Move to the beginning of the current command.
Be default, users @code{semantic-command-separation-character}.
Override with @code{beginning-of-command}.
@end defun

@defun semantic-ctxt-current-symbol &optional point
Return the current symbol the cursor is on at @var{POINT} in a list.
This will include a list of type/field names when applicable.
This can be overridden using @code{ctxt-current-symbol}.
@end defun

@defun semantic-ctxt-current-assignment &optional point
Return the current assignment near the cursor at @var{POINT}.
Return a list as per @dfn{semantic-ctxt-current-symbol}.
Return @code{nil} if there is nothing relevant.
Override with @code{ctxt-current-assignment}.
@end defun

@defun semantic-ctxt-current-function &optional point
Return the current function the cursor is in at @var{POINT}.
The function returned is the one accepting the arguments that
the cursor is currently in.
This can be overridden with @code{ctxt-current-function}.
@end defun

@defun semantic-ctxt-current-argument &optional point
Return the current symbol the cursor is on at @var{POINT}.
Override with @code{ctxt-current-argument}.
@end defun

@defun semantic-ctxt-scoped-types &optional point
Return a list of type names currently in scope at @var{POINT}.
Override with @code{ctxt-scoped-types}.
@end defun

For details on using these functions to get more detailed information
about the current context: @xref{Context Analysis}.

@node Making New Methods,  , Local Context, Override Methods
@subsection Making New Methods

@node Parser Hooks, Example Programs, Override Methods, Programming
@section Parser Hooks

If you write a program that uses the stream of tokens in a persistent
display or database, it is necessary to know when tokens change so
that your displays can be updated.  This is especially important as
tokens can be replaced, changed, or deleted, and the associated
overlays will then throw errors when you try to use them.  Complete
integration with token changes can be achieved via several very
important hooks.

One interesting way to interact with the parser is to let it know that
changes you are going to make will not require re-parsing.

@defvar semantic-edits-are-safe
When non-@code{nil}, modifications do not require a reparse.
This prevents tokens from being marked dirty, and it
prevents top level edits from causing a cache check.
Use this when writing programs that could cause a full
reparse, but will not change the tag structure, such
as adding or updating top-level comments.
@end defvar

Next, it is sometimes useful to know what the current parsing state
is.  These function can let you know what level of re-parsing may be
needed.  Careful choices on when to reparse can make your program much
faster.

@defun semantic-bovine-toplevel-full-reparse-needed-p &optional checkcache
Return non-@code{nil} if the current buffer needs a full reparse.
Optional argument @var{CHECKCACHE} indicates if the cache check should be made.
@end defun

@defun semantic-bovine-toplevel-partial-reparse-needed-p &optional checkcache
Return non-@code{nil} if the current buffer needs a partial reparse.
This only returns non-@code{nil} if @dfn{semantic-bovine-toplevel-full-reparse-needed-p}
returns nil.
Optional argument @var{CHECKCACHE} indicates if the cache check should be made
when checking @dfn{semantic-bovine-toplevel-full-reparse-needed-p}.
@end defun

If you need very close interaction with the user's editing, then these
two hooks can be used to find out when a given tag is being changed.
These hooks could even be used to cut down on re-parsing if used correctly.

For all hooks, make sure you are careful to add it as a local hook if
you only want to effect a single buffer.  Setting it globally can
cause unwanted effects if your program is concerned with a single buffer.

@defvar semantic-dirty-token-hooks
Hooks run after when a token is marked as dirty (edited by the user).
The functions must take @var{TOKEN}, @var{START}, and @var{END} as a parameters.
This hook will only be called once when a token is first made dirty,
subsequent edits will not cause this to run a second time unless that
token is first cleaned.  Any token marked as dirty will
also be called with @code{semantic-clean-token-hooks}, unless a full
reparse is done instead.
@end defvar

@defvar semantic-clean-token-hooks
Hooks run after a token is marked as clean (re-parsed after user edits.)
The functions must take a @var{TOKEN} as a parameter.
Any token sent to this hook will have first been called with
@code{semantic-dirty-token-hooks}.  This hook is not called for tokens
marked dirty if the buffer is completely re-parsed.  In that case, use
@code{semantic-after-toplevel-cache-change-hook}.
@end defvar

@defvar semantic-change-hooks
Hooks run when semantic detects a change in a buffer.
Each hook function must take three arguments, identical to the
common hook @code{after-change-function}.
@end defvar

Lastly, if you just want to know when a buffer changes, use this hook.

@defvar semantic-after-toplevel-bovinate-hook
Hooks run after a toplevel token parse.
It is not run if the toplevel parse command is called, and buffer does
not need to be fully re-parsed.
This function is also called when the toplevel cache is flushed, and
the cache is emptied.
For language specific hooks, make sure you define this as a local hook.

This hook should not be used any more.
Use @code{semantic-after-toplevel-cache-change-hook} instead.
@end defvar

@defvar semantic-after-toplevel-cache-change-hook
Hooks run after the buffer token list has changed.
This list will change when a buffer is re-parsed, or when the token
list in a buffer is cleared.  It is *NOT* called if the current token
list partially re-parsed.

Hook functions must take one argument, which is the new list of
tokens associated with this buffer.

For language specific hooks, make sure you define this as a local hook.
@end defvar

@defvar semantic-after-partial-cache-change-hook
Hooks run after the buffer token list has been updated.
This list will change when the current token list has been partially
re-parsed.

Hook functions must take one argument, which is the list of tokens
updated among the ones associated with this buffer.

For language specific hooks, make sure you define this as a local hook.
@end defvar

@defvar semantic-before-toplevel-cache-flush-hook
Hooks run before the toplevel nonterminal cache is flushed.
For language specific hooks, make sure you define this as a local hook.
This hook is called before a corresponding
@code{semantic-after-toplevel-cache-change-hook} which is also called
during a flush when the cache is given a new value of nil.
@end defvar

@node Example Programs,  , Parser Hooks, Programming
@section Programming Examples

Here are some simple examples that use different aspects of the
semantic library APIs.  For fully functional example programs with
lots of comments, see the file @file{semantic-examples.el}.

@subsection Interactively querying for a token name

If you need a command that asks the user for a token name, you can
get full range completion using the query functions
@ref{Nonterminal Completion}.

@example
(interactive (list (semantic-read-symbol "Symbol: ")))
@end example

@subsection Finding a symbol in a buffer

If you have the name of a function or variable, and need to find its
location in a buffer, you need a search function.  There is a wide
range of searches you can perform @ref{Nonterminal Streams}.

@example
(semantic-find-nonterminal-by-name
 "some-name"
 (current-buffer)
 t    ;; look inside structures and classes for these symbols
 nil) ;; do not look inside header files.
@end example

@subsection Finding a symbol in a project

If you have the name of a function or variable, and need to find its
location somewhere in a project, you need to use the Semantic Database
@ref{semanticdb}.  There are many search functions similar to the
ones found in @ref{Nonterminal Streams}.

The Semantic Database is interesting in that the return structure is
not

@subsection Locating a token in a buffer

If you have a nonterminal token, or a list of them, you may want to
find their position in a buffer.

@example
(semanticdb-find-nonterminal-by-name
 "symbol"
 nil   ;; Defaults to the current project's database list.
 t     ;; Search inside types
 nil   ;; Do not search include files
 nil   ;; Only search files in the same mode (all C files)
 t     ;; When a token is found, make sure it is loaded in a buffer.
 )
@end example

Of interesting note above, semanticdb can find symbols in files that
are not loaded into an Emacs buffer.  These tokens do not have an
associated overlay, and the function @dfn{semantic-token-buffer} will
fail.

The last parameter's tells the search function to
@dfn{find-file-noselect} any file in which a matching token was
found.  This will allow you to merge all the tokens into a completion
list, or other flat list needed by most functions that use
association lists.

If you do not ask semanticdb to load those files, you will need to
explicitly request the database object (found in the @code{car} of
each sublist) get the file loaded.  It is useful to not auto find all
files if you don't need to jump to that token.

@subsection Converting a token into a human readable string.

A nonterminal token is a rather unpleasant Lisp structure when trying
to decipher what is going on.  As such, there is a wide range of
functions available that can convert a token into a human readable,
and colorful string @ref{Token->Text}.

If you program interfaces with lots of users, you will probably want
to have your program define a configurable variable that will let
users change the visible portion of your program.

@example
(defcustom my-summary-function 'semantic-uml-prototype-nonterminal
  "*Function to use when showing info about my token."
  :group 'my-program
  :type semantic-token->text-custom-list)
@end example

Note the special type provided by semantic.

Next, you can call this function to create a string.

@example
(funcall my-summary-function token
                             token-parent
                             t ; use color
                             )
@end example

In this case, @var{token-parent} is an optional argument.  In many
cases, parent is not used by the outputting function.  The parent
may be a struct or class that contains @var{token}, or nil for
top-level definitions.  In particular, C++ needs the parent to
correctly calculate the protection of each method.


@node Current Context, Tools, Programming, Top
@comment  node-name,  next,  previous,  up
@chapter Deriving the Current Context

This chapter deals with how to derive the current context, and also
how a language maintainer can get the current context API to work
with their language.

By default, the behavior will function in C like languages.  This
means languages with parenthetical blocks, and type dereferencing
which uses a similar form.

@menu
* Blocks::
* Local Variables::             Getting lists of local variables.
* Derived Context::             What goes at a given location?
* Context Analysis::            Analysis information about the local context.
@end menu

@node Blocks, Local Variables, Current Context, Current Context
@section Blocks and Navigation

Source code is typically built up of control structures, and blocks of
context, or lexical scope.  Semantic terms these lexical scopes as a
``context''.  The following functions can be used to navigate contexts.
Some of them are override functions.  Language authors can override
a subset of them to make them work for their language.

@defun semantic-up-context &optional point
Move point up one context from @var{POINT}.
Return non-@code{nil} if there are no more context levels.
Overloaded functions using @code{up-context} take no parameters.
@end defun

@defun semantic-beginning-of-context &optional point
Move @var{POINT} to the beginning of the current context.
Return non-@code{nil} if there is no upper context.
The default behavior uses @code{semantic-up-context}.  It can
be overridden with @code{beginning-of-context}.
@end defun

@defun semantic-end-of-context &optional point
Move @var{POINT} to the end of the current context.
Return non-@code{nil} if there is no upper context.
Be default, this uses @code{semantic-up-context}, and assumes parenthetical
block delimiters.  This can be overridden with @code{end-of-context}.
@end defun

These next set of functions can be used to navigate across commands.

@defun semantic-end-of-command
Move to the end of the current command.
Be default, uses @code{semantic-command-separation-character}.
Override with @code{end-of-command}.
@end defun

@defun semantic-beginning-of-command
Move to the beginning of the current command.
Be default, users @code{semantic-command-separation-character}.
Override with @code{beginning-of-command}.
@end defun

@node Local Variables, Derived Context, Blocks, Current Context
@section Deriving local variables

Within a given context, or block of code, local variables are often
defined.  These functions can be used to retrieve lists of locally
scoped variables.

@defun semantic-get-local-variables &optional point
Get the local variables based on POINT's context.
Local variables are returned in Semantic token format.
By default, this calculates the current bounds using context blocks
navigation, then uses the parser with @code{bovine-inner-scope} to
parse tokens at the beginning of the context.
This can be overridden with @code{get-local-variables}.
@end defun

@defun semantic-get-local-arguments &optional point
Get arguments (variables) from the current context at @var{POINT}.
Parameters are available if the point is in a function or method.
This function returns a list of tokens.  If the local token returns
just a list of strings, then this function will convert them to tokens.
Part of this behavior can be overridden with @code{get-local-arguments}.
@end defun

@defun semantic-get-all-local-variables &optional point
Get all local variables for this context, and parent contexts.
Local variables are returned in Semantic token format.
Be default, this gets local variables, and local arguments.
This can be overridden with @code{get-all-local-variables}.
Optional argument @var{POINT} is the location to start getting the variables from.
@end defun

@node Derived Context, Context Analysis, Local Variables, Current Context
@section Deriving the Current Context

While a context has already been used to describe blocks of code,
other context include more local details, such as the symbol the
cursor is on, or the fact we are assigning into some other variable.

These context deriving functions can be overridden to provide language
specific behavior.  By default, it assumes a C like language.

@defun semantic-ctxt-current-symbol &optional point
Return the current symbol the cursor is on at @var{POINT} in a list.
This will include a list of type/field names when applicable.
This can be overridden using @code{ctxt-current-symbol}.
@end defun

@defun semantic-ctxt-current-assignment &optional point
Return the current assignment near the cursor at @var{POINT}.
Return a list as per @code{semantic-ctxt-current-symbol}.
Return @code{nil} if there is nothing relevant.
Override with @code{ctxt-current-assignment}.
@end defun

@defun semantic-ctxt-current-function &optional point
Return the current symbol the cursor is on at @var{POINT}.
The function returned is the one accepting the arguments that
the cursor is currently in.
This can be overridden with `ctxt.current-function'.
@end defun

@defun semantic-ctxt-current-argument &optional point
Return the current symbol the cursor is on at @var{POINT}.
Override with @code{ctxt-current-argument}.
@end defun

@node Context Analysis,  , Derived Context, Current Context
@section Analysis of the current context

The context parsing API is used in a context analysis library.  This
library provides high level routines for scanning through token
databases to create lists of token associates.  At it's core is a set
of EIEIO classes defining a context.  The context contains information
about what was parsed at a given position, such as the strings there,
and they type of assignment.  The analysis library then searches the
databases to determine the types and names available.

Two high level functions which can be run interactively are:

*NOTE TO SELF: Add more here*

@deffn Command semantic-analyze-current-context position
Analyze the current context at @var{POSITION}.
If called interactively, display interesting information about @var{POSITION}
in a separate buffer.
Returns an object based on symbol @code{semantic-analyze-context}.
@end deffn


@deffn Command semantic-analyze-possible-completions point
Return a list of semantic tokens which are possible completions.
Analysis is done at @var{POINT}.
@end deffn

@node Tools, Index, Current Context, Top
@comment  node-name,  next,  previous,  up
@chapter Tools

Several tools come with Semantic which would not be possible without it.
In general, these tools will work with any language supported by
Semantic.

@menu
* speedbar::                    How to use Semantic speedbar support
* imenu::                       Special support for Imenu.
* semanticdb::                  Cache your parsed buffers between sessions.
* senator::                     The Semantic Navigator
* analyzer::                    Analyze local context
* class browser::               Class hierarchy browser
* document::                    Document generation functions
* charts::                      Charting code statistics
* minor modes::                 Useful minor modes
@end menu

@node speedbar, imenu, Tools, Tools
@comment  node-name,  next,  previous,  up
@section Speedbar

Speedbar supports the display of tags through the Semantic parser.  To
use this utility, add a line like this to your @file{.emacs} file:

@example
(add-hook 'speedbar-load-hook (lambda () (require 'semantic-sb)))
@end example

or you can simply add:

@example
(require 'semantic-sb)
@end example

Once installed, speedbar will use semantic to find tokens, and will
display them appropriately.  Tags from semantic will have additional
details which can be seen, such as return type, or arguments to
functions.

If you use @file{semantic-load.el}, you do not need to add the above
lines in your .emacs file.

Two additional speedbar modes are described in @ref{Speedbar Analysis}, and
@ref{class browser}.

@node imenu, semanticdb, speedbar, Tools
@comment  node-name,  next,  previous,  up
@section Imenu support

There is special support for creating Imenu entries using semantic.
This is a highly customizable tool which can create specialized menu
systems for navigating your source file.

By default, each language that wants special imenu support will set
itself up for it.  To setup imenu for your buffers, use this command
in your @file{.emacs} file:

@example
(add-hook 'semantic-init-hooks (lambda ()
                               	   (imenu-add-to-menubar "TOKENS")))
@end example

Also supported is @dfn{which-func-mode}.  This usually uses imenu
tags to show the current function.  The semantic support for this
function uses overlays, which is much faster.

If you use @file{semantic-load.el}, you do not need to add the above
lines in your .emacs file.

You can customize imenu with the following options:

@deffn Option semantic-imenu-summary-function
Function to use when creating items in Imenu.
Some useful functions are:
@dfn{semantic-abbreviate-nonterminal}
@dfn{semantic-summarize-nonterminal}
@dfn{semantic-prototype-nonterminal}
@end deffn

@deffn Option semantic-imenu-bucketize-file
Non-@code{nil} if tokens in a file are to be grouped into buckets.
@end deffn

@deffn Option semantic-imenu-buckets-to-submenu
Non-@code{nil} if buckets of tokens are to be turned into submenus.
This option is ignored if @code{semantic-imenu-bucketize-file} is nil.
@end deffn

@deffn Option semantic-imenu-expand-type-parts
Non-@code{nil} if types should have submenus with parts in it.
@end deffn

@deffn Option semantic-imenu-bucketize-type-parts
Non-@code{nil} if elements of a type should be placed grouped into buckets.
@code{Nil} means to keep them in the same order.
Overridden to @code{nil} if @code{semantic-imenu-bucketize-file} is nil.
@end deffn

@deffn Option semantic-imenu-sort-bucket-function
Function to use when sorting tags in the buckets of functions.
@end deffn

@deffn Option semantic-imenu-index-directory
Non @code{nil} to index the entire directory for tags.
Doesn't actually parse the entire directory, but displays tags for all files
currently listed in the current Semantic database.
This variable has no meaning if semanticdb is not active.
@end deffn

@deffn Option semantic-imenu-auto-rebuild-directory-indexes
If non-@code{nil} automatically rebuild directory index imenus.
That is when a directory index imenu is updated, automatically rebuild
other buffer local ones based on the same semanticdb.
@end deffn

When adding support to a language, this variable may need to be set:

@defvar semantic-imenu-expandable-token
Tokens of this token type will be given submenu with children.
By default, a @code{type} has interesting children.  In Texinfo, however,
a @code{section} has interesting children.
@end defvar


@node semanticdb, senator, imenu, Tools
@comment  node-name,  next,  previous,  up
@section Semantic Database

Semanticdb is a utility which tracks your parsed files, and saves the
parsed information to files.  When you reload your source files,
semanticdb automatically associates the file with the cached copy,
saving time by not re-parsing your buffer.

Semanticdb also provides an API for programs to use.  These functions
will return token information without loading the source file into
memory by checking the disk cache.

To use semanticdb, add the following to your @file{.emacs} file:

@example
(require 'semanticdb)
(global-semanticdb-minor-mode 1)
@end example

If you have a tool which optionally uses the semantic database, it may
be important to track if the database mode is turned on or off.

@deffn Option semanticdb-mode-hooks
Hooks run whenever @dfn{global-semanticdb-minor-mode} is run.
Use @dfn{semanticdb-minor-mode-p} to determine if the mode has been turned
on or off.
@end deffn

@deffn Option semanticdb-persistent-path
List of valid paths that semanticdb will cache tokens to.
When @dfn{global-semanticdb-minor-mode} is active, token lists will
be saved to disk when Emacs exits.  Not all directories will have
tokens that should be saved.
The value should be a list of valid paths.  @var{A} path can be a string,
indicating a directory in which to save a variable.  An element in the
list can also be a symbol.  Valid symbols are @code{never}, which will
disable any saving anywhere, @code{always}, which enables saving
everywhere, or @code{project}, which enables saving in any directory that
passes a list of predicates in @code{semantic-project-predicates}.
@end deffn

@deffn Option semanticdb-project-roots
List of directories, where each directory is the root of some project.
All subdirectories of a root project are considered a part of one project.
Values in this string can be overridden by project management programs
via the @code{semanticdb-project-root-functions} variable.
@end deffn

The important difference between these two is that you may put just
``~'' in @code{semanticdb-persistent-path}, but you may put individual
project directories into @code{semanticdb-project-roots} so that
different database lists don't get cross referenced incorrectly.

@subsection Searching

You can search for tokens in the database using the following
functions.  It is important to note that database search functions do
not return a plain list of tokens.  This is because some tokens may
not be loaded in a buffer, which means that the found token would not
have an overlay, and no way to determine where it came from.

As such, all search functions return a list of the form:

@example
   ( (DATABASE TOKEN1 TOKEN2 ...) (DATABASE2 TOKEN3 TOKEN4 ...) ...)
@end example

@defun semanticdb-find-nonterminal-by-function function &optional databases search-parts search-includes diff-mode find-file-match
Find all occurrences of nonterminals which match @var{FUNCTION}.
Search in all @var{DATABASES}.  If @var{DATABASES} is @code{nil}, search a range of
associated databases.
When @var{SEARCH-PARTS} is non-@code{nil} the search will include children of tokens.
When @var{SEARCH-INCLUDES} is non-@code{nil}, the search will include dependency files.
When @var{DIFF-MODE} is non-@code{nil}, search databases which are of a different mode.
@var{A} Mode is the @code{major-mode} that file was in when it was last parsed.
When @var{FIND-@var{FILE}-MATCH} is non-@code{nil}, the make sure any found token's file is
in an Emacs buffer.
@end defun

@defun semanticdb-find-nonterminal-by-name name &optional databases search-parts search-includes diff-mode find-file-match
Find all occurrences of nonterminals with name @var{NAME} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES}, @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
@end defun

@defun semanticdb-find-nonterminal-by-name-regexp regex &optional databases search-parts search-includes diff-mode find-file-match
Find all occurrences of nonterminals with name matching @var{REGEX} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES} @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
@end defun

@defun semanticdb-find-nonterminal-by-type type &optional databases search-parts search-includes diff-mode find-file-match
Find all nonterminals with a type of @var{TYPE} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES} @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
@end defun

@defun semanticdb-find-nonterminal-by-property property value &optional databases search-parts search-includes diff-mode find-file-match
Find all nonterminals with a @var{PROPERTY} equal to @var{VALUE} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES} @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
Return a list ((@var{DB-TABLE} . @var{TOKEN-LIST}) ...).
@end defun

@defun semanticdb-find-nonterminal-by-extra-spec spec &optional databases search-parts search-includes diff-mode find-file-match
Find all nonterminals with a @var{SPEC} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES} @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
Return a list ((@var{DB-TABLE} . @var{TOKEN-LIST}) ...).
@end defun

@defun semanticdb-find-nonterminal-by-extra-spec-value spec value &optional databases search-parts search-includes diff-mode find-file-match
Find all nonterminals with a @var{SPEC} equal to @var{VALUE} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES} @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
Return a list ((@var{DB-TABLE} . @var{TOKEN-LIST}) ...).
@end defun

@defun semanticdb-file-stream file
Return a list of tokens belonging to @var{FILE}.
If file has database tokens available in the database, return them.
If file does not have tokens available, then load the file, and create them.
@end defun

@node senator, analyzer, semanticdb, Tools
@comment  node-name,  next,  previous,  up
@section Semantic Navigator

Senator stands for SEmantic NAvigaTOR and was written by David Ponce.

This library defines commands and a minor mode to navigate between
semantic language tokens in the current buffer.

@subsection Commands

The following user level commands are provided by Senator.

@subsubsection Navigation

@deffn Command senator-next-token
Move to the next token in the current buffer.
@end deffn

@deffn Command senator-previous-token
Move to the previous token in the current buffer.
@end deffn

@deffn Command senator-jump sym
Jump to the semantic symbol @var{SYM}.
If called interactively and a prefix argument is supplied jump in the
local type's context (see function @dfn{senator-current-type-context}).
@end deffn

@subsubsection Searching

Searching using senator mode restricts the search only to the
definition text, such as the name of the functions or variables in a
given buffer.

@deffn Command senator-isearch-toggle-semantic-mode
Toggles semantic search in isearch mode.  When semantic search is
enabled, isearch is restricted to token names.
@end deffn

@deffn Command senator-search-forward string
@deffnx Command senator-search-backward string
Search forward and backward for a token matching @var{string}.
@end deffn

@deffn Command re-search-forward regex
@deffnx Command re-search-backward regex
Search forward and backward for a token matching the regular expression
@var{regex}.
@end deffn

@deffn Command word-search-forward word
@deffnx word-search-backward word
Search forward and backward for a token whose name matches @var{word}.
@end deffn

@subsubsection Completion

Completion in senator scans all known definitions in the local file,
and uses that information to provide the completion.

@deffn Command senator-complete-symbol
Complete the current symbol under point.
@end deffn

@deffn Command senator-completion-menu-keyboard-popup
Popup a completion menu for the symbol at point.
@end deffn

@subsubsection Copy/Paste

Token Copy/Paste is a high level form of the typical copy yank used by
Emacs.  Copying a token saves the meta-information related to the
function or item the cursor is currently in.  When that information is
yanked into a new buffer, the form of the text created is based on the
current status of the programming buffer.

For example, pasting a function into a different file results in a
function call template to be inserted.  In a Texinfo file, a @@deffn
is created with documentation for that function or command.

@deffn Command senator-copy-token
Take the current token, and place it in the token ring.
@end deffn

@deffn Command senator-kill-token
Take the current token, place it in the token ring, and kill it.
Killing the token removes the text for that token, and places it into
the kill ring.  Retrieve that text with @code{yank}.
@end deffn

@deffn Command senator-yank-token
Yank a token from the token ring.
The form the token takes is different depending on where it is being
yanked to.
@end deffn

@deffn Command senator-copy-token-to-register register &optional kill-flag
Copy the current token into @var{REGISTER}.
Optional argument @var{KILL-FLAG} will delete the text of the token to the
kill ring.
@end deffn

For programmers, to provide specialized pasting, created an override
function for @code{insert-foreign-token} (see @xref{Settings}.)

@subsubsection Minor Mode

@deffn Command senator-minor-mode
Toggle the SEmantic NAvigaTOR key bindings in the current buffer.

The following default key bindings are provided when semantic minor
mode is enabled:


@table @key
@item C-c , n
senator-next-token
@item C-c , p
senator-previous-token
@item C-c , i
senator-isearch-toggle-semantic-mode
@item C-c , j
senator-jump
@item C-c , TAB
senator-complete-symbol
@item C-c , SPC
senator-completion-menu-keyboard-popup
@item C-c , C-y
senator-yank-token
@item C-c , C-w
senator-kill-token
@item C-c , M-w
senator-copy-token
@item C-c ,
@end table
@end deffn

@subsection Customization

To enable the Senator keymap in all modes that support semantic parsing,
use this:

@example
(add-hook 'semantic-init-hooks 'senator-minor-mode)
@end example

To customize navigation around different types of tokens, use the
following variables:

@deffn Option senator-step-at-token-ids
List of token identifiers where to step.
Token identifier is symbol @code{'variable}, @code{'function}, @code{'type}, or other.  If
@code{nil} navigation steps at any token found.  This is a buffer local
variable.  It can be set in a mode hook to get a specific language
navigation.
@end deffn

@deffn Option senator-step-at-start-end-token-ids
List of token identifiers where to step at start and end.
Token identifier is symbol @code{'variable}, @code{'function}, @code{'type}, or other.  If
@code{nil} navigation only step at beginning of tokens.  If @code{t} step at start
and end of any token where it is allowed to step.  Also, stepping at
start and end of a token prevent stepping inside its children.  This
is a buffer local variable.  It can be set in a mode hook to get a
specific language navigation.
@end deffn

To have a mode specific customization, do something like this in a hook:

@example
(add-hook 'mode-hook
          (lambda ()
            (setq senator-step-at-token-ids '(function variable))
            (setq senator-step-at-start-end-token-ids '(function))
            ))
@end example

This will cause navigation and search commands to stop only between
functions and variables, and to step at start and end of functions
only.

@subsection Contact information for Senator

Any comments, suggestions, bug reports or upgrade requests are
welcome.  Please send them to David Ponce at david@@dponce.com

@node analyzer, class browser, senator, Tools
@comment  node-name,  next,  previous,  up
@section Semantic Analyzer

The semantic analyzer is a library tool that performs context
analysis and can derive useful information.

@deffn Command semantic-analyze-current-context position
Analyze the current context at @var{POSITION}.
If called interactively, display interesting information about @var{POSITION}
in a separate buffer.
Returns an object based on symbol @dfn{semantic-analyze-context}.
@end deffn

While this can be used as a command, it is mostly useful that way
while debugging the analyzer, or tools using the return value.  Use
the Emacs command @dfn{describe-class} to learn more about using
@code{semantic-analyze-context}.

Another command that uses the analyzer context derives a completion
list.

@deffn Command semantic-analyze-possible-completions context
Return a list of semantic tokens which are possible completions.
@var{CONTEXT} is either a position (such as point), or a pre-calculated
context.  Passing in a context is useful if the caller also needs
to access parts of the analysis.

Completions run through the following filters:

@table @asis
@item Elements currently in scope
@item Constants currently in scope
@item Elements match the @code{:prefix} in the @var{CONTEXT}.
@item Type of the completion matches the type of the context.
@end table

Context type matching can identify the following:

@table @asis
@item No specific type
@item Assignment into a variable of some type.
@item Argument to a function with type constraints.
@end table

When called interactively, this function displays the list of
possible completions.  This is useful for debugging.
@end deffn

@menu
* Smart Completion ::           Commands for smart completion.
* Speedbar Analysis ::          Display an analysis in speedbar.
@end menu

@node Smart Completion, Speedbar Analysis, analyzer, analyzer
@comment  node-name,  next,  previous,  up
@subsection Smart Completion

The file @file{semantic-ia.el} contains two commands for performing
smart completion using the analysis library.  Analysis to calculate
these completions are done through the analyzer and completion
mechanism.  These functions just provide commands that can be bound
to key bindings.

@deffn Command semantic-ia-complete-symbol point
Complete the current symbol at @var{POINT}.
Completion options are calculated with
@dfn{semantic-analyze-possible-completions}.
@end deffn

@deffn Command semantic-ia-complete-symbol-menu point
Complete the current symbol via a menu based at @var{POINT}.
Completion options are calculated with
@dfn{semantic-analyze-possible-completions}.
@end deffn

@node Speedbar Analysis,  , Smart Completion, analyzer
@comment  node-name,  next,  previous,  up
@subsection Speedbar Analysis

The Analyzer output can be used through a speedbar interface.
This interface lists details about the analysis, such as the current
function, local arguments and variables, details on the prefix (the
symbol the cursor is on), and a list of all possible completions.
Completions are specified in
@dfn{semantic-analyze-possible-completions}
@ref{analyzer}.

Each entry can be jumped to by clicking on the name.  For strongly
typed languages, this means you will jump to the definition of the
variable, slot, or type definition.

In addition each entry has an <i> button.  Clicking on this will
display a summary of everything that is known about the variable or
type displayed on that line.

If you click on the name of a variable in the ``Completions'' menu,
then the text that was recently analyzed will be replaced with the
name of the token that was clicked on in speedbar.

@deffn Command semantic-speedbar-analysis
Start Speedbar in semantic analysis mode.
The analyzer displays information about the current context, plus a smart
list of possible completions.
@end deffn

You can also enter speedbar analyzer mode by selecting ``Analyze''
from the ``Display'' menu item on speedbar's menu.

@node class browser, document, analyzer, Tools
@comment  node-name,  next,  previous,  up
@section Class Browser

The semantic class browser is a library that can covert a project
group of files into an EIEIO based structure that contains links
between structures so that the inheritance links between semantic
tokens can be easily navigated.

The core to this library is one function in @file{semantic-cb.el}.

@defun semantic-cb-new-class-browser
Create an object representing this project's organization.
The object returned is of type @dfn{semantic-cb-project}, which contains
the slot `:types', a list of all top-level types.  Each element is a
class of type @dfn{semantic-cb-token}, or @dfn{semantic-cb-type}.
@end defun

Use the Emacs function `describe-class' to learn more about these
classes.

You can access the class inheritance structure through a speedbar
interface.  You can choose the ``Class Browser'' option from
Speedbar's ``Display'' menu item, or use the following command:

@deffn Command semantic-cb-speedbar-mode
Bring speedbar up, and put it into Class Browser mode.
This will use the Class Browser logic applied to the current Semantic
project database to build the available relations.  The structure of
the class hierarchy can then be navigated using traditional speedbar
interactions.
@end deffn

@node document, charts, class browser, Tools
@comment  node-name,  next,  previous,  up
@section Document generation

The document program uses semantic token streams to aid in the
creation of texinfo documentation.
For example, the following is a code fragment from @file{document.el}
that comes with semantic:

@lisp
(defun document (&optional resetfile)
  "Document the function or variable the cursor is in.
Optional argument RESETFILE is provided w/ universal argument.
When non-nil, query for a new documentation file."
  ...
  )
@end lisp

While visiting @file{document.el}, put the cursor somewhere within the
function shown above. Then type @kbd{M-x document}.
After asking for the texinfo file name, which in this case is
@file{semantic.texi}, this will update the texinfo
documentation of the @code{document} function in that file.
The result is that the following texinfo text will be either created
or updated in @file{semantic.texi} file:

@example
@@deffn Command document &optional resetfile
Document the function or variable the cursor is in.
Optional argument @@var@{RESETFILE@} is provided w/ universal argument.
When non-@@code@{nil@}, query for a new documentation file.
@@end deffn
@end example

Note that the function name, arguments and documentation string
is put in the right place.
Within the doc-string, the function arguments are marked with
the @@var command and the @code{nil} code fragment is marked with
@@code command.
This example provides just a glimpse of what is possible with the
syntactic information provided by semantic.

The main entry point for the
documentation generator are the following commands:

@deffn Command document &optional resetfile
Document the function or variable the cursor is in.
Optional argument @var{RESETFILE} is provided w/ universal argument.
When non-@code{nil}, query for a new documentation file.
@end deffn

@deffn Command document-inline
Document the current function with an inline comment.
@end deffn

@deffn Command document-insert-defun-comment nonterm buffer
Insert mode-comment documentation about @var{NONTERM} from @var{BUFFER}.
@end deffn

@deffn Command document-insert-new-file-header header
Insert a new header file into this buffer.  Add reference to @var{HEADER}.
Used by @code{prototype} if this file doesn't have an introductory comment.
@end deffn

In addition to these base documentation commands, the texinfo semantic
parser includes a two convenience functions when working directly with
texinfo files.

@deffn Command semantic-texi-update-doc &optional token
Update the documentation for @var{TOKEN}.
If the current buffer is a texinfo file, then find the source doc, and
update it.  If the current buffer is a source file, then get the
documentation for this item, find the existing doc in the associated
manual, and update that.
@end deffn

@deffn Command semantic-texi-goto-source &optional token
Jump to the source for the definition in the texinfo file @var{TOKEN}.
If @var{TOKEN} is @code{nil}, it is derived from the deffn under @var{POINT}.
@end deffn

@node charts, minor modes, document, Tools
@comment  node-name,  next,  previous,  up
@section Charting Commands

Some commands to draw charts of statistics generated from parsing:

@deffn Command semantic-chart-nonterminals-by-token &optional buffer-or-stream
Create a bar chart representing the number of nonterminals for a token.
Each bar represents how many toplevel nonterminal in @var{BUFFER-OR-STREAM}
exist with a given token type.  See `semantic-symbol->name-assoc-list'
for tokens which will be charted.
@end deffn

@deffn Command semantic-chart-database-size &optional buffer-or-stream
Create a bar chart representing the size of each file in semanticdb.
Each bar represents how many toplevel nonterminals in @var{BUFFER-OR-STREAM}
exist in each database entry.
@end deffn

@deffn Command semantic-chart-nonterminal-complexity-token &optional symbol buffer-or-stream
Create a bar chart representing the complexity of some tokens.
Complexity is calculated for tokens with a token of @var{SYMBOL}.  Each bar
represents the complexity of some nonterminal in @var{BUFFER-OR-STREAM}.
Only the most complex items are charted.
@end deffn


@node minor modes,  , charts, Tools
@comment  node-name,  next,  previous,  up
@section Minor Modes

@deffn Command semantic-show-dirty-mode &optional arg
Minor mode for highlighting dirty tokens.
With prefix argument @var{ARG}, turn on if positive, otherwise off.  The
minor mode can be turned on only if semantic feature is available and
the current buffer was set up for parsing.  Return non-@code{nil} if the
minor mode is enabled.
@end deffn

@deffn Command global-semantic-show-dirty-mode &optional arg
Toggle global use of @code{semantic-show-dirty-mode}.
If @var{ARG} is positive, enable, if it is negative, disable.
If @var{ARG} is @code{nil}, then toggle.
@end deffn

@deffn Option semantic-dirty-token-face
Face used to show dirty tokens in @code{semantic-show-dirty-token-mode}.
@end deffn

@deffn Command semantic-show-unmatched-syntax-mode &optional arg
Minor mode to highlight unmatched-syntax tokens.
With prefix argument @var{ARG}, turn on if positive, otherwise off.  The
minor mode can be turned on only if semantic feature is available and
the current buffer was set up for parsing.  Return non-@code{nil} if the
minor mode is enabled.
@end deffn

@deffn Command global-semantic-show-unmatched-syntax-mode &optional arg
Toggle global use of @dfn{semantic-show-unmatched-syntax-mode}.
If @var{ARG} is positive, enable, if it is negative, disable.
If @var{ARG} is @code{nil}, then toggle.
@end deffn

@deffn Option semantic-unmatched-syntax-face
Face used to show unmatched-syntax in.
The face is used in  @dfn{semantic-show-unmatched-syntax-mode}.
@end deffn

@deffn Command global-semantic-auto-parse-mode &optional arg
Toggle global use of @dfn{semantic-auto-parse-mode}.
If @var{ARG} is positive, enable, if it is negative, disable.
If @var{ARG} is @code{nil}, then toggle.
@end deffn

@deffn Command semantic-auto-parse-mode &optional arg
Minor mode to auto parse buffer following changes.
With prefix argument @var{ARG}, turn on if positive, otherwise off.  The
minor mode can be turned on only if semantic feature is available and
the current buffer was set up for parsing.  Return non-@code{nil} if the
minor mode is enabled.
@end deffn

@deffn Option semantic-auto-parse-no-working-message
Non-@code{nil} disable display of working message during parse.
@end deffn

@deffn Option semantic-auto-parse-idle-time
Time in seconds of idle time before auto-reparse.
This time should be short enough to ensure that auto-parse will be
run as soon as Emacs is idle.
@end deffn

@deffn Option semantic-auto-parse-max-buffer-size
Maximum size in bytes of buffers automatically re-parsed.
If this value is less than or equal to @var{0} buffers are automatically
re-parsed regardless of their size.
@end deffn

@node Index,  , Tools, Top
@comment  node-name,  next,  previous,  up
@chapter Index

@unnumbered Function Index
@printindex fn

@bye

@c LocalWords: alist API APIs arg argc args argv asis assoc autoload
@c LocalWords: backquote bnf bovinate bovinates
@c LocalWords: bovinating bovination bovinator bucketize
@c LocalWords: cb cdr charquote checkcache cindex CLOS
@c LocalWords: concat concocting const constantness ctxt Decl defcustom
@c LocalWords: deffn deffnx defun defvar destructor's dfn diff dir
@c LocalWords: doc docstring EDE EIEIO elisp emacsman emph enum
@c LocalWords: eq Exp EXPANDFULL expresssion fn foo func funcall
@c LocalWords: ia ids iff ifinfo imenu imenus init int isearch itemx java kbd
@c LocalWords: keymap keywordtable lang languagemode lexer lexing Ludlam
@c LocalWords: menubar metaparent metaparents min minibuffer Misc mode's
@c LocalWords: multitable NAvigaTOR noindent nomedian nonterm noselect
@c LocalWords: nosnarf obarray OLE OO outputfile paren parsetable POINT's
@c LocalWords: popup positionalonly positiononly positionormarker pre
@c LocalWords: printf printindex Programmatically pt punctuations quotemode
@c LocalWords: ref regex regexp Regexps reparse resetfile samp sb
@c LocalWords: scopestart SEmantic semanticdb setfilename setq
@c LocalWords: settitle setupfunction sexp sp SPC speedbar speedbar's
@c LocalWords: streamorbuffer struct subalist submenu submenus
@c LocalWords: subsubsection sw sym texi texinfo titlefont titlepage
@c LocalWords: tok TOKEN's toplevel typemodifiers uml unset untar
@c LocalWords: uref usedb var vskip xref yak
