Contents | Previous | Next

#
Adding Parsers

Each parser in Source-Navigator is capable of understanding source files written in a specific programming language. In the Source-Navigator suite, parsers are stand-alone executables which adhere to a consistent command line interface. This interface allows Source-Navigator to control the parser's behavior through command line switches that the parser is expected to observe.

The Source-Navigator Software Development Kit (SDK) provides a C-based application programming interface (API) which enables parsers to insert information into a project database.

All files listed in SDK-Related Files can be found in the `.../share/sdk` directory.

SDK-Related Files

Subdirectory | Files | Description
--- | --- | ---
`/include/` | `sn.h snptools.h` | Include files
`/lib/` | `libdbutils.a libdb.a libsnptools.a libtcl8.1.a` | Source-Navigator library files
`/parsers/examples/assembly` | `Makefile README a.c abrowser.l.in b.c build-macros linux-i486-elf.m4 solaris-sparc.m4 toolbox.m4` |
`/parsers/examples/elf` | `Makefile README blobsql.elf ebrowser.l` | Makefile for ebrowser.l The latest release information Example source Parser (lex)

Parsers may be implemented in any programming language. Naturally, the task of writing a parser is significantly simpler when using compiler generation tools such as GNU flex. All of the examples provided with Source-Navigator are written using GNU flex.

A sample parser for the Elf language (an embedded SQL-like language) is provided with the SDK. To experiment with this parser, change to the `.../share/sdk/parsers` directory and type

    make test

This will compile the parser and then parse a number of Elf source files, placing items of interest into a project database.

## The Parser Toolbox Library

A library, implemented above the API, simplifies the task of writing new parsers. This library, known as the parser toolbox, allows programmers to focus on the issues of parsing source files in their chosen language and then storing the relevant information in the database.

The toolbox provides a number of C functions that can help you write your parser with less effort. These functions can be grouped as follows:

- functions to maintain line and column counts in your source files. Whenever the parser encounters an interesting symbol in the source program, it must know where in the source file the symbol occurred.

  That is, the function `sn_advance_line()` will increment a line counter maintained internally by the toolbox library. This function would be called along with other actions that the parser might perform when it encounters a newline character in the source text.

- a function to determine the name of the source file currently being parsed.

- miscellaneous text processing functions for counting the number of lines and columns consumed by a given block of text and so on.

- an entry point, called `sn_main`, which manages all of the details of interfacing with Source-Navigator and parsing each source file. In most circumstances, it is sufficient for the parser to call `sn_main` and allow this function to call back to your actual parsing function when required.

  This introduces a number of important concepts best explained with an example:

      int sn_main(int argc, char *argv[], char *lang_string,
          FILE **infile_ref, int(*parse)(), void(*reset)());
      char group[] = "java";
      int main(int argc, char *argv[])
      {
        return sn_main(argc, argv, group, &yyin, yylex, reset);
      }

  When calling `sn_main`, it is necessary to pass:

  - the `argc` and `argv` variables as passed into your `main()` function. This allows the library to access the command line options given by Source-Navigator.

  - a pointer to a string identifying the language. In the example above, the string is called `group`.

  - a pointer to a `FILE *` stream variable. You must pass the address of this variable as the library will manipulate the stream when opening new source files. If you are using GNU flex/bison, then pass a pointer to the global variable `yyin`.

  - A pointer to a function which takes no arguments and returns an `int`. This is a pointer to your actual parsing function. If you are using GNU flex/bison, then pass `yylex` or `yyparse`.

  - A pointer to a function which takes no arguments and returns `void`. This function is expected to perform any actions prior to processing the next source file. Typically this function might look like:

        void reset()
        {
          sn_reset_line(); /* reset line count */
          sn_reset_column(); /* reset column count */
        }

A detailed description of the functions available in the parser toolbox library can be found in `.../share/sdk/include/snptools.h`.

Unless the parser has to do something out of the ordinary, it should be possible to create a new parser by following these steps:

1. `#include snptools.h` in your program (or within the verbatim section of your lex specification).

2. Define a `main` function which calls `sn_main` with the appropriate arguments and returns the result of `sn_main` to the host environment.

3. Utilize the parser toolbox routines to simplify your work within the parser. For example, `sn_message` may be called to display messages in a dialog box that is shown to the user during the parsing process.

4. When your parser recognizes important language constructs such as comments, function declarations, or function invocations, call the appropriate `sn_insert` function to insert this information into the project database.

5. Link your final program with the `sntools` library. A typical command line for linking a parser can be found in the Makefile given in the `.../share/sdk/parsers/examples/elf` directory.

## Project database calls

The toolbox library provides a number of functions for inserting information into the database that the parser will encounter in the source text. Each of these functions return an `int` with two possible return values:

0 | success
-1 | failure

### sn_insert_symbol

    int sn_insert_symbol(int id_type, char *classname,
      char *identifier, char *filename, int start_lineno,
      int startCol, int endLine, int endCol, unsigned long attrib,
      char *returnType, char *argTypes, char *argNames,
      char *comment, int highStartLine, int highStartCol,
      int highEndLine, int highEndCol);

`sn_insert_symbol` inserts a symbol into the project database.

`type` determines the type of the symbol. Possible values are:

`SN_TYPE_DEF` | type definitions (e.g. a C `typedef` )
`SN_CLASS_DEF` | class definition (particularly for object-oriented languages)
`SN_MBR_FUNC_DEF` | member functions
`SN_MBR_VAR_DEF` | member variables
`SN_ENUM_DEF` | enumerations
`SN_CONS_DEF` | constants
`SN_MACRO_DEF` | macros
`SN_FUNC_DEF` | functions
`SN_SUBR_DEF` | subroutines
`SN_GLOB_VAR_DEF` | global variables
`SN_COMMON_DEF` | common blocks
`SN_COMMON_MBR_VAR_DEF` | member variables within a common block
`SN_CLASS_INHERIT` | class inheritance
`SN_MBR_FUNC_DCL` | member function declarations
`SN_FUNC_DCL` | functions
`SN_ENUM_CONST_DEF` | enumeration constant
`SN_UNION_DEF` | definitions of unions or variant records
`SN_FRIEND_DCL` | friends (for C++)
`SN_NAMESPACE_DEF` | name spaces
`SN_EXCEPTION_DEF` | exceptions
`SN_LOCAL_VAR_DEF` | local variables
`SN_VAR_DCL` | variables
`SN_INCLUDE_DEF` | include files
`SN_REF_UNDEFINED` | reference to undefined symbols

`classname` | the name of the class, structure or common block of the symbol if the symbol's type is one of `SN_MBR_FUNC_DEF`, `SN_MBR_VAR_DEF`, `SN_COMMON_MBR_VAR_DEF`, or `SN_CLASS_INHERIT`. Otherwise, `classname` must be a NULL pointer. If the symbol's type is `SN_CLASS_INHERIT`, `classname` contains the name of the base class.
`identifier` | the name of the symbol to be inserted into the project database.
`filename` | the name of the source file in which this symbol was encountered.
`start_lineno` | the line number of the position where the symbol starts.
`startCol` | the column number of the position where the symbol starts.
`endLine` | the line number of the position where the symbol ends.
`endCol` | the column number of the position where the symbol ends.
`attrib` | contains attributes of the symbol definitions (see `sn.h`).
`returnType` | a string describing the return type of the function, subroutine or method. If the symbol is not one of these types, pass a NULL pointer.
`argTypes` | a string containing a comma-separated list of argument types for the argument list of functions, subroutines or methods. If the symbol is not one of these types, pass a NULL pointer.
`argNames` | a string containing a comma-separated list of argument names for the argument list of functions, subroutines or methods. If the symbol is not one of these types, pass a NULL pointer.
`comment` | a string containing the comment that often occurs after a definition in the source text. Note that 8-bit characters including `'\n'` may be used in the string. If there is no comment, pass a NULL pointer.
`highStartLine` | the line number of the position where the highlighting of the symbol starts.
`highStartCol` | the column number of the position where the highlighting of the symbol starts.
`highEndLine` | the line number of the position where the highlighting of the symbol ends.
`highEndCol` | the column number of the position where the highlighting of the symbol ends.

#### Examples

The following example inserts a class definition:

    sn_insert_symbol(SN_CLASS_DEF, NULL, classname,
      sn_current_file(), sn_line(), sn_column(), sn_line(),
      sn_column() + strlen(classname), 0L, NULL, NULL,
      NULL, NULL, sn_line(), sn_column(), sn_line(),
      sn_column() + strlen(classname));

The following example inserts a method definition:

    sn_insert_symbol(SN_MBR_FUNC_DEF, classname, methodname,
      sn_current_file(), sn_line(), sn_column(), sn_line(),
      sn_column() + strlen(methodname), 0L, NULL, NULL,
      NULL, NULL, sn_line(), sn_column(), sn_line(),
      sn_column() + strlen(methodname));

For a C function with the following prototype:

    int transform(struct coord * p, int x, int y, unsigned att);

the following example inserts the definition into the database:

    sn_insert_symbol(SN_FUNC_DEF, NULL, "transform",
      sn_current_file(), sn_line(), sn_column(), sn_line(),
      sn_column + len, 0L, "int", "struct coord *, int, int,
      unsigned", "p,x,y,att", NULL, NULL, sn_line(), sn_column(),
      sn_line(), sn_column() + len);

### sn_insert_xref

`sn_insert_xref` inserts cross-referencing information.

One of the most useful aspects of Source-Navigator is its cross-referencing capabilities. For instance, it is possible to see which functions are used by other functions or which functions modify a particular global variable.

Where possible, a parser should attempt to collect information of this nature and insert it into the project database. In a language with a flat namespace such as C, this can be achieved by noting the name of the current function within the parser. If a function invocation is encountered in the source text, then the cross-referencing can be inferred based on the current function's name and the function being called.

Cross-referencing information is added using:

    int sn_insert_xref(int type, int scope_type, int scope_level,
      char *classname, char *funcname, char *argtypes,
      char *refclass, char *refsymbol, char *ref_arg_types,
      char *filename, int lineno, int acc);

`type` | describes the type of the referenced symbol. It must be one of the following:
| `SN_REF_TO_TYPEDEF`
| `SN_REF_TO_DEFINE`
`` | `SN_REF_TO_ENUM`
| `SN_REF_TO_STRUCT`
| `SN_REF_TO_UNION`
`` | `SN_REF_TO_CLASS`
| `SN_REF_TO_FUNCTION`
`` | `SN_REF_TO_MBR_FUNC`
| `SN_REF_TO_MBR_VAR`
`` | `SN_REF_TO_COMM_VAR`
| `SN_REF_TO_CONSTANT`
`` | `SN_REF_TO_SUBROUTINE`
| `SN_REF_TO_GLOB_VAR`
`` | `SN_REF_TO_LOCAL_VAR`
| `SN_REF_TO_TEMPLATE`
| `SN_REF_TO_NAMESPACE`
| `SN_REF_TO_EXCEPTION`
| `SN_REF_TO_LABEL`

`scope_type` | describes the type of the location where the cross-reference information is reported. It must be one of the following: `SN_FUNC_DEF` `SN_MBR_FUNC_DEF` `SN_SUBR_DEF`
`scope_level` | describes the scope level of the referenced symbol. It must be one of: `SN_REF_SCOPE_LOCAL` `SN_REF_SCOPE_GLOBAL`
`classname` | a string containing the class name of the method if `scope_type` is `SN_MBR_FUNC_DEF`; otherwise, it must be a NULL pointer.
`funcname` | a string containing the function, method or subroutine name in which the reference information is reported.
`argtypes` | a string containing a comma-separated list of argument types for the argument list of functions, subroutines or methods. Pass NULL if there are no arguments.
`refclass` | a string containing the class, structure or common block name of the referred symbol. If the symbol is not within a namespace, pass a NULL pointer.
`refsymbol` | the name of the referred symbol.
`ref_arg_types` | a string containing a comma-separated list of argument types if the referred symbol's type is a method, subroutine or function. Pass NULL if there are no arguments or the referred symbol is not one of these types.
`filename` | the name of the source file in which the reference information is reported.
`lineno` | the line number of the source file in which the reference information is reported.
`acc` | the level of access to the referenced symbol and is one of:

| `SN_REF_READ` | symbol is read (e.g. if `(x) { ... }`)
| `SN_REF_WRITE` | symbol is modified (e.g. `x = 10` )
| `SN_REF_PASS` | variable is passed to a subroutine/function
| `SN_REF_UNUSED` | symbol is never used

#### Examples

The following example inserts cross-referencing information for a function that is called from another function.

    sn_insert_xref(SN_REF_TO_FUNCTION, SN_FUNC_DEF,
      SN_REF_SCOPE_GLOBAL, NULL, currentFunction, NULL, NULL,
      calledFunction, NULL, sn_current_file(), sn_line(),
      SN_REF_PASS);

The following example inserts cross-referencing information for a function that is called from a member function called `insert` which belongs to a C++ class called `Stack`.

    sn_insert_xref(SN_REF_TO_MBR_VAR, SN_MBR_FUNC_DEF,
      SN_REF_SCOPE_GLOBAL, "Stack", "insert", NULL, "Stack",
      "i", NULL, sn_current_file(), sn_line(), SN_REF_READ);

### sn_insert_comment

`sn_insert_comment` inserts comments into the project database.

When comments are encountered in the source text, the parser should call `sn_insert_comment` to add these comments to the project database. In some Source-Navigator projects, the user will choose to not include comments, but the parser should call this function regardless. The library function will decide whether or not to actually store the information in the database.

Comments are added to the database using:

    int sn_insert_comment(char *classname, char *funcname,
      char *filename, char *comment, int beg_line, int beg_col);

`classname` | a string containing the name of the class or method where the comment was found or NULL.
`funcname` | a string containing the name of the function or method in which the comment was found, or NULL if the comment is outside any function or method scope.
`filename` | a string containing the name of the current file being parsed.
`comment` | a string containing the comment without the comment separators. For example, in Tcl this would exclude the leading "`#`" character.
`beg_line` | the line number of the source file where the comment begins.
`beg_col` | the column number of the source file where the comment begins.

## Integration with Source-Navigator

A completed parser can be integrated into Source-Navigator by following these steps:

1. Copy the parser executable into the `.../bin` directory.

2. Edit `sn_prop.cfg` in the directory `.../share/etc`. The Tcl procedure `sn_add_parser` is used to add parsers to the configuration. The following example shows how to include support for Java:

   `sn_add_parser java -suffix {*.java} \
   -brow_cmd $odd_path(bindir)/jbrowser \
   -high_cmd $odd_path(bindir)/jbrowser \
   -high_switch "-h"`

When Source-Navigator is restarted, support for the new language becomes available. When creating a project, the Parsers tab in the Project Preferences dialog shows the new language and its associated filename extensions.

This is all that is required to add new language support to Source-Navigator. If a project is created which contains files with any of the specified filename extensions, the new parser is invoked to process those files as a part of the overall parsing process.

Contents | Previous | Next