lexi(1)

NAME

lexi - lexical analyser tool

SYNOPSIS

lexi [options] input-file [output-file]

Description

lexi translates a description of a lexical analyser, input-file, into code implementing that analyser. This is output into the output-file argument, if present, or the standard output otherwise.

Options

-V  Causes lexi print its version number.

-f  Causes lexi to generate code describing any functions specified in the input.

-k  Causes lexi to generate code describing any keywords specified in the input.

-lstring
    Causes lexi to use string as a prefix for its generated lexical tokens. The default is lex_.

-m  Causes lexi to generate code describing any macros specified in the input.

-t  Causes lexi to generate code describing the lookup table.

Input File

The lexi input file defines the lexical analyzer to be created. It contains character group definitions, token mappings, keyword mappings, character mappings, and control flow sequences.

Character group definitions are of form GROUP identifier = string; where identifier is composed of one or more letters and underscores. Strings may contain arbitrary characters surrounded by quotes. It is possible to concatenate strings with a ``+'' operator. For example, qabc + qdef will form qabcdef.

There are pre-defined character sequences {A-Z}, {a-z} and {0-9}. It is possible to use already defined character groups to define new character groups. For example, if we have definitions for letters (GROUP alpha = {A-Z};) and for digits (GROUP digit = {0-9};) then definition for alphanumeric characters will be GROUP alnum = "[alpha][digit]";. Special group white is supposed to contain whitespace characters, wich are silently ignored by the lexer.

The maximum number of character groups wich can be defined is 31.

The statement TOKEN string -> identifier; maps character sequence to the C or sid identifier. The C identifier may be one of variable or function, the variable or result of the function call is returned by the lexer routine. Matched characters are passed as an argument to the function.

The statement KEYWORD string -> identifier; maps character string to the sid terminal.

The statement MAPPING string -> identifier; maps string of the arbitrary length to the one character. If any MAPPING command is present, pre-pass analyzer will be created. Pre-pass analyzer is run before the main pass, this is particulary usable, for example, when mapping ANSI trigraph sequences to characters.

Character groups, tokens, keywords and mappings can be defined conditionally. This is accomplished with IF and ELSE keywords. The general form of control sequence is IF (identifier) command; ELSE command; where identifier is a C variable and command is character group or mapping definition. The ELSE part is optional. It is possible use multiple commands, surrounding them with curly brackets.

It is possible to use C style comments in the input file.

Examples

The first example is taken from lexi sources, this is snippet from input grammar specification.

/* Character groups */
/* These characters are silently ignored. */
GROUP white = " \t\n\r" ;
GROUP alpha = {A-Z} + {a-z} + "_";
GROUP digit = {0-9};
GROUP alphanum = "[alpha][digit]";
/* Simple symbols */
TOKEN "(" -> $open;
TOKEN ")" -> $close;
/* Comments, strings and identifiers */
TOKEN "/*" -> get_comment();
TOKEN """ -> get_string();
/* Keywords */
KEYWORD "GROUP" -> $group;
KEYWORD "TOKEN" -> $token;

Next example illustrates some additional features of lexi.

/* Map ANSI trigraph sequence to the ordinary bracket. */
MAPPING "??>" -> "}";
/* Define conditionally some character groups and mappings. */
IF (with_trigraphs) {
	GROUP foo = "foo";
	MAPPING "??<" -> "{";
}

The C interface

int read_token(void)
" 5 The main analyzer routine produced by
    lexi.

Next functions are used by analyzer.

int read_char(void)
" 5 Reads next token from input stream.

void unread_char(
int);
" 5 Returns character to the input stream.

int unknown_token(
int);
" 5
This function is called by read_token()         when unknown token is found.

Bugs

The lexi input file must contain at least one TOKEN directive, unless invalid lexer is produced.

See Also

tcc(1) , sid(1) .