Package libxyz :: Package parser :: Module lexer :: Class Lexer |
[hide private]
[frames] | no frames]

# Class Lexer

source code

    object --+
             |
            Lexer

    Lexical analyzer

    Lexer rules:
    -----------
    * Blank chars are usually ignored. Except from in quotes.
    * Quote can be one-line: "quoted value", or multiline:
      '''quoted value1,
         quoted value2,
      '''
    * New-line char ends commented line if any.
    * Values can be provided as simple literals or quoted ones.
    * If value contains spaces or any other non-alphanumeric values it is better
      to quote it or escape it using escapechar.
    * Variable can take list of values, separated by comma
    * Escaping can only be used in rval position.

    Macros:
    ------
    Macros are special internal variables that get expanded upon parsing.
    Macro definition is similar to variable definition, but macro char
    (default '&') is prepended to var name:
    &macro = value
    var = &macro

Instance Methods | [hide private]
__init__(self, source, tokens, comment=u"#", macro=u"&") x.__init__(...) initializes x; see x.__class__.__doc__ for signature | source code
lexer(self) Scan input for lexemes and return to parser | source code
get_idt(self) Return current state of token buffer | source code
done(self) Order lexer to stop processing | source code
unget(self, token) Put read token back to input stream | source code
escaping_on(self) Enable escaping | source code
escaping_off(self) Disable escaping | source code
Inherited from `object`: `__delattr__`, `__getattribute__`, `__hash__`, `__new__`, `__reduce__`, `__reduce_ex__`, `__repr__`, `__setattr__`, `__str__`

Class Variables | [hide private]
| TOKEN_IDT = `0`
| TOKEN_MACRO = `1`

Properties | [hide private]
Inherited from `object`: `__class__`

Method Details | [hide private]

### __init__(self, source, tokens, comment=u"#", macro=u"&") (Constructor) | source code
x.__init__(...) initializes x; see x.__class__.__doc__ for signature Parameters: `source` (string, file-like object or SourceData object) - Parsing source. If file object is passed, it must be closed by caller function after parsing completes. `tokens` (sequence) - List of tokens `comment` - Comment char `macro` - Macros char Overrides: object.__init__

### lexer(self) | source code
Scan input for lexemes and return to parser Returns: typle (token_type, token_value)