Package libxyz :: Package parser :: Module lexer :: Class Lexer

[hide private]
[frames] | no frames]

# Class Lexer

source code

    object --+
             |
            Lexer

    Lexical analyzer

    Lexer rules:
    -----------
    * Blank chars are usually ignored. Except from in quotes.
    * Quote can be one-line: "quoted value", or multiline:
      '''quoted value1,
         quoted value2,
      '''
    * New-line char ends commented line if any.
    * Values can be provided as simple literals or quoted ones.
    * If value contains spaces or any other non-alphanumeric values it is better
      to quote it or escape it using escapechar.
    * Variable can take list of values, separated by comma
    * Escaping can only be used in rval position.

    Macros:
    ------
    Macros are special internal variables that get expanded upon parsing.
    Macro definition is similar to variable definition, but macro char
    (default '&') is prepended to var name:
    &macro = value
    var = &macro

Instance Methods | [hide private]

__init__(self, source, tokens, comment=u"#", macro=u"&") x.__init__(...) initializes x; see x.__class__.__doc__ for signature | source code

lexer(self) Scan input for lexemes and return to parser | source code

get_idt(self) Return current state of token buffer | source code

done(self) Order lexer to stop processing | source code

unget(self, token) Put read token back to input stream | source code

escaping_on(self) Enable escaping | source code

escaping_off(self) Disable escaping | source code

Inherited from `object`: `__delattr__`, `__getattribute__`, `__hash__`, `__new__`, `__reduce__`, `__reduce_ex__`, `__repr__`, `__setattr__`, `__str__`

Class Variables | [hide private]

| TOKEN_IDT = `0`
| TOKEN_MACRO = `1`

Properties | [hide private]

Inherited from `object`: `__class__`

Method Details | [hide private]

### __init__(self, source, tokens, comment=u"#", macro=u"&")
(Constructor)

source code

x.__init__(...) initializes x; see x.__class__.__doc__ for signature

Parameters:
  - `source` (string, file-like object or SourceData object) - Parsing source. If file object is passed, it must be closed by caller function after parsing completes.
  - `tokens` (sequence) - List of tokens
  - `comment` - Comment char
  - `macro` - Macros char
Overrides: object.__init__

### lexer(self)

source code

Scan input for lexemes and return to parser

Returns:
  typle (token_type, token_value)