Package babel :: Package messages :: Module jslexer

# Module jslexer

A simple JavaScript 1.5 lexer which is used for the JavaScript extractor.

Classes
| Token Represents a token as returned by tokenize.

Functions

indicates_division(token) A helper function that helps the tokenizer to decide if the current token may be followed by a division operator. |

unquote_string(string) Unquote a string with JavaScript rules. |

tokenize(source) Tokenize a JavaScript source. |

Variables
| operators = ``[``'``>>>=``'``,` `'``>>>``'``,` `'``<<=``'``,` `'``>>=``'``,` `'``!=``'``,` `'``==``'``,` `'``<=``'``,` `'``>``...``
| escapes = ``{``'``b``'``:` `'``\x08``'``,` `'``f``'``:` `'``\x0c``'``,` `'``n``'``:` `'``\n``'``,` `'``r``'``:` `'``\r``'``,` `'``t``'``...``
| rules = ``[``(`None`,` re.compile(r'`(?u)`\s`+`')`)``,` `(`None`,` re.compile(r'<`...``
| division_re = `re.compile(r'/=`?`')`
| regex_re = `re.compile(r'`(?s)`/`(?:``[``^`/\\`]``*``(?:`\\.`[``^`/\\`]``*``)``*``)`/`[`a`-`zA`-``...``
| line_re = `re.compile(r'`(`\r\n`|`\n`|`\r`)`')`
| line_join_re = `re.compile(r'\\`(`\r\n`|`\n`|`\r`)`')`
| uni_escape_re = `re.compile(r'`[`a`-`fA`-`F0`-`9`]``{1,4}`')`

Function Details

### unquote_string(string)

Unquote a string with JavaScript rules. The string has to start with string delimiters (`'` or `"`.)

Returns:
  a string

### tokenize(source)

Tokenize a JavaScript source.

Returns:
  generator of Tokens

Variables Details

### operators

Value:

      ['>>>=',
       '>>>',
       '<<=',
       '>>=',
       '!=',
       '==',
       '<=',
       '>=',
      ...

### escapes

Value:

      {'b': '\x08', 'f': '\x0c', 'n': '\n', 'r': '\r', 't': '\t'}

### rules

Value:

      [(None, re.compile(r'(?u)\s+')),
       (None, re.compile(r'<!--.*')),
       ('linecomment', re.compile(r'//.*')),
       ('multilinecomment', re.compile(r'(?su)/\*.*?\*/')),
       ('name', re.compile(r'(?u)(\$+\w*|[^\W\d]\w*)')),
       ('number',
        re.compile(r'(?x)((?:0|[1-9]\d*)(\.\d+)?([eE][-\+]?\d+)?|(0x[a-fA-F0
      -9]+))')),
      ...

### regex_re

Value:

      re.compile(r'(?s)/(?:[^/\\]*(?:\\.[^/\\]*)*)/[a-zA-Z]*')