## net.sf.saxon.expr
Class Tokenizer

    java.lang.Object
      |
      +--net.sf.saxon.expr.Tokenizer

public final class Tokenizer
extends java.lang.Object

Tokenizer for expressions and inputs. This code was originally derived from James Clark's xt, though it has been greatly modified since. See copyright notice at end of file.

Field Summary
`static int` `BARE_NAME_STATE` State in which a name is NOT to be merged with what comes next, for example "("
`int` `currentToken` The number identifying the most recently read token
`int` `currentTokenStartOffset` The position in the input expression where the current token starts
`java.lang.String` `currentTokenValue` The string value of the most recently read token
`static int` `DEFAULT_STATE` Initial default state of the Tokenizer
`java.lang.String` `input` The string being parsed
`int` `inputOffset` The current position within the input string
`java.lang.String` `lastPragma`
`static int` `OPERATOR_STATE` State in which the next thing to be read is an operator
`boolean` `recognizePragmas`
`static int` `SEQUENCE_TYPE_STATE` State in which the next thing to be read is a SequenceType
`int` `startLineNumber` The starting line number (for XPath in XSLT, the line number in the stylesheet)

Constructor Summary
`Tokenizer()`

Method Summary
`int` `getColumnNumber()` Get the column number of the current token
`int` `getColumnNumber(int offset)`
`long` `getLineAndColumn()` Get the line and column number of the current token, as a long value with the line number in the top half and the column number in the lower half
`long` `getLineAndColumn(int offset)` Get the line and column number corresponding to a given offset in the input expression, as a long value with the line number in the top half and the column number in the lower half
`int` `getLineNumber()` Get the line number of the current token
`int` `getLineNumber(int offset)`
`int` `getState()`
`void` `lookAhead()` Look ahead by one token.
`void` `next()` Get the next token from the input expression.
`char` `nextChar()` Read next character directly.
`java.lang.String` `recentText()` Get the most recently read text (for use in an error message)
`void` `setState(int state)`
`void` `tokenize(java.lang.String input, int start, int end, int lineNumber)` Prepare a string for tokenization.
`void` `treatCurrentAsOperator()` Force the current token to be treated as an operator if possible
`void` `unreadChar()` Step back one character.

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### DEFAULT_STATE

    public static final int DEFAULT_STATE

  Initial default state of the Tokenizer

  See Also:
    Constant Field Values

### BARE_NAME_STATE

    public static final int BARE_NAME_STATE

  State in which a name is NOT to be merged with what comes next, for example "("

  See Also:
    Constant Field Values

### SEQUENCE_TYPE_STATE

    public static final int SEQUENCE_TYPE_STATE

  State in which the next thing to be read is a SequenceType

  See Also:
    Constant Field Values

### OPERATOR_STATE

    public static final int OPERATOR_STATE

  State in which the next thing to be read is an operator

  See Also:
    Constant Field Values

### startLineNumber

    public int startLineNumber

  The starting line number (for XPath in XSLT, the line number in the stylesheet)

### currentToken

    public int currentToken

  The number identifying the most recently read token

### currentTokenValue

    public java.lang.String currentTokenValue

  The string value of the most recently read token

### currentTokenStartOffset

    public int currentTokenStartOffset

  The position in the input expression where the current token starts

### input

    public java.lang.String input

  The string being parsed

### inputOffset

    public int inputOffset

  The current position within the input string

### recognizePragmas

    public boolean recognizePragmas

### lastPragma

    public java.lang.String lastPragma

Constructor Detail

### Tokenizer

    public Tokenizer()

Method Detail

### getState

    public int getState()

### setState

    public void setState(int state)

### tokenize

    public void tokenize(java.lang.String input,
                         int start,
                         int end,
                         int lineNumber)
                  throws StaticError

  Prepare a string for tokenization. The actual tokens are obtained by calls on next()

  Parameters:
    `input` - the string to be tokenized
    `start` - start point within the string
    `end` - end point within the string (last character not read): -1 means end of string
  Throws:
    `StaticError` - if a lexical error occurs, e.g. unmatched string quotes

### next

    public void next()
              throws StaticError

  Get the next token from the input expression. The type of token is returned in the currentToken variable, the string value of the token in currentTokenValue.

  Throws:
    `StaticError` - if a lexical error is detected

### treatCurrentAsOperator

    public void treatCurrentAsOperator()

  Force the current token to be treated as an operator if possible

### lookAhead

    public void lookAhead()
                   throws StaticError

  Look ahead by one token. This method does the real tokenization work. The method is normally called internally, but the XQuery parser also calls it to resume normal tokenization after dealing with pseudo-XML syntax.

  Throws:
    `StaticError` - if a lexical error occurs

### nextChar

    public char nextChar()
                  throws java.lang.StringIndexOutOfBoundsException

  Read next character directly. Used by the XQuery parser when parsing pseudo-XML syntax

  Returns:
    the next character from the input
  Throws:
    `java.lang.StringIndexOutOfBoundsException` - if an attempt is made to read beyond the end of the string. This will only occur in the event of a syntax error in the input.

### unreadChar

    public void unreadChar()

  Step back one character. If this steps back to a previous line, adjust the line number.

### recentText

    public java.lang.String recentText()

  Get the most recently read text (for use in an error message)

### getLineNumber

    public int getLineNumber()

  Get the line number of the current token

### getColumnNumber

    public int getColumnNumber()

  Get the column number of the current token

### getLineAndColumn

    public long getLineAndColumn()

  Get the line and column number of the current token, as a long value with the line number in the top half and the column number in the lower half

  Returns:
    the line and column number, packed together

### getLineAndColumn

    public long getLineAndColumn(int offset)

  Get the line and column number corresponding to a given offset in the input expression, as a long value with the line number in the top half and the column number in the lower half

  Returns:
    the line and column number, packed together

### getLineNumber

    public int getLineNumber(int offset)

### getColumnNumber

    public int getColumnNumber(int offset)