# class tokenizer

Abstract class to define a source of tokens for the parser. More...

Contains pure virtuals
Full name | `toSQLParse::tokenizer`
Definition | `#include <tosqlparse.h>`
Inherited by | editorTokenizer, stringTokenizer
List of all Methods

Annotated List
Files
Globals
Hierarchy
Index

#### Public Methods

- tokenizer (int offset=0,int line=0)
- tokenizer (toSyntaxAnalyzer &analyzer,int offset=0,int line=0)
- virtual ~tokenizer ()
- virtual QString getToken (bool forward=true,bool comment=false)
- virtual void setAnalyzer (toSyntaxAnalyzer &analyzer)
- virtual toSyntaxAnalyzer & analyzer ()
- virtual int line (void)
- virtual int offset (void)
- virtual void setOffset (int offset)
- virtual void setLine (int line)
- virtual QString remaining (bool eol)

#### Protected Members

- int Offset
- int Line

## Detailed Description

Abstract class to define a source of tokens for the parser.

int Offset

### Offset

[protected]

int Line

### Line

[protected]

tokenizer (int offset=0,int line=0)

### tokenizer

Create a tokenizer. Optionally specify which line and offset to start at.

tokenizer (toSyntaxAnalyzer &analyzer,int offset=0,int line=0)

### tokenizer

Create a tokenizer. Optionally specify which line and offset to start at.

~tokenizer ()

### ~tokenizer

[virtual]

QString getToken (bool forward=true,bool comment=false)

### getToken

[pure virtual]

Get a token from the string.

Parameters:

forward | Go forward or backwards to get next token.
comment | Include comments as tokens.

void setAnalyzer (toSyntaxAnalyzer &analyzer)

### setAnalyzer

[virtual]

Specify the syntax analyzer to use for the tokanizer

Parameters:

analyzer |

toSyntaxAnalyzer & analyzer ()

### analyzer

[virtual]

int line (void)

### line

[virtual]

Get the current line of the tokenizer. A line is defined by a \n character

int offset (void)

### offset

[virtual]

Current offset of the tokenizer (Should point to the character after the last token in the specified direction)

void setOffset (int offset)

### setOffset

[virtual]

Set a new offset of tokenzer.

void setLine (int line)

### setLine

[virtual]

Set new current line of tokenizer. This will not affect the current position of the tokenizer.

QString remaining (bool eol)

### remaining

[pure virtual]

Get the data remaining after the current position.

Parameters:

eol | If true end of line, otherwise end of tokenizer.

Generated by: nneul on skyhawk on Wed Feb 23 19:49:58 2005, using kdoc 2.0a54.