# TokenizerFunction Concept

A TokenizerFunction is a functor whose purpose is to parse a given sequence until exactly 1 token has been found or the end is reached. It then updates the token, and informs the caller of the location in the sequence of the next element immediately after the last element of the sequence that was parsed for the current token.

## Refinement of

Assignable, CopyConstructable

## Notation

`X` | A type that is a model of TokenizerFunction
`func` | Object of type `X`
`tok` | Object of Token
next | iterator that points to the first unparsed element of the sequence being parsed
end | iterator that points to the past the end of the sequence being parsed

## Definitions

A token is the result of parsing a sequence.

## Valid expressions

In addition to the expression in Assignable and CopyConstructable the following expressions are valid

Name | Expression | Return type
--- | --- | ---
Functor | `func(next, end, tok)` | `bool`
reset | `reset()` | `void`

## Expression semantics

In addition to the expression semantics in Assignable and CopyConstructable, TokenizerFunction has the following expression semantcs

Name | Expression | Precondition | Semantics | Postcondition
--- | --- | --- | --- | ---
operator() | `func(next, end, tok)` | `next` and `end` are valid iterators to the same sequence. next is a reference the function is free to modify. tok is constructed. | The return value indicates whether a new token was found in the sequence (next,end] | If the return value is true, the new token is assigned to tok. next is always updated to the position where parsing should start on the subsequent call.
reset | `reset()` | `None` | Clears out all state variables that are used by the object in parsing the current sequence. | A new sequence to parse can be given.

## Complexity guarantees

No guarantees. Models of TokenizerFunction are free to define their own complexity

## Models

escaped_list_separator

offset_separator

char_delimiters_separator

© Copyright John R. Bandela 2001. Permission to copy, use, modify, sell and distribute this document is granted provided this copyright notice appears in all copies. This document is provided "as is" without express or implied warranty, and with no claim as to its suitability for any purpose.