# Token Iterator

    template <
    	class TokenizerFunc = char_delimiters_separator<char>,
    	class Iterator = std::string::const_iterator,
    	class Type = std::string
    >
    class token_iterator_generator

    template<class Type, class Iterator, class TokenizerFunc>
    typename token_iterator_generator<TokenizerFunc,Iterator,Type>::type
    make_token_iterator(Iterator begin, Iterator end,const TokenizerFunc& fun)

The token iterator serves to provide an iterator view of the tokens in a parsed sequence.

## Example

    /// simple_example_5.cpp
    #include<iostream>
    #include<boost/token_iterator.hpp>
    #include<string>

    int main(){
       using namespace std;
       using namespace boost;
       string s = "12252001";
       int offsets[] = {2,2,4};
       offset_separator f(offsets, offsets+3);
       typedef token_iterator_generator<offset_separator>::type Iter;
       Iter beg = make_token_iterator<string>(s.begin(),s.end(),f);
       Iter end = make_token_iterator<string>(s.end(),s.end(),f);
       // The above statement could also have been what is below
       // Iter end;
       for(;beg!=end;++beg){
         cout << *beg << "\n";
       }
    }

### Template Parameters

Parameter | Description
--- | ---
`TokenizerFunc` | The TokenizerFunction used to parse the sequence.
`Iterator` | The type of the iterator the specifies the sequence.
`Type` | The type of the token, typically string.

## Model of

The category of Iterator, up to and including Forward Iterator. Anything higher will get scaled down to Forward Iterator.

## Related Types

Type | Remarks
token_iterator_generator::type | The type of the token iterator.

## Creation

    template<class Type, class Iterator, class TokenizerFunc>
    typename token_iterator_generator<TokenizerFunc,Iterator,Type>::type
    make_token_iterator(Iterator begin, Iterator end,const TokenizerFunc& fun)

Parameter | Description
begin | The beginning of the sequence to be parsed.
end | Past the end of the sequence to be parsed.
fun | A functor that is a model of TokenizerFunction

© Copyright John R. Bandela 2001. Permission to copy, use, modify, sell and distribute this document is granted provided this copyright notice appears in all copies. This document is provided "as is" without express or implied warranty, and with no claim as to its suitability for any purpose.