# Tokenizer Class

      template <
    	class TokenizerFunc = char_delimiters_separator<char>,
    	class Iterator = std::string::const_iterator,
    	class Type = std::string
      >
      class tokenizer

The tokenizer class provides a container view of a series of tokens contained in a sequence. You set the sequence to parse and the TokenizerFunction to use to parse the sequence either upon construction or using the assign member function. Note: No parsing is actually done upon construction. Parsing is done on demand as the tokens are accessed via the iterator provided by begin.

## Example

    // simple_example_1.cpp
    #include<iostream>
    #include<boost/tokenizer.hpp>
    #include<string>

    int main(){
       using namespace std;
       using namespace boost;
       string s = "This is,  a test";
       tokenizer<> tok(s);
       for(tokenizer<>::iterator beg=tok.begin(); beg!=tok.end();++beg){
           cout << *beg << "\n";
       }
    }

### Template Parameters

Parameter | Description
--- | ---
`TokenizerFunc` | The TokenizerFunction used to parse the sequence.
`Iterator` | The type of the iterator the specifies the sequence.
`Type` | The type of the token, typically string.

## Related Types

Type | Remarks
iterator | The type returned by begin and end. Note: the category of iterator will be at most ForwardIterator. It will be InputIterator if the Iterator template parameter is an InputIterator. For any other category, it will be ForwardIterator.
const_iterator | Same type as iterator.
value_type | Same type as the template parameter Type
reference | Same type as value_type&
const_reference | Same type as const reference
pointer | Same type as value_type*
const_pointer | Same type as const pointer
size_type | void
difference_type | void

## Construction and Member Functions

    tokenizer(Iterator first, Iterator last,const TokenizerFunc& f = TokenizerFunc())

    template<class Container>
    tokenizer(const Container& c,const TokenizerFunc& f = TokenizerFunc())

    void assign(Iterator first, Iterator last)

    void assign(Iterator first, Iterator last, const TokenizerFunc& f)

    template<class Container>
    void assign(const Container& c)

    template<class Container>
    void assign(const Container& c, const TokenizerFunc& f)

    iterator begin() const

    iterator end() const

Parameter | Description
c | A container that contains the sequence to parse. Note: c.begin() and c.end() must be convertible to the template parameter Iterator.
f | A functor that is a model of TokenizerFunction that will be used to parse the sequence.
first | The iterator that represents the beginning position in the sequence to be parsed.
last | The iterator that represents the past the end position in the sequence to be parsed.

© Copyright John R. Bandela 2001. Permission to copy, use, modify, sell and distribute this document is granted provided this copyright notice appears in all copies. This document is provided "as is" without express or implied warranty, and with no claim as to its suitability for any purpose.