# Boost.Range

## Range Adaptors

#### `tokenized`

>     rng | boost::adaptors::tokenized( regex )

>     rng | boost::adaptors::tokenized( regex, i )

>     rng | boost::adaptors::tokenized( regex, rndRng )

>     rng | boost::adaptors::tokenized( regex, i, flags )

>     rng | boost::adaptors::tokenized( regex, rndRng, flags )

>     boost::make_tokenized_range( rng, regex, i, flags )

>     boost::make_tokenized_range( rng, regex, rngRng, flags )

- Precondition:
  - Let `T` denote `typename range_value< decltype(rng) >::type`, then `regex` has the type `basic_regex<T>` or is implicitly convertible to one of these types.
  - `i` has the type `int`.
  - the value-type of `rndRng` is `int`.
  - `flags` has the type `regex_constants::syntax_option_type`.
- Returns: A range whose iterators behave as if they were the original iterators wrapped in `regex_token_iterator`. The first iterator in the range would be constructed by forwarding all the arguments of `tokenized()` to the `regex_token_iterator` constructor.
- Throws: Whatever constructing and copying equivalent `regex_token_iterator`s might throw.
- Range Category: RandomAccessRange

### Example

            #include <boost/range/adaptor/tokenized.hpp>
            #include <boost/range/algorithm_ext/push_back.hpp>
            #include <boost/assert.hpp>
            #include <algorithm>
            #include <string>
            #include <vector>

            int main(int argc, const char* argv)
            {
                using namespace boost::adaptors;

                std::string input = "a b c d e f g hijklmnopqrstuvwxyz";
                std::vector< boost::sub_match< std::string::iterator > > result;
                boost::push_back(result, input | tokenized(boost::regex("\\b")));

                BOOST_ASSERT( boost::size(result) == 16u );

                return 0;
            }

(C) Copyright Neil Groves 2009 (C) Copyright Thorsten Ottosen 2003-2004