Class | Ferret::Analysis::AsciiWhiteSpaceTokenizer
In: | ext/r_analysis.c
Parent: | Ferret::Analysis::TokenStream

A WhiteSpaceTokenizer is a tokenizer that divides text at white-space. Adjacent sequences of non-WhiteSpace characters form tokens.

### Example

      "Dave's résumé, at http://www.davebalmain.com/ 1234"
        => ["Dave's", "résumé,", "at", "http://www.davebalmain.com", "1234"]

### Methods

new

### Public Class methods

AsciiWhiteSpaceTokenizer.new() → tokenizer

Create a new AsciiWhiteSpaceTokenizer

[Validate]