Class | Ferret::Analysis::RegExpTokenizer
In: | ext/r_analysis.c
Parent: | Ferret::Analysis::TokenStream

A tokenizer that recognizes tokens based on a regular expression passed to the constructor. Most possible tokenizers can be created using this class.

### Example

Below is an example of a simple implementation of a LetterTokenizer using an RegExpTokenizer. Basically, a token is a sequence of alphabetic characters separated by one or more non-alphabetic characters.

      # of course you would add more than just é
      RegExpTokenizer.new(input, /[[:alpha:]é]+/)

      "Dave's résumé, at http://www.davebalmain.com/ 1234"
        => ["Dave", "s", "résumé", "at", "http", "www", "davebalmain", "com"]

### Methods

new text text=

### Constants

REGEXP | = | rtoken_re

### Public Class methods

RegExpTokenizer.new(input, /[[:alpha:]]+/)

Create a new tokenizer based on a regular expression

input: | text to tokenizer
regexp: | regular expression used to recognize tokens in the input

### Public Instance methods

tokenizer.text = text → text

Get the text being tokenized by the tokenizer.

tokenizer.text = text → text

Set the text to be tokenized by the tokenizer. The tokenizer gets reset to tokenize the text from the beginning.

[Validate]