## org.apache.lucene.wikipedia.analysis
Class WikipediaTokenizer

    java.lang.Object
      org.apache.lucene.analysis.TokenStream
          org.apache.lucene.analysis.Tokenizer
              org.apache.lucene.wikipedia.analysis.WikipediaTokenizer

    public class WikipediaTokenizer

extends Tokenizer

Extension of StandardTokenizer that is aware of Wikipedia syntax. It is based off of the Wikipedia tutorial available at http://en.wikipedia.org/wiki/Wikipedia:Tutorial, but it may not be complete.

EXPERIMENTAL !!!!!!!!! NOTE: This Tokenizer is considered experimental and the grammar is subject to change in the trunk and in follow up releases.

Field Summary
`static int` `ACRONYM_ID`
`static int` `ALPHANUM_ID`
`static int` `APOSTROPHE_ID`
`static String` `BOLD`
`static int` `BOLD_ID`
`static String` `BOLD_ITALICS`
`static int` `BOLD_ITALICS_ID`
`static int` `BOTH`
`static String` `CATEGORY`
`static int` `CATEGORY_ID`
`static String` `CITATION`
`static int` `CITATION_ID`
`static int` `CJ_ID`
`static int` `COMPANY_ID`
`static int` `EMAIL_ID`
`static String` `EXTERNAL_LINK`
`static int` `EXTERNAL_LINK_ID`
`static String` `EXTERNAL_LINK_URL`
`static int` `EXTERNAL_LINK_URL_ID`
`static String` `HEADING`
`static int` `HEADING_ID`
`static int` `HOST_ID`
`static String` `INTERNAL_LINK`
`static int` `INTERNAL_LINK_ID`
`static String` `ITALICS`
`static int` `ITALICS_ID`
`static int` `NUM_ID`
`static String` `SUB_HEADING`
`static int` `SUB_HEADING_ID`
`static String[]` `TOKEN_TYPES` String token types that correspond to token type int constants
`static String[]` `tokenImage` Deprecated. Please use `TOKEN_TYPES` instead
`static int` `TOKENS_ONLY`
`static int` `UNTOKENIZED_ONLY`

Fields inherited from class org.apache.lucene.analysis.Tokenizer
`input`

Constructor Summary
`WikipediaTokenizer(Reader input)` Creates a new instance of the `WikipediaTokenizer`.

Method Summary
`Token` `next(Token result)`
`void` `reset()`
`void` `reset(Reader reader)`

Methods inherited from class org.apache.lucene.analysis.Tokenizer
`close`

Methods inherited from class org.apache.lucene.analysis.TokenStream
`next`

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### INTERNAL_LINK

    public static final String INTERNAL_LINK

See Also:
  Constant Field Values

### EXTERNAL_LINK

    public static final String EXTERNAL_LINK

See Also:
  Constant Field Values

### EXTERNAL_LINK_URL

    public static final String EXTERNAL_LINK_URL

See Also:
  Constant Field Values

### CITATION

    public static final String CITATION

See Also:
  Constant Field Values

### CATEGORY

    public static final String CATEGORY

See Also:
  Constant Field Values

### BOLD

    public static final String BOLD

See Also:
  Constant Field Values

### ITALICS

    public static final String ITALICS

See Also:
  Constant Field Values

### BOLD_ITALICS

    public static final String BOLD_ITALICS

See Also:
  Constant Field Values

### HEADING

    public static final String HEADING

See Also:
  Constant Field Values

### SUB_HEADING

    public static final String SUB_HEADING

See Also:
  Constant Field Values

### ALPHANUM_ID

    public static final int ALPHANUM_ID

See Also:
  Constant Field Values

### APOSTROPHE_ID

    public static final int APOSTROPHE_ID

See Also:
  Constant Field Values

### ACRONYM_ID

    public static final int ACRONYM_ID

See Also:
  Constant Field Values

### COMPANY_ID

    public static final int COMPANY_ID

See Also:
  Constant Field Values

### EMAIL_ID

    public static final int EMAIL_ID

See Also:
  Constant Field Values

### HOST_ID

    public static final int HOST_ID

See Also:
  Constant Field Values

### NUM_ID

    public static final int NUM_ID

See Also:
  Constant Field Values

### CJ_ID

    public static final int CJ_ID

See Also:
  Constant Field Values

### INTERNAL_LINK_ID

    public static final int INTERNAL_LINK_ID

See Also:
  Constant Field Values

### EXTERNAL_LINK_ID

    public static final int EXTERNAL_LINK_ID

See Also:
  Constant Field Values

### CITATION_ID

    public static final int CITATION_ID

See Also:
  Constant Field Values

### CATEGORY_ID

    public static final int CATEGORY_ID

See Also:
  Constant Field Values

### BOLD_ID

    public static final int BOLD_ID

See Also:
  Constant Field Values

### ITALICS_ID

    public static final int ITALICS_ID

See Also:
  Constant Field Values

### BOLD_ITALICS_ID

    public static final int BOLD_ITALICS_ID

See Also:
  Constant Field Values

### HEADING_ID

    public static final int HEADING_ID

See Also:
  Constant Field Values

### SUB_HEADING_ID

    public static final int SUB_HEADING_ID

See Also:
  Constant Field Values

### EXTERNAL_LINK_URL_ID

    public static final int EXTERNAL_LINK_URL_ID

See Also:
  Constant Field Values

### TOKEN_TYPES

    public static final String[] TOKEN_TYPES

  String token types that correspond to token type int constants

### tokenImage

    public static final String[] tokenImage

  Deprecated. Please use `TOKEN_TYPES` instead

### TOKENS_ONLY

    public static final int TOKENS_ONLY

See Also:
  Constant Field Values

### UNTOKENIZED_ONLY

    public static final int UNTOKENIZED_ONLY

See Also:
  Constant Field Values

### BOTH

    public static final int BOTH

See Also:
  Constant Field Values

Constructor Detail

### WikipediaTokenizer

    public WikipediaTokenizer(Reader input)

  Creates a new instance of the `WikipediaTokenizer`. Attaches the `input` to a newly created JFlex scanner.

  Parameters:
    `input` - The Input Reader

Method Detail

### next

    public Token next(Token result)
               throws IOException

  Overrides:
    `next` in class `TokenStream`
  Throws:
    `IOException`

### reset

    public void reset()
               throws IOException

  Overrides:
    `reset` in class `TokenStream`
  Throws:
    `IOException`

### reset

    public void reset(Reader reader)
               throws IOException

  Overrides:
    `reset` in class `Tokenizer`
  Throws:
    `IOException`

Copyright © 2000-2008 Apache Software Foundation. All Rights Reserved.