Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV NEXT | FRAMES NO FRAMES

## Uses of Class
org.apache.lucene.analysis.TokenStream

Packages that use TokenStream
org.apache.lucene.analysis | API and code to convert text into indexable/searchable tokens.
org.apache.lucene.analysis.br | Analyzer for Brazilian.
org.apache.lucene.analysis.cjk | Analyzer for Chinese, Japanese and Korean.
org.apache.lucene.analysis.cn | Analyzer for Chinese.
org.apache.lucene.analysis.compound | A filter that decomposes compound words you find in many Germanic languages to the word parts.
org.apache.lucene.analysis.cz | Analyzer for Czech.
org.apache.lucene.analysis.de | Analyzer for German.
org.apache.lucene.analysis.el | Analyzer for Greek.
org.apache.lucene.analysis.fr | Analyzer for French.
org.apache.lucene.analysis.miscellaneous |
org.apache.lucene.analysis.ngram |
org.apache.lucene.analysis.nl | Analyzer for Dutch.
org.apache.lucene.analysis.payloads | Provides various convenience classes for creating payloads on Tokens.
org.apache.lucene.analysis.query |
org.apache.lucene.analysis.ru | Analyzer for Russian.
org.apache.lucene.analysis.shingle |
org.apache.lucene.analysis.sinks | Implementations of the SinkTokenizer that might be useful.
org.apache.lucene.analysis.snowball | `TokenFilter` and `Analyzer` implementations that use Snowball stemmers.
org.apache.lucene.analysis.standard | A fast grammar-based tokenizer constructed with JFlex.
org.apache.lucene.analysis.th |
org.apache.lucene.benchmark.byTask.tasks | Extendable benchmark tasks.
org.apache.lucene.document | The logical representation of a `Document` for indexing and searching.
org.apache.lucene.index.memory | High-performance single-document main memory Apache Lucene fulltext search index.
org.apache.lucene.search.highlight | The highlight package contains classes to provide "keyword in context" features typically used to highlight search terms in the text of results pages.
org.apache.lucene.wikipedia.analysis |

Uses of TokenStream in org.apache.lucene.analysis

Subclasses of TokenStream in org.apache.lucene.analysis
`class` | `CachingTokenFilter` This class can be used if the Tokens of a TokenStream are intended to be consumed more than once.
`class` | `CharTokenizer` An abstract base class for simple, character-oriented tokenizers.
`class` | `ISOLatin1AccentFilter` A filter that replaces accented characters in the ISO Latin 1 character set (ISO-8859-1) by their unaccented equivalent.
`class` | `KeywordTokenizer` Emits the entire input as a single token.
`class` | `LengthFilter` Removes words that are too long and too short from the stream.
`class` | `LetterTokenizer` A LetterTokenizer is a tokenizer that divides text at non-letters.
`class` | `LowerCaseFilter` Normalizes token text to lower case.
`class` | `LowerCaseTokenizer` LowerCaseTokenizer performs the function of LetterTokenizer and LowerCaseFilter together.
`class` | `PorterStemFilter` Transforms the token stream as per the Porter stemming algorithm.
`class` | `SinkTokenizer` A SinkTokenizer can be used to cache Tokens for use in an Analyzer
`class` | `StopFilter` Removes stop words from a token stream.
`class` | `TeeTokenFilter` Works in conjunction with the SinkTokenizer to provide the ability to set aside tokens that have already been analyzed.
`class` | `TokenFilter` A TokenFilter is a TokenStream whose input is another token stream.
`class` | `Tokenizer` A Tokenizer is a TokenStream whose input is a Reader.
`class` | `WhitespaceTokenizer` A WhitespaceTokenizer is a tokenizer that divides text at whitespace.

Fields in org.apache.lucene.analysis declared as TokenStream
`protected TokenStream` | `TokenFilter.input` The source of tokens for this filter.

Methods in org.apache.lucene.analysis that return TokenStream
`TokenStream` | `WhitespaceAnalyzer.reusableTokenStream(String fieldName, Reader reader)`
`TokenStream` | `StopAnalyzer.reusableTokenStream(String fieldName, Reader reader)`
`TokenStream` | `PerFieldAnalyzerWrapper.reusableTokenStream(String fieldName, Reader reader)`
`TokenStream` | `KeywordAnalyzer.reusableTokenStream(String fieldName, Reader reader)`
`TokenStream` | `Analyzer.reusableTokenStream(String fieldName, Reader reader)` Creates a TokenStream that is allowed to be re-used from the previous time that the same thread called this method.
`TokenStream` | `SimpleAnalyzer.reusableTokenStream(String fieldName, Reader reader)`
`TokenStream` | `WhitespaceAnalyzer.tokenStream(String fieldName, Reader reader)`
`TokenStream` | `StopAnalyzer.tokenStream(String fieldName, Reader reader)` Filters LowerCaseTokenizer with StopFilter.
`TokenStream` | `PerFieldAnalyzerWrapper.tokenStream(String fieldName, Reader reader)`
`TokenStream` | `KeywordAnalyzer.tokenStream(String fieldName, Reader reader)`
`abstract TokenStream` | `Analyzer.tokenStream(String fieldName, Reader reader)` Creates a TokenStream which tokenizes all the text in the provided Reader.
`TokenStream` | `SimpleAnalyzer.tokenStream(String fieldName, Reader reader)`

Constructors in org.apache.lucene.analysis with parameters of type TokenStream
`CachingTokenFilter(TokenStream input)`
`ISOLatin1AccentFilter(TokenStream input)`
`LengthFilter(TokenStream in, int min, int max)` Build a filter that removes words that are too long or too short from the text.
`LowerCaseFilter(TokenStream in)`
`PorterStemFilter(TokenStream in)`
`StopFilter(TokenStream in, Set stopWords)` Constructs a filter which removes words from the input TokenStream that are named in the Set.
`StopFilter(TokenStream input, Set stopWords, boolean ignoreCase)` Construct a token stream filtering the given input.
`StopFilter(TokenStream input, String[] stopWords)` Construct a token stream filtering the given input.
`StopFilter(TokenStream in, String[] stopWords, boolean ignoreCase)` Constructs a filter which removes words from the input TokenStream that are named in the array of words.
`TeeTokenFilter(TokenStream input, SinkTokenizer sink)`
`TokenFilter(TokenStream input)` Construct a token stream filtering the given input.

Uses of TokenStream in org.apache.lucene.analysis.br

Subclasses of TokenStream in org.apache.lucene.analysis.br
`class` | `BrazilianStemFilter` Based on GermanStemFilter

Methods in org.apache.lucene.analysis.br that return TokenStream
`TokenStream` | `BrazilianAnalyzer.tokenStream(String fieldName, Reader reader)` Creates a TokenStream which tokenizes all the text in the provided Reader.

Constructors in org.apache.lucene.analysis.br with parameters of type TokenStream
`BrazilianStemFilter(TokenStream in)`
`BrazilianStemFilter(TokenStream in, Set exclusiontable)`

Uses of TokenStream in org.apache.lucene.analysis.cjk

Subclasses of TokenStream in org.apache.lucene.analysis.cjk
`class` | `CJKTokenizer` CJKTokenizer was modified from StopTokenizer which does a decent job for most European languages.

Methods in org.apache.lucene.analysis.cjk that return TokenStream
`TokenStream` | `CJKAnalyzer.tokenStream(String fieldName, Reader reader)` get token stream from input

Uses of TokenStream in org.apache.lucene.analysis.cn

Subclasses of TokenStream in org.apache.lucene.analysis.cn
`class` | `ChineseFilter` Title: ChineseFilter Description: Filter with a stop word table Rule: No digital is allowed.
`class` | `ChineseTokenizer` Title: ChineseTokenizer Description: Extract tokens from the Stream using Character.getType() Rule: A Chinese character as a single token Copyright: Copyright (c) 2001 Company: The difference between thr ChineseTokenizer and the CJKTokenizer (id=23545) is that they have different token parsing logic.

Methods in org.apache.lucene.analysis.cn that return TokenStream
`TokenStream` | `ChineseAnalyzer.tokenStream(String fieldName, Reader reader)` Creates a TokenStream which tokenizes all the text in the provided Reader.

Constructors in org.apache.lucene.analysis.cn with parameters of type TokenStream
`ChineseFilter(TokenStream in)`

Uses of TokenStream in org.apache.lucene.analysis.compound

Subclasses of TokenStream in org.apache.lucene.analysis.compound
`class` | `CompoundWordTokenFilterBase` Base class for decomposition token filters.
`class` | `DictionaryCompoundWordTokenFilter` A TokenFilter that decomposes compound words found in many germanic languages "Donaudampfschiff" becomes Donau, dampf, schiff so that you can find "Donaudampfschiff" even when you only enter "schiff".
`class` | `HyphenationCompoundWordTokenFilter` A TokenFilter that decomposes compound words found in many germanic languages "Donaudampfschiff" becomes Donau, dampf, schiff so that you can find "Donaudampfschiff" even when you only enter "schiff" It uses a hyphenation grammar and a word dictionary to achieve this.

Constructors in org.apache.lucene.analysis.compound with parameters of type TokenStream
`CompoundWordTokenFilterBase(TokenStream input, Set dictionary)`
`CompoundWordTokenFilterBase(TokenStream input, Set dictionary, boolean onlyLongestMatch)`
`CompoundWordTokenFilterBase(TokenStream input, Set dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)`
`CompoundWordTokenFilterBase(TokenStream input, String[] dictionary)`
`CompoundWordTokenFilterBase(TokenStream input, String[] dictionary, boolean onlyLongestMatch)`
`CompoundWordTokenFilterBase(TokenStream input, String[] dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)`
`DictionaryCompoundWordTokenFilter(TokenStream input, Set dictionary)`
`DictionaryCompoundWordTokenFilter(TokenStream input, Set dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)`
`DictionaryCompoundWordTokenFilter(TokenStream input, String[] dictionary)`
`DictionaryCompoundWordTokenFilter(TokenStream input, String[] dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)`
`HyphenationCompoundWordTokenFilter(TokenStream input, HyphenationTree hyphenator, Set dictionary)`
`HyphenationCompoundWordTokenFilter(TokenStream input, HyphenationTree hyphenator, Set dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)`
`HyphenationCompoundWordTokenFilter(TokenStream input, HyphenationTree hyphenator, String[] dictionary)`
`HyphenationCompoundWordTokenFilter(TokenStream input, HyphenationTree hyphenator, String[] dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)`

Uses of TokenStream in org.apache.lucene.analysis.cz

Methods in org.apache.lucene.analysis.cz that return TokenStream
`TokenStream` | `CzechAnalyzer.tokenStream(String fieldName, Reader reader)` Creates a TokenStream which tokenizes all the text in the provided Reader.

Uses of TokenStream in org.apache.lucene.analysis.de

Subclasses of TokenStream in org.apache.lucene.analysis.de
`class` | `GermanStemFilter` A filter that stems German words.

Methods in org.apache.lucene.analysis.de that return TokenStream
`TokenStream` | `GermanAnalyzer.tokenStream(String fieldName, Reader reader)` Creates a TokenStream which tokenizes all the text in the provided Reader.

Constructors in org.apache.lucene.analysis.de with parameters of type TokenStream
`GermanStemFilter(TokenStream in)`
`GermanStemFilter(TokenStream in, Set exclusionSet)` Builds a GermanStemFilter that uses an exclusion table.

Uses of TokenStream in org.apache.lucene.analysis.el

Subclasses of TokenStream in org.apache.lucene.analysis.el
`class` | `GreekLowerCaseFilter` Normalizes token text to lower case, analyzing given ("greek") charset.

Methods in org.apache.lucene.analysis.el that return TokenStream
`TokenStream` | `GreekAnalyzer.tokenStream(String fieldName, Reader reader)` Creates a TokenStream which tokenizes all the text in the provided Reader.

Constructors in org.apache.lucene.analysis.el with parameters of type TokenStream
`GreekLowerCaseFilter(TokenStream in, char[] charset)`

Uses of TokenStream in org.apache.lucene.analysis.fr

Subclasses of TokenStream in org.apache.lucene.analysis.fr
`class` | `ElisionFilter` Removes elisions from a token stream.
`class` | `FrenchStemFilter` A filter that stemms french words.

Methods in org.apache.lucene.analysis.fr that return TokenStream
`TokenStream` | `FrenchAnalyzer.tokenStream(String fieldName, Reader reader)` Creates a TokenStream which tokenizes all the text in the provided Reader.

Constructors in org.apache.lucene.analysis.fr with parameters of type TokenStream
`ElisionFilter(TokenStream input)` Constructs an elision filter with standard stop words
`ElisionFilter(TokenStream input, Set articles)` Constructs an elision filter with a Set of stop words
`ElisionFilter(TokenStream input, String[] articles)` Constructs an elision filter with an array of stop words
`FrenchStemFilter(TokenStream in)`
`FrenchStemFilter(TokenStream in, Set exclusiontable)`

Uses of TokenStream in org.apache.lucene.analysis.miscellaneous

Subclasses of TokenStream in org.apache.lucene.analysis.miscellaneous
`class` | `EmptyTokenStream` An always exhausted token stream.
`class` | `PrefixAndSuffixAwareTokenFilter` Links two PrefixAwareTokenFilter
`class` | `PrefixAwareTokenFilter` Joins two token streams and leaves the last token of the first stream available to be used when updating the token values in the second stream based on that token.
`class` | `SingleTokenTokenStream` A token stream containing a single token.

Methods in org.apache.lucene.analysis.miscellaneous that return TokenStream
`TokenStream` | `PrefixAwareTokenFilter.getPrefix()`
`TokenStream` | `PrefixAwareTokenFilter.getSuffix()`

Methods in org.apache.lucene.analysis.miscellaneous with parameters of type TokenStream
`void` | `PrefixAwareTokenFilter.setPrefix(TokenStream prefix)`
`void` | `PrefixAwareTokenFilter.setSuffix(TokenStream suffix)`

Constructors in org.apache.lucene.analysis.miscellaneous with parameters of type TokenStream
`PrefixAndSuffixAwareTokenFilter(TokenStream prefix, TokenStream input, TokenStream suffix)`
`PrefixAwareTokenFilter(TokenStream prefix, TokenStream suffix)`

Uses of TokenStream in org.apache.lucene.analysis.ngram

Subclasses of TokenStream in org.apache.lucene.analysis.ngram
`class` | `EdgeNGramTokenFilter` Tokenizes the given token into n-grams of given size(s).
`class` | `EdgeNGramTokenizer` Tokenizes the input from an edge into n-grams of given size(s).
`class` | `NGramTokenFilter` Tokenizes the input into n-grams of the given size(s).
`class` | `NGramTokenizer` Tokenizes the input into n-grams of the given size(s).

Constructors in org.apache.lucene.analysis.ngram with parameters of type TokenStream
`EdgeNGramTokenFilter(TokenStream input)`
`EdgeNGramTokenFilter(TokenStream input, EdgeNGramTokenFilter.Side side, int minGram, int maxGram)` Creates EdgeNGramTokenFilter that can generate n-grams in the sizes of the given range
`EdgeNGramTokenFilter(TokenStream input, String sideLabel, int minGram, int maxGram)` Creates EdgeNGramTokenFilter that can generate n-grams in the sizes of the given range
`NGramTokenFilter(TokenStream input)` Creates NGramTokenFilter with default min and max n-grams.
`NGramTokenFilter(TokenStream input, int minGram, int maxGram)` Creates NGramTokenFilter with given min and max n-grams.

Uses of TokenStream in org.apache.lucene.analysis.nl

Subclasses of TokenStream in org.apache.lucene.analysis.nl
`class` | `DutchStemFilter` A filter that stems Dutch words.

Methods in org.apache.lucene.analysis.nl that return TokenStream
`TokenStream` | `DutchAnalyzer.tokenStream(String fieldName, Reader reader)` Creates a TokenStream which tokenizes all the text in the provided TextReader.

Constructors in org.apache.lucene.analysis.nl with parameters of type TokenStream
`DutchStemFilter(TokenStream _in)`
`DutchStemFilter(TokenStream _in, Set exclusiontable)` Builds a DutchStemFilter that uses an exclusion table.
`DutchStemFilter(TokenStream _in, Set exclusiontable, Map stemdictionary)`

Uses of TokenStream in org.apache.lucene.analysis.payloads

Subclasses of TokenStream in org.apache.lucene.analysis.payloads
`class` | `NumericPayloadTokenFilter` Assigns a payload to a token based on the `Token.type()`
`class` | `TokenOffsetPayloadTokenFilter` Adds the `Token.setStartOffset(int)` and `Token.setEndOffset(int)` First 4 bytes are the start
`class` | `TypeAsPayloadTokenFilter` Makes the `Token.type()` a payload.

Constructors in org.apache.lucene.analysis.payloads with parameters of type TokenStream
`NumericPayloadTokenFilter(TokenStream input, float payload, String typeMatch)`
`TokenOffsetPayloadTokenFilter(TokenStream input)`
`TypeAsPayloadTokenFilter(TokenStream input)`

Uses of TokenStream in org.apache.lucene.analysis.query

Methods in org.apache.lucene.analysis.query that return TokenStream
`TokenStream` | `QueryAutoStopWordAnalyzer.tokenStream(String fieldName, Reader reader)`

Uses of TokenStream in org.apache.lucene.analysis.ru

Subclasses of TokenStream in org.apache.lucene.analysis.ru
`class` | `RussianLetterTokenizer` A RussianLetterTokenizer is a tokenizer that extends LetterTokenizer by additionally looking up letters in a given "russian charset".
`class` | `RussianLowerCaseFilter` Normalizes token text to lower case, analyzing given ("russian") charset.
`class` | `RussianStemFilter` A filter that stems Russian words.

Methods in org.apache.lucene.analysis.ru that return TokenStream
`TokenStream` | `RussianAnalyzer.tokenStream(String fieldName, Reader reader)` Creates a TokenStream which tokenizes all the text in the provided Reader.

Constructors in org.apache.lucene.analysis.ru with parameters of type TokenStream
`RussianLowerCaseFilter(TokenStream in, char[] charset)`
`RussianStemFilter(TokenStream in, char[] charset)`

Uses of TokenStream in org.apache.lucene.analysis.shingle

Subclasses of TokenStream in org.apache.lucene.analysis.shingle
`class` | `ShingleFilter` A ShingleFilter constructs shingles (token n-grams) from a token stream.
`class` | `ShingleMatrixFilter` A ShingleMatrixFilter constructs shingles (token n-grams) from a token stream.

Methods in org.apache.lucene.analysis.shingle that return TokenStream
`TokenStream` | `ShingleAnalyzerWrapper.tokenStream(String fieldName, Reader reader)`

Constructors in org.apache.lucene.analysis.shingle with parameters of type TokenStream
`ShingleFilter(TokenStream input)` Construct a ShingleFilter with default shingle size.
`ShingleFilter(TokenStream input, int maxShingleSize)` Constructs a ShingleFilter with the specified single size from the TokenStream `input`
`ShingleFilter(TokenStream input, String tokenType)` Construct a ShingleFilter with the specified token type for shingle tokens.
`ShingleMatrixFilter(TokenStream input, int minimumShingleSize, int maximumShingleSize)` Creates a shingle filter using default settings.
`ShingleMatrixFilter(TokenStream input, int minimumShingleSize, int maximumShingleSize, Character spacerCharacter)` Creates a shingle filter using default settings.
`ShingleMatrixFilter(TokenStream input, int minimumShingleSize, int maximumShingleSize, Character spacerCharacter, boolean ignoringSinglePrefixOrSuffixShingle)` Creates a shingle filter using the default `ShingleMatrixFilter.TokenSettingsCodec`.
`ShingleMatrixFilter(TokenStream input, int minimumShingleSize, int maximumShingleSize, Character spacerCharacter, boolean ignoringSinglePrefixOrSuffixShingle, ShingleMatrixFilter.TokenSettingsCodec settingsCodec)` Creates a shingle filter with ad hoc parameter settings.

Uses of TokenStream in org.apache.lucene.analysis.sinks

Subclasses of TokenStream in org.apache.lucene.analysis.sinks
`class` | `DateRecognizerSinkTokenizer` Attempts to parse the `Token.termBuffer()` as a Date using a `DateFormat`.
`class` | `TokenRangeSinkTokenizer` Counts the tokens as they go by and saves to the internal list those between the range of lower and upper, exclusive of upper
`class` | `TokenTypeSinkTokenizer` If the `Token.type()` matches the passed in `typeToMatch` then add it to the sink

Uses of TokenStream in org.apache.lucene.analysis.snowball

Subclasses of TokenStream in org.apache.lucene.analysis.snowball
`class` | `SnowballFilter` A filter that stems words using a Snowball-generated stemmer.

Methods in org.apache.lucene.analysis.snowball that return TokenStream
`TokenStream` | `SnowballAnalyzer.tokenStream(String fieldName, Reader reader)` Constructs a `StandardTokenizer` filtered by a `StandardFilter`, a `LowerCaseFilter` and a `StopFilter`.

Constructors in org.apache.lucene.analysis.snowball with parameters of type TokenStream
`SnowballFilter(TokenStream input, SnowballProgram stemmer)`
`SnowballFilter(TokenStream in, String name)` Construct the named stemming filter.

Uses of TokenStream in org.apache.lucene.analysis.standard

Subclasses of TokenStream in org.apache.lucene.analysis.standard
`class` | `StandardFilter` Normalizes tokens extracted with `StandardTokenizer`.
`class` | `StandardTokenizer` A grammar-based tokenizer constructed with JFlex

Methods in org.apache.lucene.analysis.standard that return TokenStream
`TokenStream` | `StandardAnalyzer.reusableTokenStream(String fieldName, Reader reader)`
`TokenStream` | `StandardAnalyzer.tokenStream(String fieldName, Reader reader)` Constructs a `StandardTokenizer` filtered by a `StandardFilter`, a `LowerCaseFilter` and a `StopFilter`.

Constructors in org.apache.lucene.analysis.standard with parameters of type TokenStream
`StandardFilter(TokenStream in)` Construct filtering in.

Uses of TokenStream in org.apache.lucene.analysis.th

Subclasses of TokenStream in org.apache.lucene.analysis.th
`class` | `ThaiWordFilter` TokenFilter that use java.text.BreakIterator to break each Token that is Thai into separate Token(s) for each Thai word.

Methods in org.apache.lucene.analysis.th that return TokenStream
`TokenStream` | `ThaiAnalyzer.tokenStream(String fieldName, Reader reader)`

Constructors in org.apache.lucene.analysis.th with parameters of type TokenStream
`ThaiWordFilter(TokenStream input)`

Uses of TokenStream in org.apache.lucene.benchmark.byTask.tasks

Methods in org.apache.lucene.benchmark.byTask.tasks with parameters of type TokenStream
`protected int` | `ReadTask.doHighlight(TokenStream ts, String text, Highlighter highlighter, boolean mergeContiguous, int maxFragments)`

Uses of TokenStream in org.apache.lucene.document

Methods in org.apache.lucene.document that return TokenStream
`TokenStream` | `Fieldable.tokenStreamValue()` The value of the field as a TokenStream, or null.
`TokenStream` | `Field.tokenStreamValue()` The value of the field as a TokesStream, or null.

Methods in org.apache.lucene.document with parameters of type TokenStream
`void` | `Field.setValue(TokenStream value)` Expert: change the value of this field.

Constructors in org.apache.lucene.document with parameters of type TokenStream
`Field(String name, TokenStream tokenStream)` Create a tokenized and indexed field that is not stored.
`Field(String name, TokenStream tokenStream, Field.TermVector termVector)` Create a tokenized and indexed field that is not stored, optionally with storing term vectors.

Uses of TokenStream in org.apache.lucene.index.memory

Subclasses of TokenStream in org.apache.lucene.index.memory
`class` | `SynonymTokenFilter` Injects additional tokens for synonyms of token terms fetched from the underlying child stream; the child stream must deliver lowercase tokens for synonyms to be found.

Methods in org.apache.lucene.index.memory that return TokenStream
`TokenStream` | `MemoryIndex.keywordTokenStream(Collection keywords)` Convenience method; Creates and returns a token stream that generates a token for each keyword in the given collection, "as is", without any transforming text analysis.
`TokenStream` | `PatternAnalyzer.tokenStream(String fieldName, Reader reader)` Creates a token stream that tokenizes all the text in the given Reader; This implementation forwards to `tokenStream(String, String)` and is less efficient than `tokenStream(String, String)`.
`TokenStream` | `PatternAnalyzer.tokenStream(String fieldName, String text)` Creates a token stream that tokenizes the given string into token terms (aka words).

Methods in org.apache.lucene.index.memory with parameters of type TokenStream
`void` | `MemoryIndex.addField(String fieldName, TokenStream stream)` Equivalent to `addField(fieldName, stream, 1.0f)`.
`void` | `MemoryIndex.addField(String fieldName, TokenStream stream, float boost)` Iterates over the given token stream and adds the resulting terms to the index; Equivalent to adding a tokenized, indexed, termVectorStored, unstored, Lucene `Field`.

Constructors in org.apache.lucene.index.memory with parameters of type TokenStream
`SynonymTokenFilter(TokenStream input, SynonymMap synonyms, int maxSynonyms)` Creates an instance for the given underlying stream and synonym table.

Uses of TokenStream in org.apache.lucene.search.highlight

Methods in org.apache.lucene.search.highlight that return TokenStream
`static TokenStream` | `TokenSources.getAnyTokenStream(IndexReader reader, int docId, String field, Analyzer analyzer)` A convenience method that tries a number of approaches to getting a token stream.
`static TokenStream` | `TokenSources.getAnyTokenStream(IndexReader reader, int docId, String field, Document doc, Analyzer analyzer)` A convenience method that tries to first get a TermPositionVector for the specified docId, then, falls back to using the passed in `Document` to retrieve the TokenStream.
`static TokenStream` | `TokenSources.getTokenStream(Document doc, String field, Analyzer analyzer)`
`static TokenStream` | `TokenSources.getTokenStream(IndexReader reader, int docId, String field)`
`static TokenStream` | `TokenSources.getTokenStream(IndexReader reader, int docId, String field, Analyzer analyzer)`
`static TokenStream` | `TokenSources.getTokenStream(String field, String contents, Analyzer analyzer)`
`static TokenStream` | `TokenSources.getTokenStream(TermPositionVector tpv)`
`static TokenStream` | `TokenSources.getTokenStream(TermPositionVector tpv, boolean tokenPositionsGuaranteedContiguous)` Low level api.

Methods in org.apache.lucene.search.highlight with parameters of type TokenStream
`String` | `Highlighter.getBestFragment(TokenStream tokenStream, String text)` Highlights chosen terms in a text, extracting the most relevant section.
`String[]` | `Highlighter.getBestFragments(TokenStream tokenStream, String text, int maxNumFragments)` Highlights chosen terms in a text, extracting the most relevant sections.
`String` | `Highlighter.getBestFragments(TokenStream tokenStream, String text, int maxNumFragments, String separator)` Highlights terms in the text , extracting the most relevant sections and concatenating the chosen fragments with a separator (typically "...").
`TextFragment[]` | `Highlighter.getBestTextFragments(TokenStream tokenStream, String text, boolean mergeContiguousFragments, int maxNumFragments)` Low level api to get the most relevant (formatted) sections of the document.
`Map` | `WeightedSpanTermExtractor.getWeightedSpanTermsWithScores(Query query, TokenStream tokenStream, String fieldName, IndexReader reader)` Creates a Map of `WeightedSpanTerms` from the given `Query` and `TokenStream`.

Uses of TokenStream in org.apache.lucene.wikipedia.analysis

Subclasses of TokenStream in org.apache.lucene.wikipedia.analysis
`class` | `WikipediaTokenizer` Extension of StandardTokenizer that is aware of Wikipedia syntax.

Overview | Package | Class | Use | Tree | Deprecated | Index | Help
PREV NEXT | FRAMES NO FRAMES

Copyright © 2000-2009 Apache Software Foundation. All Rights Reserved.