StandardTokenizer
implements the Word Break rules from the Unicode Text Segmentation algorithm, as specified in
Unicode Standard Annex #29.See: Description
| Class | Description |
|---|---|
| ClassicAnalyzer |
Filters
ClassicTokenizer with ClassicFilter, LowerCaseFilter and StopFilter, using a list of
English stop words. |
| ClassicFilter |
Normalizes tokens extracted with
ClassicTokenizer. |
| ClassicFilterFactory |
Factory for
ClassicFilter. |
| ClassicTokenizer |
A grammar-based tokenizer constructed with JFlex
|
| ClassicTokenizerFactory |
Factory for
ClassicTokenizer. |
| ClassicTokenizerImpl |
This class implements the classic lucene StandardTokenizer up until 3.0
|
| StandardAnalyzer |
Filters
StandardTokenizer with LowerCaseFilter and
StopFilter, using a configurable list of stop words. |
| StandardTokenizer |
A grammar-based tokenizer constructed with JFlex.
|
| StandardTokenizerFactory |
Factory for
StandardTokenizer. |
| StandardTokenizerImpl |
This class implements Word Break rules from the Unicode Text Segmentation
algorithm, as specified in
Unicode Standard Annex #29.
|
| UAX29URLEmailAnalyzer |
Filters
UAX29URLEmailTokenizer
with LowerCaseFilter and
StopFilter, using a list of
English stop words. |
| UAX29URLEmailTokenizer |
This class implements Word Break rules from the Unicode Text Segmentation
algorithm, as specified in
Unicode Standard Annex #29
URLs and email addresses are also tokenized according to the relevant RFCs.
|
| UAX29URLEmailTokenizerFactory |
Factory for
UAX29URLEmailTokenizer. |
| UAX29URLEmailTokenizerImpl |
This class implements Word Break rules from the Unicode Text Segmentation
algorithm, as specified in
Unicode Standard Annex #29
URLs and email addresses are also tokenized according to the relevant RFCs.
|
StandardTokenizer
implements the Word Break rules from the Unicode Text Segmentation algorithm, as specified in
Unicode Standard Annex #29.
Unlike UAX29URLEmailTokenizer from the analysis module, URLs and email addresses are
not tokenized as single tokens, but are instead split up into
tokens according to the UAX#29 word break rules.
StandardAnalyzer includes
StandardTokenizer,
LowerCaseFilter
and StopFilter.