org.apache.solr.analysis

Class PatternTokenizer

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.Tokenizer

org.apache.solr.analysis.PatternTokenizer

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

public final class PatternTokenizer
extends org.apache.lucene.analysis.Tokenizer

This tokenizer uses regex pattern matching to construct distinct tokens for the input stream. It takes two arguments: "pattern" and "group".

• "pattern" is the regular expression.
• "group" says which group to extract into tokens.

group=-1 (the default) is equivalent to "split". In this case, the tokens will be equivalent to the output from (without empty tokens): String.split(java.lang.String)

Using group >= 0 selects the matching group as the token. For example, if you have:

  pattern = \'([^\']+)\'
  group = 0
  input = aaa 'bbb' 'ccc'

the output will be two tokens: 'bbb' and 'ccc' (including the ' marks). With the same input but using group=1, the output would be: bbb and ccc (no ' marks)

NOTE: This Tokenizer does not output tokens that are of zero length.

See Also:

Pattern

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State

Field Summary

Fields

Modifier and Type	Field and Description
(package private) char[]	buffer
private int	group
private int	index
private java.util.regex.Matcher	matcher
private org.apache.lucene.analysis.tokenattributes.OffsetAttribute	offsetAtt
private java.util.regex.Pattern	pattern
private java.lang.StringBuilder	str
private org.apache.lucene.analysis.tokenattributes.CharTermAttribute	termAtt

Fields inherited from class org.apache.lucene.analysis.Tokenizer

input

Constructor Summary

Constructors

Constructor and Description
PatternTokenizer(java.io.Reader input, java.util.regex.Pattern pattern, int group) creates a new PatternTokenizer returning tokens from group (-1 for split functionality)

Method Summary

Modifier and Type	Method and Description
void	end()
private void	fillBuffer(java.lang.StringBuilder sb, java.io.Reader input)
boolean	incrementToken()
void	reset(java.io.Reader input)

Methods inherited from class org.apache.lucene.analysis.Tokenizer

close, correctOffset

Methods inherited from class org.apache.lucene.analysis.TokenStream

reset

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

termAtt

private final org.apache.lucene.analysis.tokenattributes.CharTermAttribute termAtt

offsetAtt

private final org.apache.lucene.analysis.tokenattributes.OffsetAttribute offsetAtt

str

private final java.lang.StringBuilder str

index

private int index

pattern

private final java.util.regex.Pattern pattern

group

private final int group

matcher

private final java.util.regex.Matcher matcher

buffer

final char[] buffer

Constructor Detail

PatternTokenizer

public PatternTokenizer(java.io.Reader input,
                        java.util.regex.Pattern pattern,
                        int group)
                 throws java.io.IOException

creates a new PatternTokenizer returning tokens from group (-1 for split functionality)

Throws:

java.io.IOException

Method Detail

incrementToken

public boolean incrementToken()
                       throws java.io.IOException

Specified by:

incrementToken in class org.apache.lucene.analysis.TokenStream

Throws:

java.io.IOException

end

public void end()
         throws java.io.IOException

Overrides:

end in class org.apache.lucene.analysis.TokenStream

Throws:

java.io.IOException

reset

public void reset(java.io.Reader input)
           throws java.io.IOException

Overrides:

reset in class org.apache.lucene.analysis.Tokenizer

Throws:

java.io.IOException

fillBuffer

private void fillBuffer(java.lang.StringBuilder sb,
                        java.io.Reader input)
                 throws java.io.IOException

Throws:

java.io.IOException