org.apache.solr.analysis

Class PatternTokenizerFactory

java.lang.Object

org.apache.solr.analysis.BaseTokenStreamFactory

org.apache.solr.analysis.BaseTokenizerFactory

org.apache.solr.analysis.PatternTokenizerFactory

All Implemented Interfaces:

TokenizerFactory

public class PatternTokenizerFactory
extends BaseTokenizerFactory

Factory for PatternTokenizer. This tokenizer uses regex pattern matching to construct distinct tokens for the input stream. It takes two arguments: "pattern" and "group".

• "pattern" is the regular expression.
• "group" says which group to extract into tokens.

group=-1 (the default) is equivalent to "split". In this case, the tokens will be equivalent to the output from (without empty tokens): String.split(java.lang.String)

Using group >= 0 selects the matching group as the token. For example, if you have:

  pattern = \'([^\']+)\'
  group = 0
  input = aaa 'bbb' 'ccc'

the output will be two tokens: 'bbb' and 'ccc' (including the ' marks). With the same input but using group=1, the output would be: bbb and ccc (no ' marks)

NOTE: This Tokenizer does not output tokens that are of zero length.

 <fieldType name="text_ptn" class="solr.TextField" positionIncrementGap="100">
   <analyzer>
     <tokenizer class="solr.PatternTokenizerFactory" pattern="\'([^\']+)\'" group="1"/>
   </analyzer>
 </fieldType>

Since:

solr1.2

Version:

$Id:$

See Also:

PatternTokenizer

Field Summary

Fields

Modifier and Type	Field and Description
protected int	group
static java.lang.String	GROUP
protected java.util.regex.Pattern	pattern
static java.lang.String	PATTERN

Fields inherited from class org.apache.solr.analysis.BaseTokenizerFactory

log

Fields inherited from class org.apache.solr.analysis.BaseTokenStreamFactory

args, luceneMatchVersion

Constructor Summary

Constructors

Constructor and Description
PatternTokenizerFactory()

Method Summary

Modifier and Type	Method and Description
org.apache.lucene.analysis.Tokenizer	create(java.io.Reader in) Split the input using configured pattern
static java.util.List<org.apache.lucene.analysis.Token>	group(java.util.regex.Matcher matcher, java.lang.String input, int group) Deprecated.
void	init(java.util.Map<java.lang.String,java.lang.String> args) Require a configured pattern
static java.util.List<org.apache.lucene.analysis.Token>	split(java.util.regex.Matcher matcher, java.lang.String input) Deprecated.

Methods inherited from class org.apache.solr.analysis.BaseTokenStreamFactory

assureMatchVersion, getArgs, getBoolean, getBoolean, getInt, getInt, getInt, getSnowballWordSet, getWordSet, warnDeprecated

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.solr.analysis.TokenizerFactory

getArgs

Field Detail

PATTERN

public static final java.lang.String PATTERN

See Also:

Constant Field Values

GROUP

public static final java.lang.String GROUP

See Also:

Constant Field Values

pattern

protected java.util.regex.Pattern pattern

group

protected int group

Constructor Detail

PatternTokenizerFactory

public PatternTokenizerFactory()

Method Detail

init

public void init(java.util.Map<java.lang.String,java.lang.String> args)

Require a configured pattern

Specified by:

init in interface TokenizerFactory

Overrides:

init in class BaseTokenStreamFactory

create

public org.apache.lucene.analysis.Tokenizer create(java.io.Reader in)

Split the input using configured pattern

split

@Deprecated
public static java.util.List<org.apache.lucene.analysis.Token> split(java.util.regex.Matcher matcher,
                                                                                 java.lang.String input)

Deprecated.

This behaves just like String.split( ), but returns a list of Tokens rather then an array of strings NOTE: This method is not used in 1.4.

group

@Deprecated
public static java.util.List<org.apache.lucene.analysis.Token> group(java.util.regex.Matcher matcher,
                                                                                 java.lang.String input,
                                                                                 int group)

Deprecated.

Create tokens from the matches in a matcher NOTE: This method is not used in 1.4.