org.apache.lucene.analysis.core

Class WhitespaceTokenizerFactory

java.lang.Object

org.apache.lucene.analysis.util.AbstractAnalysisFactory

org.apache.lucene.analysis.util.TokenizerFactory

org.apache.lucene.analysis.core.WhitespaceTokenizerFactory

public class WhitespaceTokenizerFactory
extends TokenizerFactory

Factory for WhitespaceTokenizer.

 <fieldType name="text_ws" class="solr.TextField" positionIncrementGap="100">
   <analyzer>
     <tokenizer class="solr.WhitespaceTokenizerFactory" rule="unicode"  maxTokenLen="256"/>
   </analyzer>
 </fieldType>

Options:

• rule: either "java" for WhitespaceTokenizer or "unicode" for UnicodeWhitespaceTokenizer
• maxTokenLen: max token length, should be greater than 0 and less than MAX_TOKEN_LENGTH_LIMIT (1024*1024). It is rare to need to change this else CharTokenizer::DEFAULT_MAX_TOKEN_LEN

Since:

3.1

Field Summary

Fields

Modifier and Type	Field and Description
private int	maxTokenLen
private java.lang.String	rule
static java.lang.String	RULE_JAVA
private static java.util.Collection<java.lang.String>	RULE_NAMES
static java.lang.String	RULE_UNICODE

Fields inherited from class org.apache.lucene.analysis.util.AbstractAnalysisFactory

LUCENE_MATCH_VERSION_PARAM, luceneMatchVersion

Constructor Summary

Constructors

Constructor and Description
WhitespaceTokenizerFactory(java.util.Map<java.lang.String,java.lang.String> args) Creates a new WhitespaceTokenizerFactory

Method Summary

Modifier and Type	Method and Description
Tokenizer	create(AttributeFactory factory) Creates a TokenStream of the specified input using the given AttributeFactory

Methods inherited from class org.apache.lucene.analysis.util.TokenizerFactory

availableTokenizers, create, forName, lookupClass, reloadTokenizers

Methods inherited from class org.apache.lucene.analysis.util.AbstractAnalysisFactory

get, get, get, get, get, getBoolean, getChar, getClassArg, getFloat, getInt, getLines, getLuceneMatchVersion, getOriginalArgs, getPattern, getSet, getSnowballWordSet, getWordSet, isExplicitLuceneMatchVersion, require, require, require, requireBoolean, requireChar, requireFloat, requireInt, setExplicitLuceneMatchVersion, splitAt, splitFileNames

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

RULE_JAVA

public static final java.lang.String RULE_JAVA

See Also:

Constant Field Values

RULE_UNICODE

public static final java.lang.String RULE_UNICODE

See Also:

Constant Field Values

RULE_NAMES

private static final java.util.Collection<java.lang.String> RULE_NAMES

rule

private final java.lang.String rule

maxTokenLen

private final int maxTokenLen

Constructor Detail

WhitespaceTokenizerFactory

public WhitespaceTokenizerFactory(java.util.Map<java.lang.String,java.lang.String> args)

Creates a new WhitespaceTokenizerFactory

Method Detail

create

public Tokenizer create(AttributeFactory factory)

Description copied from class: TokenizerFactory

Creates a TokenStream of the specified input using the given AttributeFactory

Specified by:

create in class TokenizerFactory