PrefixAwareTokenFilter

java.lang.Object
- org.apache.lucene.util.AttributeSource
- - org.apache.lucene.analysis.TokenStream
  - - org.apache.lucene.analysis.miscellaneous.PrefixAwareTokenFilter

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable
```
public class PrefixAwareTokenFilter
extends TokenStream
```
Joins two token streams and leaves the last token of the first stream available to be used when updating the token values in the second stream based on that token. The default implementation adds last prefix token end offset to the suffix token start and end offsets.
NOTE: This filter might not behave correctly if used with custom Attributes, i.e. Attributes other than the ones located in org.apache.lucene.analysis.tokenattributes.

Nested Class Summary
- Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
  AttributeSource.State

Field Summary

Fields
Modifier and Type	Field and Description
`private FlagsAttribute`	`flagsAtt`
`private OffsetAttribute`	`offsetAtt`
`private FlagsAttribute`	`p_flagsAtt`
`private OffsetAttribute`	`p_offsetAtt`
`private PayloadAttribute`	`p_payloadAtt`
`private PositionIncrementAttribute`	`p_posIncrAtt`
`private CharTermAttribute`	`p_termAtt`
`private TypeAttribute`	`p_typeAtt`
`private PayloadAttribute`	`payloadAtt`
`private PositionIncrementAttribute`	`posIncrAtt`
`private TokenStream`	`prefix`
`private boolean`	`prefixExhausted`
`private Token`	`previousPrefixToken`
`private Token`	`reusableToken`
`private TokenStream`	`suffix`
`private CharTermAttribute`	`termAtt`
`private TypeAttribute`	`typeAtt`

Fields inherited from class org.apache.lucene.analysis.TokenStream
DEFAULT_TOKEN_ATTRIBUTE_FACTORY

Constructor Summary

Constructors
Constructor and Description

PrefixAwareTokenFilter(TokenStream prefix, TokenStream suffix)

Constructors
Constructor and Description
`PrefixAwareTokenFilter(TokenStream prefix, TokenStream suffix)`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`close()` Releases resources associated with this stream.
`void`	`end()` This method is called by the consumer after the last token has been consumed, after `TokenStream.incrementToken()` returned `false` (using the new `TokenStream` API).
`private Token`	`getNextPrefixInputToken(Token token)`
`private Token`	`getNextSuffixInputToken(Token token)`
`TokenStream`	`getPrefix()`
`TokenStream`	`getSuffix()`
`boolean`	`incrementToken()` Consumers (i.e., `IndexWriter`) use this method to advance the stream to the next token.
`void`	`reset()` This method is called by a consumer before it begins consumption using `TokenStream.incrementToken()`.
`private void`	`setCurrentToken(Token token)`
`void`	`setPrefix(TokenStream prefix)`
`void`	`setSuffix(TokenStream suffix)`
`Token`	`updateSuffixToken(Token suffixToken, Token lastPrefixToken)` The default implementation adds last prefix token end offset to the suffix token start and end offsets.

Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString

Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait

- Field Detail
  - prefix
```
private TokenStream prefix
```
  - suffix
```
private TokenStream suffix
```
  - termAtt
```
private CharTermAttribute termAtt
```
  - posIncrAtt
```
private PositionIncrementAttribute posIncrAtt
```
  - payloadAtt
```
private PayloadAttribute payloadAtt
```
  - offsetAtt
```
private OffsetAttribute offsetAtt
```
  - typeAtt
```
private TypeAttribute typeAtt
```
  - flagsAtt
```
private FlagsAttribute flagsAtt
```
  - p_termAtt
```
private CharTermAttribute p_termAtt
```
  - p_posIncrAtt
```
private PositionIncrementAttribute p_posIncrAtt
```
  - p_payloadAtt
```
private PayloadAttribute p_payloadAtt
```
  - p_offsetAtt
```
private OffsetAttribute p_offsetAtt
```
  - p_typeAtt
```
private TypeAttribute p_typeAtt
```
  - p_flagsAtt
```
private FlagsAttribute p_flagsAtt
```
  - previousPrefixToken
```
private Token previousPrefixToken
```
  - reusableToken
```
private Token reusableToken
```
  - prefixExhausted
```
private boolean prefixExhausted
```
- Constructor Detail
  - PrefixAwareTokenFilter
```
public PrefixAwareTokenFilter(TokenStream prefix,
                              TokenStream suffix)
```
- Method Detail
  - incrementToken
```
public final boolean incrementToken()
                             throws java.io.IOException
```
    Description copied from class: TokenStream
    
    Consumers (i.e., IndexWriter) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate AttributeImpls with the attributes of the next token.
    The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use AttributeSource.captureState() to create a copy of the current attribute state.
    This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to AttributeSource.addAttribute(Class) and AttributeSource.getAttribute(Class), references to all AttributeImpls that this stream uses should be retrieved during instantiation.
    To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in TokenStream.incrementToken().
    
    Specified by:
    
    incrementToken in class TokenStream
    
    Returns:
    
    false for end of stream; true otherwise
    
    Throws:
    
    java.io.IOException
  - setCurrentToken
```
private void setCurrentToken(Token token)
```
  - getNextPrefixInputToken
```
private Token getNextPrefixInputToken(Token token)
                               throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - getNextSuffixInputToken
```
private Token getNextSuffixInputToken(Token token)
                               throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - updateSuffixToken
```
public Token updateSuffixToken(Token suffixToken,
                               Token lastPrefixToken)
```
    The default implementation adds last prefix token end offset to the suffix token start and end offsets.
    
    Parameters:
    
    suffixToken - a token from the suffix stream
    
    lastPrefixToken - the last token from the prefix stream
    
    Returns:
    
    consumer token
  - end
```
public void end()
         throws java.io.IOException
```
    Description copied from class: TokenStream
    
    This method is called by the consumer after the last token has been consumed, after TokenStream.incrementToken() returned false (using the new TokenStream API). Streams implementing the old API should upgrade to use this feature.
    This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but a WhitespaceTokenizer was used.
    Additionally any skipped positions (such as those removed by a stopfilter) can be applied to the position increment, or any adjustment of other attributes where the end-of-stream value may be important.
    If you override this method, always call super.end().
    
    Overrides:
    
    end in class TokenStream
    
    Throws:
    
    java.io.IOException - If an I/O error occurs
  - close
```
public void close()
           throws java.io.IOException
```
    Description copied from class: TokenStream
    
    Releases resources associated with this stream.
    If you override this method, always call super.close(), otherwise some internal state will not be correctly reset (e.g., Tokenizer will throw IllegalStateException on reuse).
    
    Specified by:
    
    close in interface java.io.Closeable
    
    Specified by:
    
    close in interface java.lang.AutoCloseable
    
    Overrides:
    
    close in class TokenStream
    
    Throws:
    
    java.io.IOException
  - reset
```
public void reset()
           throws java.io.IOException
```
    Description copied from class: TokenStream
    
    This method is called by a consumer before it begins consumption using TokenStream.incrementToken().
    Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.
    If you override this method, always call super.reset(), otherwise some internal state will not be correctly reset (e.g., Tokenizer will throw IllegalStateException on further usage).
    
    Overrides:
    
    reset in class TokenStream
    
    Throws:
    
    java.io.IOException
  - getPrefix
```
public TokenStream getPrefix()
```
  - setPrefix
```
public void setPrefix(TokenStream prefix)
```
  - getSuffix
```
public TokenStream getSuffix()
```
  - setSuffix
```
public void setSuffix(TokenStream suffix)
```

Class PrefixAwareTokenFilter

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

Field Summary

Fields inherited from class org.apache.lucene.analysis.TokenStream

Constructor Summary

Method Summary

Methods inherited from class org.apache.lucene.util.AttributeSource

Methods inherited from class java.lang.Object

Field Detail

prefix

suffix

termAtt

posIncrAtt

payloadAtt

offsetAtt

typeAtt

flagsAtt

p_termAtt

p_posIncrAtt

p_payloadAtt

p_offsetAtt

p_typeAtt

p_flagsAtt

previousPrefixToken

reusableToken

prefixExhausted

Constructor Detail

PrefixAwareTokenFilter

Method Detail

incrementToken

setCurrentToken

getNextPrefixInputToken

getNextSuffixInputToken

updateSuffixToken

end

close

reset

getPrefix

setPrefix

getSuffix

setSuffix