org.apache.lucene.search.postingshighlight

Class PostingsHighlighter

java.lang.Object

org.apache.lucene.search.postingshighlight.PostingsHighlighter

public class PostingsHighlighter
extends java.lang.Object

Simple highlighter that does not analyze fields nor use term vectors. Instead it requires IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS.

PostingsHighlighter treats the single original document as the whole corpus, and then scores individual passages as if they were documents in this corpus. It uses a BreakIterator to find passages in the text; by default it breaks using getSentenceInstance(Locale.ROOT). It then iterates in parallel (merge sorting by offset) through the positions of all terms from the query, coalescing those hits that occur in a single passage into a Passage, and then scores each Passage using a separate PassageScorer. Passages are finally formatted into highlighted snippets with a PassageFormatter.

You can customize the behavior by subclassing this highlighter, some important hooks:

• getBreakIterator(String): Customize how the text is divided into passages.
• getScorer(String): Customize how passages are ranked.
• getFormatter(String): Customize how snippets are formatted.
• getIndexAnalyzer(String): Enable highlighting of MultiTermQuerys such as WildcardQuery.

WARNING: The code is very new and probably still has some exciting bugs!

Example usage:

   // configure field with offsets at index time
   FieldType offsetsType = new FieldType(TextField.TYPE_STORED);
   offsetsType.setIndexOptions(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS);
   Field body = new Field("body", "foobar", offsetsType);

   // retrieve highlights at query time 
   PostingsHighlighter highlighter = new PostingsHighlighter();
   Query query = new TermQuery(new Term("body", "highlighting"));
   TopDocs topDocs = searcher.search(query, n);
   String highlights[] = highlighter.highlight("body", query, searcher, topDocs);
 

This is thread-safe, and can be used across different readers.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	PostingsHighlighter.LimitedStoredFieldVisitor
private static class	PostingsHighlighter.OffsetsEnum

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_MAX_LENGTH Default maximum content size to process.
private PassageFormatter	defaultFormatter Set the first time getFormatter(java.lang.String) is called, and then reused.
private PassageScorer	defaultScorer Set the first time getScorer(java.lang.String) is called, and then reused.
private static PostingsEnum	EMPTY
private static IndexSearcher	EMPTY_INDEXSEARCHER for rewriting: we don't want slow processing from MTQs
private int	maxLength

Constructor Summary

Constructors

Constructor and Description
PostingsHighlighter() Creates a new highlighter with DEFAULT_MAX_LENGTH.
PostingsHighlighter(int maxLength) Creates a new highlighter, specifying maximum content length.

Method Summary

Modifier and Type	Method and Description
protected java.text.BreakIterator	getBreakIterator(java.lang.String field) Returns the BreakIterator to use for dividing text into passages.
protected Passage[]	getEmptyHighlight(java.lang.String fieldName, java.text.BreakIterator bi, int maxPassages) Called to summarize a document when no hits were found.
protected PassageFormatter	getFormatter(java.lang.String field) Returns the PassageFormatter to use for formatting passages into highlighted snippets.
protected Analyzer	getIndexAnalyzer(java.lang.String field) Returns the analyzer originally used to index the content for field.
protected char	getMultiValuedSeparator(java.lang.String field) Returns the logical separator between values for multi-valued fields.
protected PassageScorer	getScorer(java.lang.String field) Returns the PassageScorer to use for ranking passages.
java.lang.String[]	highlight(java.lang.String field, Query query, IndexSearcher searcher, TopDocs topDocs) Highlights the top passages from a single field.
java.lang.String[]	highlight(java.lang.String field, Query query, IndexSearcher searcher, TopDocs topDocs, int maxPassages) Highlights the top-N passages from a single field.
private Passage[]	highlightDoc(java.lang.String field, BytesRef[] terms, int contentLength, java.text.BreakIterator bi, int doc, TermsEnum termsEnum, PostingsEnum[] postings, int n)
private java.util.Map<java.lang.Integer,java.lang.Object>	highlightField(java.lang.String field, java.lang.String[] contents, java.text.BreakIterator bi, BytesRef[] terms, int[] docids, java.util.List<LeafReaderContext> leaves, int maxPassages, Query query)
java.util.Map<java.lang.String,java.lang.String[]>	highlightFields(java.lang.String[] fieldsIn, Query query, IndexSearcher searcher, int[] docidsIn, int[] maxPassagesIn) Highlights the top-N passages from multiple fields, for the provided int[] docids.
java.util.Map<java.lang.String,java.lang.String[]>	highlightFields(java.lang.String[] fields, Query query, IndexSearcher searcher, TopDocs topDocs) Highlights the top passages from multiple fields.
java.util.Map<java.lang.String,java.lang.String[]>	highlightFields(java.lang.String[] fields, Query query, IndexSearcher searcher, TopDocs topDocs, int[] maxPassages) Highlights the top-N passages from multiple fields.
protected java.util.Map<java.lang.String,java.lang.Object[]>	highlightFieldsAsObjects(java.lang.String[] fieldsIn, Query query, IndexSearcher searcher, int[] docidsIn, int[] maxPassagesIn) Expert: highlights the top-N passages from multiple fields, for the provided int[] docids, to custom Object as returned by the PassageFormatter.
protected java.lang.String[][]	loadFieldValues(IndexSearcher searcher, java.lang.String[] fields, int[] docids, int maxLength) Loads the String values for each field X docID to be highlighted.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

EMPTY_INDEXSEARCHER

private static final IndexSearcher EMPTY_INDEXSEARCHER

for rewriting: we don't want slow processing from MTQs

DEFAULT_MAX_LENGTH

public static final int DEFAULT_MAX_LENGTH

Default maximum content size to process. Typically snippets closer to the beginning of the document better summarize its content

See Also:

Constant Field Values

maxLength

private final int maxLength

defaultFormatter

private PassageFormatter defaultFormatter

Set the first time getFormatter(java.lang.String) is called, and then reused.

defaultScorer

private PassageScorer defaultScorer

Set the first time getScorer(java.lang.String) is called, and then reused.

EMPTY

private static final PostingsEnum EMPTY

Constructor Detail

PostingsHighlighter

public PostingsHighlighter()

Creates a new highlighter with DEFAULT_MAX_LENGTH.

PostingsHighlighter

public PostingsHighlighter(int maxLength)

Creates a new highlighter, specifying maximum content length.

Parameters:

maxLength - maximum content size to process.

Throws:

java.lang.IllegalArgumentException - if maxLength is negative or Integer.MAX_VALUE

Method Detail

getBreakIterator

protected java.text.BreakIterator getBreakIterator(java.lang.String field)

Returns the BreakIterator to use for dividing text into passages. This returns BreakIterator.getSentenceInstance(Locale) by default; subclasses can override to customize.

getFormatter

protected PassageFormatter getFormatter(java.lang.String field)

Returns the PassageFormatter to use for formatting passages into highlighted snippets. This returns a new PassageFormatter by default; subclasses can override to customize.

getScorer

protected PassageScorer getScorer(java.lang.String field)

Returns the PassageScorer to use for ranking passages. This returns a new PassageScorer by default; subclasses can override to customize.

highlight

public java.lang.String[] highlight(java.lang.String field,
                                    Query query,
                                    IndexSearcher searcher,
                                    TopDocs topDocs)
                             throws java.io.IOException

Highlights the top passages from a single field.

Parameters:

field - field name to highlight. Must have a stored string value and also be indexed with offsets.

query - query to highlight.

searcher - searcher that was previously used to execute the query.

topDocs - TopDocs containing the summary result documents to highlight.

Returns:

Array of formatted snippets corresponding to the documents in topDocs. If no highlights were found for a document, the first sentence for the field will be returned.

Throws:

java.io.IOException - if an I/O error occurred during processing

java.lang.IllegalArgumentException - if field was indexed without IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS

highlight

public java.lang.String[] highlight(java.lang.String field,
                                    Query query,
                                    IndexSearcher searcher,
                                    TopDocs topDocs,
                                    int maxPassages)
                             throws java.io.IOException

Highlights the top-N passages from a single field.

Parameters:

field - field name to highlight. Must have a stored string value and also be indexed with offsets.

query - query to highlight.

searcher - searcher that was previously used to execute the query.

topDocs - TopDocs containing the summary result documents to highlight.

maxPassages - The maximum number of top-N ranked passages used to form the highlighted snippets.

Returns:

Array of formatted snippets corresponding to the documents in topDocs. If no highlights were found for a document, the first maxPassages sentences from the field will be returned.

Throws:

java.io.IOException - if an I/O error occurred during processing

java.lang.IllegalArgumentException - if field was indexed without IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS

highlightFields

public java.util.Map<java.lang.String,java.lang.String[]> highlightFields(java.lang.String[] fields,
                                                                          Query query,
                                                                          IndexSearcher searcher,
                                                                          TopDocs topDocs)
                                                                   throws java.io.IOException

Highlights the top passages from multiple fields.

Conceptually, this behaves as a more efficient form of:

 Map m = new HashMap();
 for (String field : fields) {
   m.put(field, highlight(field, query, searcher, topDocs));
 }
 return m;
 

Parameters:

fields - field names to highlight. Must have a stored string value and also be indexed with offsets.

query - query to highlight.

searcher - searcher that was previously used to execute the query.

topDocs - TopDocs containing the summary result documents to highlight.

Returns:

Map keyed on field name, containing the array of formatted snippets corresponding to the documents in topDocs. If no highlights were found for a document, the first sentence from the field will be returned.

Throws:

java.io.IOException - if an I/O error occurred during processing

java.lang.IllegalArgumentException - if field was indexed without IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS

highlightFields

public java.util.Map<java.lang.String,java.lang.String[]> highlightFields(java.lang.String[] fields,
                                                                          Query query,
                                                                          IndexSearcher searcher,
                                                                          TopDocs topDocs,
                                                                          int[] maxPassages)
                                                                   throws java.io.IOException

Highlights the top-N passages from multiple fields.

Conceptually, this behaves as a more efficient form of:

 Map m = new HashMap();
 for (String field : fields) {
   m.put(field, highlight(field, query, searcher, topDocs, maxPassages));
 }
 return m;
 

Parameters:

fields - field names to highlight. Must have a stored string value and also be indexed with offsets.

query - query to highlight.

searcher - searcher that was previously used to execute the query.

topDocs - TopDocs containing the summary result documents to highlight.

maxPassages - The maximum number of top-N ranked passages per-field used to form the highlighted snippets.

Returns:

Map keyed on field name, containing the array of formatted snippets corresponding to the documents in topDocs. If no highlights were found for a document, the first maxPassages sentences from the field will be returned.

Throws:

java.io.IOException - if an I/O error occurred during processing

java.lang.IllegalArgumentException - if field was indexed without IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS

highlightFields

public java.util.Map<java.lang.String,java.lang.String[]> highlightFields(java.lang.String[] fieldsIn,
                                                                          Query query,
                                                                          IndexSearcher searcher,
                                                                          int[] docidsIn,
                                                                          int[] maxPassagesIn)
                                                                   throws java.io.IOException

Highlights the top-N passages from multiple fields, for the provided int[] docids.

Parameters:

fieldsIn - field names to highlight. Must have a stored string value and also be indexed with offsets.

query - query to highlight.

searcher - searcher that was previously used to execute the query.

docidsIn - containing the document IDs to highlight.

maxPassagesIn - The maximum number of top-N ranked passages per-field used to form the highlighted snippets.

Returns:

Map keyed on field name, containing the array of formatted snippets corresponding to the documents in docidsIn. If no highlights were found for a document, the first maxPassages from the field will be returned.

Throws:

java.io.IOException - if an I/O error occurred during processing

java.lang.IllegalArgumentException - if field was indexed without IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS

highlightFieldsAsObjects

protected java.util.Map<java.lang.String,java.lang.Object[]> highlightFieldsAsObjects(java.lang.String[] fieldsIn,
                                                                                      Query query,
                                                                                      IndexSearcher searcher,
                                                                                      int[] docidsIn,
                                                                                      int[] maxPassagesIn)
                                                                               throws java.io.IOException

Expert: highlights the top-N passages from multiple fields, for the provided int[] docids, to custom Object as returned by the PassageFormatter. Use this API to render to something other than String.

Parameters:

fieldsIn - field names to highlight. Must have a stored string value and also be indexed with offsets.

query - query to highlight.

searcher - searcher that was previously used to execute the query.

docidsIn - containing the document IDs to highlight.

maxPassagesIn - The maximum number of top-N ranked passages per-field used to form the highlighted snippets.

Returns:

Map keyed on field name, containing the array of formatted snippets corresponding to the documents in docidsIn. If no highlights were found for a document, the first maxPassages from the field will be returned.

Throws:

java.io.IOException - if an I/O error occurred during processing

java.lang.IllegalArgumentException - if field was indexed without IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS

loadFieldValues

protected java.lang.String[][] loadFieldValues(IndexSearcher searcher,
                                               java.lang.String[] fields,
                                               int[] docids,
                                               int maxLength)
                                        throws java.io.IOException

Loads the String values for each field X docID to be highlighted. By default this loads from stored fields, but a subclass can change the source. This method should allocate the String[fields.length][docids.length] and fill all values. The returned Strings must be identical to what was indexed.

Throws:

java.io.IOException

getMultiValuedSeparator

protected char getMultiValuedSeparator(java.lang.String field)

Returns the logical separator between values for multi-valued fields. The default value is a space character, which means passages can span across values, but a subclass can override, for example with U+2029 PARAGRAPH SEPARATOR (PS) if each value holds a discrete passage for highlighting.

getIndexAnalyzer

protected Analyzer getIndexAnalyzer(java.lang.String field)

Returns the analyzer originally used to index the content for field.

This is used to highlight some MultiTermQueries.

Returns:

Analyzer or null (the default, meaning no special multi-term processing)

highlightField

private java.util.Map<java.lang.Integer,java.lang.Object> highlightField(java.lang.String field,
                                                                         java.lang.String[] contents,
                                                                         java.text.BreakIterator bi,
                                                                         BytesRef[] terms,
                                                                         int[] docids,
                                                                         java.util.List<LeafReaderContext> leaves,
                                                                         int maxPassages,
                                                                         Query query)
                                                                  throws java.io.IOException

Throws:

java.io.IOException

highlightDoc

private Passage[] highlightDoc(java.lang.String field,
                               BytesRef[] terms,
                               int contentLength,
                               java.text.BreakIterator bi,
                               int doc,
                               TermsEnum termsEnum,
                               PostingsEnum[] postings,
                               int n)
                        throws java.io.IOException

Throws:

java.io.IOException

getEmptyHighlight

protected Passage[] getEmptyHighlight(java.lang.String fieldName,
                                      java.text.BreakIterator bi,
                                      int maxPassages)

Called to summarize a document when no hits were found. By default this just returns the first maxPassages sentences; subclasses can override to customize.