org.apache.lucene.classification

Class BM25NBClassifier

java.lang.Object

org.apache.lucene.classification.BM25NBClassifier

All Implemented Interfaces:

Classifier<BytesRef>

public class BM25NBClassifier
extends java.lang.Object
implements Classifier<BytesRef>

A classifier approximating naive bayes classifier by using pure queries on BM25.

Field Summary

Fields

Modifier and Type	Field and Description
private Analyzer	analyzer Analyzer to be used for tokenizing unseen input text
private java.lang.String	classFieldName name of the field to be used as a class / category output
private IndexReader	indexReader IndexReader used to access the Classifier's index
private IndexSearcher	indexSearcher IndexSearcher to run searches on the index for retrieving frequencies
private Query	query Query used to eventually filter the document set to be used to classify
private java.lang.String[]	textFieldNames names of the fields to be used as input text

Constructor Summary

Constructors

Constructor and Description
BM25NBClassifier(IndexReader indexReader, Analyzer analyzer, Query query, java.lang.String classFieldName, java.lang.String... textFieldNames) Creates a new NaiveBayes classifier.

Method Summary

Modifier and Type	Method and Description
ClassificationResult<BytesRef>	assignClass(java.lang.String inputDocument) Assign a class (with score) to the given text String
private java.util.List<ClassificationResult<BytesRef>>	assignClassNormalizedList(java.lang.String inputDocument) Calculate probabilities for all classes for a given input text
private double	calculateLogLikelihood(java.lang.String[] tokens, Term term)
private double	calculateLogPrior(Term term)
java.util.List<ClassificationResult<BytesRef>>	getClasses(java.lang.String text) Get all the classes (sorted by score, descending) assigned to the given text String.
java.util.List<ClassificationResult<BytesRef>>	getClasses(java.lang.String text, int max) Get the first max classes (sorted by score, descending) assigned to the given text String.
private double	getTermProbForClass(Term classTerm, java.lang.String... words)
private java.util.ArrayList<ClassificationResult<BytesRef>>	normClassificationResults(java.util.List<ClassificationResult<BytesRef>> assignedClasses) Normalize the classification results based on the max score available
private java.lang.String[]	tokenize(java.lang.String text) tokenize a String on this classifier's text fields and analyzer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

indexReader

private final IndexReader indexReader

IndexReader used to access the Classifier's index

textFieldNames

private final java.lang.String[] textFieldNames

names of the fields to be used as input text

classFieldName

private final java.lang.String classFieldName

name of the field to be used as a class / category output

analyzer

private final Analyzer analyzer

Analyzer to be used for tokenizing unseen input text

indexSearcher

private final IndexSearcher indexSearcher

IndexSearcher to run searches on the index for retrieving frequencies

query

private final Query query

Query used to eventually filter the document set to be used to classify

Constructor Detail

BM25NBClassifier

public BM25NBClassifier(IndexReader indexReader,
                        Analyzer analyzer,
                        Query query,
                        java.lang.String classFieldName,
                        java.lang.String... textFieldNames)

Creates a new NaiveBayes classifier.

Parameters:

indexReader - the reader on the index to be used for classification

analyzer - an Analyzer used to analyze unseen text

query - a Query to eventually filter the docs used for training the classifier, or null if all the indexed docs should be used

classFieldName - the name of the field used as the output for the classifier NOTE: must not be heavely analyzed as the returned class will be a token indexed for this field

textFieldNames - the name of the fields used as the inputs for the classifier, NO boosting supported per field

Method Detail

assignClass

public ClassificationResult<BytesRef> assignClass(java.lang.String inputDocument)
                                           throws java.io.IOException

Description copied from interface: Classifier

Assign a class (with score) to the given text String

Specified by:

assignClass in interface Classifier<BytesRef>

Parameters:

inputDocument - a String containing text to be classified

Returns:

a ClassificationResult holding assigned class of type T and score

Throws:

java.io.IOException - If there is a low-level I/O error.

getClasses

public java.util.List<ClassificationResult<BytesRef>> getClasses(java.lang.String text)
                                                          throws java.io.IOException

Description copied from interface: Classifier

Get all the classes (sorted by score, descending) assigned to the given text String.

Specified by:

getClasses in interface Classifier<BytesRef>

Parameters:

text - a String containing text to be classified

Returns:

the whole list of ClassificationResult, the classes and scores. Returns null if the classifier can't make lists.

Throws:

java.io.IOException - If there is a low-level I/O error.

getClasses

public java.util.List<ClassificationResult<BytesRef>> getClasses(java.lang.String text,
                                                                 int max)
                                                          throws java.io.IOException

Description copied from interface: Classifier

Get the first max classes (sorted by score, descending) assigned to the given text String.

Specified by:

getClasses in interface Classifier<BytesRef>

Parameters:

text - a String containing text to be classified

max - the number of return list elements

Returns:

the whole list of ClassificationResult, the classes and scores. Cut for "max" number of elements. Returns null if the classifier can't make lists.

Throws:

java.io.IOException - If there is a low-level I/O error.

assignClassNormalizedList

private java.util.List<ClassificationResult<BytesRef>> assignClassNormalizedList(java.lang.String inputDocument)
                                                                          throws java.io.IOException

Calculate probabilities for all classes for a given input text

Parameters:

inputDocument - the input text as a String

Returns:

a List of ClassificationResult, one for each existing class

Throws:

java.io.IOException - if assigning probabilities fails

normClassificationResults

private java.util.ArrayList<ClassificationResult<BytesRef>> normClassificationResults(java.util.List<ClassificationResult<BytesRef>> assignedClasses)

Normalize the classification results based on the max score available

Parameters:

assignedClasses - the list of assigned classes

Returns:

the normalized results

tokenize

private java.lang.String[] tokenize(java.lang.String text)
                             throws java.io.IOException

tokenize a String on this classifier's text fields and analyzer

Parameters:

text - the String representing an input text (to be classified)

Returns:

a String array of the resulting tokens

Throws:

java.io.IOException - if tokenization fails

calculateLogLikelihood

private double calculateLogLikelihood(java.lang.String[] tokens,
                                      Term term)
                               throws java.io.IOException

Throws:

java.io.IOException

getTermProbForClass

private double getTermProbForClass(Term classTerm,
                                   java.lang.String... words)
                            throws java.io.IOException

Throws:

java.io.IOException

calculateLogPrior

private double calculateLogPrior(Term term)
                          throws java.io.IOException

Throws:

java.io.IOException