org.apache.lucene.codecs.bloom

Class BloomFilteringPostingsFormat

java.lang.Object

org.apache.lucene.codecs.PostingsFormat

org.apache.lucene.codecs.bloom.BloomFilteringPostingsFormat

All Implemented Interfaces:

NamedSPILoader.NamedSPI

public final class BloomFilteringPostingsFormat
extends PostingsFormat

A PostingsFormat useful for low doc-frequency fields such as primary keys. Bloom filters are maintained in a ".blm" file which offers "fast-fail" for reads in segments known to have no record of the key. A choice of delegate PostingsFormat is used to record all other Postings data.

A choice of BloomFilterFactory can be passed to tailor Bloom Filter settings on a per-field basis. The default configuration is DefaultBloomFilterFactory which allocates a ~8mb bitset and hashes values using MurmurHash2. This should be suitable for most purposes.

The format of the blm file is as follows:

• BloomFilter (.blm) --> Header, DelegatePostingsFormatName, NumFilteredFields, Filter^{NumFilteredFields}, Footer
• Filter --> FieldNumber, FuzzySet
• FuzzySet -->See FuzzySet.serialize(DataOutput)
• Header --> IndexHeader
• DelegatePostingsFormatName --> String The name of a ServiceProvider registered PostingsFormat
• NumFilteredFields --> Uint32
• FieldNumber --> Uint32 The number of the field in this segment
• Footer --> CodecFooter

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) class	BloomFilteringPostingsFormat.BloomFilteredFieldsConsumer
(package private) static class	BloomFilteringPostingsFormat.BloomFilteredFieldsProducer

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	BLOOM_CODEC_NAME
(package private) static java.lang.String	BLOOM_EXTENSION Extension of Bloom Filters file
(package private) BloomFilterFactory	bloomFilterFactory
private PostingsFormat	delegatePostingsFormat
static int	VERSION_CURRENT
static int	VERSION_START

Fields inherited from class org.apache.lucene.codecs.PostingsFormat

EMPTY

Constructor Summary

Constructors

Constructor and Description
BloomFilteringPostingsFormat()
BloomFilteringPostingsFormat(PostingsFormat delegatePostingsFormat) Creates Bloom filters for a selection of fields created in the index.
BloomFilteringPostingsFormat(PostingsFormat delegatePostingsFormat, BloomFilterFactory bloomFilterFactory) Creates Bloom filters for a selection of fields created in the index.

Method Summary

Modifier and Type	Method and Description
FieldsConsumer	fieldsConsumer(SegmentWriteState state) Writes a new segment
FieldsProducer	fieldsProducer(SegmentReadState state) Reads a segment.
java.lang.String	toString()

Methods inherited from class org.apache.lucene.codecs.PostingsFormat

availablePostingsFormats, forName, getName, reloadPostingsFormats

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

BLOOM_CODEC_NAME

public static final java.lang.String BLOOM_CODEC_NAME

See Also:

Constant Field Values

VERSION_START

public static final int VERSION_START

See Also:

Constant Field Values

VERSION_CURRENT

public static final int VERSION_CURRENT

See Also:

Constant Field Values

BLOOM_EXTENSION

static final java.lang.String BLOOM_EXTENSION

Extension of Bloom Filters file

See Also:

Constant Field Values

bloomFilterFactory

BloomFilterFactory bloomFilterFactory

delegatePostingsFormat

private PostingsFormat delegatePostingsFormat

Constructor Detail

BloomFilteringPostingsFormat

public BloomFilteringPostingsFormat(PostingsFormat delegatePostingsFormat,
                                    BloomFilterFactory bloomFilterFactory)

Creates Bloom filters for a selection of fields created in the index. This is recorded as a set of Bitsets held as a segment summary in an additional "blm" file. This PostingsFormat delegates to a choice of delegate PostingsFormat for encoding all other postings data.

Parameters:

delegatePostingsFormat - The PostingsFormat that records all the non-bloom filter data i.e. postings info.

bloomFilterFactory - The BloomFilterFactory responsible for sizing BloomFilters appropriately

BloomFilteringPostingsFormat

public BloomFilteringPostingsFormat(PostingsFormat delegatePostingsFormat)

Creates Bloom filters for a selection of fields created in the index. This is recorded as a set of Bitsets held as a segment summary in an additional "blm" file. This PostingsFormat delegates to a choice of delegate PostingsFormat for encoding all other postings data. This choice of constructor defaults to the DefaultBloomFilterFactory for configuring per-field BloomFilters.

Parameters:

delegatePostingsFormat - The PostingsFormat that records all the non-bloom filter data i.e. postings info.

BloomFilteringPostingsFormat

public BloomFilteringPostingsFormat()

Method Detail

fieldsConsumer

public FieldsConsumer fieldsConsumer(SegmentWriteState state)
                              throws java.io.IOException

Description copied from class: PostingsFormat

Writes a new segment

Specified by:

fieldsConsumer in class PostingsFormat

Throws:

java.io.IOException

fieldsProducer

public FieldsProducer fieldsProducer(SegmentReadState state)
                              throws java.io.IOException

Description copied from class: PostingsFormat

Reads a segment. NOTE: by the time this call returns, it must hold open any files it will need to use; else, those files may be deleted. Additionally, required files may be deleted during the execution of this call before there is a chance to open them. Under these circumstances an IOException should be thrown by the implementation. IOExceptions are expected and will automatically cause a retry of the segment opening logic with the newly revised segments.

Specified by:

fieldsProducer in class PostingsFormat

Throws:

java.io.IOException

toString

public java.lang.String toString()

Overrides:

toString in class PostingsFormat