org.apache.lucene.codecs.memory

Class FSTOrdTermsWriter

java.lang.Object

org.apache.lucene.codecs.FieldsConsumer

org.apache.lucene.codecs.memory.FSTOrdTermsWriter

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

public class FSTOrdTermsWriter
extends FieldsConsumer

FST-based term dict, using ord as FST output. The FST holds the mapping between <term, ord>, and term's metadata is delta encoded into a single byte block. Typically the byte block consists of four parts: 1. term statistics: docFreq, totalTermFreq; 2. monotonic long[], e.g. the pointer to the postings list for that term; 3. generic byte[], e.g. other information customized by postings base. 4. single-level skip list to speed up metadata decoding by ord.

Files:

• .tix: Term Index
• .tbk: Term Block

Term Index

The .tix contains a list of FSTs, one for each field. The FST maps a term to its corresponding order in current field.

• TermIndex(.tix) --> Header, TermFST^NumFields, Footer
• TermFST --> FST<long>
• Header --> IndexHeader
• Footer --> CodecFooter

Notes:

• Since terms are already sorted before writing to Term Block, their ords can directly used to seek term metadata from term block.

Term Block

The .tbk contains all the statistics and metadata for terms, along with field summary (e.g. per-field data like number of documents in current field). For each field, there are four blocks:

• statistics bytes block: contains term statistics;
• metadata longs block: delta-encodes monotonic part of metadata;
• metadata bytes block: encodes other parts of metadata;
• skip block: contains skip data, to speed up metadata seeking and decoding

File Format:

• TermBlock(.tbk) --> Header, PostingsHeader, FieldSummary, DirOffset
• FieldSummary --> NumFields, <FieldNumber, NumTerms, SumTotalTermFreq?, SumDocFreq, DocCount, LongsSize, DataBlock > ^NumFields, Footer
• DataBlock --> StatsBlockLength, MetaLongsBlockLength, MetaBytesBlockLength, SkipBlock, StatsBlock, MetaLongsBlock, MetaBytesBlock
• SkipBlock --> < StatsFPDelta, MetaLongsSkipFPDelta, MetaBytesSkipFPDelta, MetaLongsSkipDelta^LongsSize >^NumTerms
• StatsBlock --> < DocFreq[Same?], (TotalTermFreq-DocFreq) ? > ^NumTerms
• MetaLongsBlock --> < LongDelta^LongsSize, BytesSize > ^NumTerms
• MetaBytesBlock --> Byte ^{MetaBytesBlockLength}
• Header --> IndexHeader
• DirOffset --> Uint64
• NumFields, FieldNumber, DocCount, DocFreq, LongsSize, FieldNumber, DocCount --> VInt
• NumTerms, SumTotalTermFreq, SumDocFreq, StatsBlockLength, MetaLongsBlockLength, MetaBytesBlockLength, StatsFPDelta, MetaLongsSkipFPDelta, MetaBytesSkipFPDelta, MetaLongsSkipStart, TotalTermFreq, LongDelta,--> VLong
• Footer --> CodecFooter

Notes:

• The format of PostingsHeader and MetaBytes are customized by the specific postings implementation: they contain arbitrary per-file data (such as parameters or versioning information), and per-term data (non-monotonic ones like pulsed postings data).
• During initialization the reader will load all the blocks into memory. SkipBlock will be decoded, so that during seek term dict can lookup file pointers directly. StatsFPDelta, MetaLongsSkipFPDelta, etc. are file offset for every SkipInterval's term. MetaLongsSkipDelta is the difference from previous one, which indicates the value of preceding metadata longs for every SkipInterval's term.
• DocFreq is the count of documents which contain the term. TotalTermFreq is the total number of occurrences of the term. Usually these two values are the same for long tail terms, therefore one bit is stole from DocFreq to check this case, so that encoding of TotalTermFreq may be omitted.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	FSTOrdTermsWriter.FieldMetaData
(package private) class	FSTOrdTermsWriter.TermsWriter

Field Summary

Fields

Modifier and Type	Field and Description
(package private) IndexOutput	blockOut
(package private) FieldInfos	fieldInfos
(package private) java.util.List<FSTOrdTermsWriter.FieldMetaData>	fields
(package private) IndexOutput	indexOut
(package private) int	maxDoc
(package private) PostingsWriterBase	postingsWriter
static int	SKIP_INTERVAL
(package private) static java.lang.String	TERMS_BLOCK_EXTENSION
(package private) static java.lang.String	TERMS_CODEC_NAME
(package private) static java.lang.String	TERMS_INDEX_CODEC_NAME
(package private) static java.lang.String	TERMS_INDEX_EXTENSION
static int	VERSION_CURRENT
static int	VERSION_START

Constructor Summary

Constructors

Constructor and Description
FSTOrdTermsWriter(SegmentWriteState state, PostingsWriterBase postingsWriter)

Method Summary

Modifier and Type	Method and Description
void	close()
void	write(Fields fields, NormsProducer norms) Write all fields, terms and postings.
private void	writeTrailer(IndexOutput out, long dirStart)

Methods inherited from class org.apache.lucene.codecs.FieldsConsumer

merge

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TERMS_INDEX_EXTENSION

static final java.lang.String TERMS_INDEX_EXTENSION

See Also:

Constant Field Values

TERMS_BLOCK_EXTENSION

static final java.lang.String TERMS_BLOCK_EXTENSION

See Also:

Constant Field Values

TERMS_CODEC_NAME

static final java.lang.String TERMS_CODEC_NAME

See Also:

Constant Field Values

TERMS_INDEX_CODEC_NAME

static final java.lang.String TERMS_INDEX_CODEC_NAME

See Also:

Constant Field Values

VERSION_START

public static final int VERSION_START

See Also:

Constant Field Values

VERSION_CURRENT

public static final int VERSION_CURRENT

See Also:

Constant Field Values

SKIP_INTERVAL

public static final int SKIP_INTERVAL

See Also:

Constant Field Values

postingsWriter

final PostingsWriterBase postingsWriter

fieldInfos

final FieldInfos fieldInfos

maxDoc

final int maxDoc

fields

final java.util.List<FSTOrdTermsWriter.FieldMetaData> fields

blockOut

IndexOutput blockOut

indexOut

IndexOutput indexOut

Constructor Detail

FSTOrdTermsWriter

public FSTOrdTermsWriter(SegmentWriteState state,
                         PostingsWriterBase postingsWriter)
                  throws java.io.IOException

Throws:

java.io.IOException

Method Detail

write

public void write(Fields fields,
                  NormsProducer norms)
           throws java.io.IOException

Description copied from class: FieldsConsumer

Write all fields, terms and postings. This the "pull" API, allowing you to iterate more than once over the postings, somewhat analogous to using a DOM API to traverse an XML tree.

Notes:

• You must compute index statistics, including each Term's docFreq and totalTermFreq, as well as the summary sumTotalTermFreq, sumTotalDocFreq and docCount.
• You must skip terms that have no docs and fields that have no terms, even though the provided Fields API will expose them; this typically requires lazily writing the field or term until you've actually seen the first term or document.
• The provided Fields instance is limited: you cannot call any methods that return statistics/counts; you cannot pass a non-null live docs when pulling docs/positions enums.

Specified by:

write in class FieldsConsumer

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in class FieldsConsumer

Throws:

java.io.IOException

writeTrailer

private void writeTrailer(IndexOutput out,
                          long dirStart)
                   throws java.io.IOException

Throws:

java.io.IOException