org.apache.lucene.index

Class DocumentsWriter

java.lang.Object

org.apache.lucene.index.DocumentsWriter

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, Accountable

final class DocumentsWriter
extends java.lang.Object
implements java.io.Closeable, Accountable

This class accepts multiple added documents and directly writes segment files. Each added document is passed to the indexing chain, which in turn processes the document into the different codec formats. Some formats write bytes to files immediately, e.g. stored fields and term vectors, while others are buffered by the indexing chain and written only on flush. Once we have used our allowed RAM buffer, or the number of added docs is large enough (in the case we are flushing by doc count instead of RAM usage), we create a real segment and flush it to the Directory. Threads: Multiple threads are allowed into addDocument at once. There is an initial synchronized call to getThreadState which allocates a ThreadState for this thread. The same thread will get the same ThreadState over time (thread affinity) so that if there are consistent patterns (for example each thread is indexing a different content source) then we make better use of RAM. Then processDocument is called on that ThreadState without synchronization (most of the "heavy lifting" is in this call). Finally the synchronized "finishDocument" is called to flush changes to the directory. When flush is called by IndexWriter we forcefully idle all threads and flush only once they are all idle. This means you can call flush with a given thread even while other threads are actively adding/deleting documents. Exceptions: Because this class directly updates in-memory posting lists, and flushes stored fields and term vectors directly to files in the directory, there are certain limited times when an exception can corrupt this state. For example, a disk full while flushing stored fields leaves this file in a corrupt state. Or, an OOM exception while appending to the in-memory posting lists can corrupt that posting list. We call such exceptions "aborting exceptions". In these cases we must call abort() to discard all docs added since the last flush. All other exceptions ("non-aborting exceptions") can still partially update the index structures. These updates are consistent, but, they represent only a part of the document seen up until the exception was hit. When this happens, we immediately mark the document as deleted so that the document is always atomically ("all or none") added to the index.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) static class	DocumentsWriter.ApplyDeletesEvent
(package private) static class	DocumentsWriter.DeleteNewFilesEvent
(package private) static class	DocumentsWriter.FlushFailedEvent
(package private) static class	DocumentsWriter.ForcedPurgeEvent
(package private) static class	DocumentsWriter.MergePendingEvent

Field Summary

Fields

Modifier and Type	Field and Description
private boolean	closed
private LiveIndexWriterConfig	config
private DocumentsWriterDeleteQueue	currentFullFlushDelQueue
(package private) DocumentsWriterDeleteQueue	deleteQueue
private Directory	directory
private Directory	directoryOrig
private java.util.Queue<IndexWriter.Event>	events
(package private) DocumentsWriterFlushControl	flushControl
(package private) FlushPolicy	flushPolicy
private InfoStream	infoStream
private java.util.concurrent.atomic.AtomicInteger	numDocsInRAM
private boolean	pendingChangesInCurrentFullFlush
(package private) DocumentsWriterPerThreadPool	perThreadPool
private DocumentsWriterFlushQueue	ticketQueue
private IndexWriter	writer

Constructor Summary

Constructors

Constructor and Description
DocumentsWriter(IndexWriter writer, LiveIndexWriterConfig config, Directory directoryOrig, Directory directory)

Method Summary

Modifier and Type	Method and Description
(package private) void	abort(IndexWriter writer) Called if we hit an exception at a bad time (when updating the index files) and must discard all currently buffered docs.
private int	abortThreadState(DocumentsWriterPerThreadPool.ThreadState perThread) Returns how many documents were aborted.
(package private) boolean	anyChanges()
boolean	anyDeletions()
private boolean	applyAllDeletes(DocumentsWriterDeleteQueue deleteQueue)
void	close()
(package private) DocumentsWriterDeleteQueue	currentDeleteSession()
(package private) boolean	deleteQueries(Query... queries)
(package private) boolean	deleteTerms(Term... terms)
private boolean	doFlush(DocumentsWriterPerThread flushingDWPT)
private void	ensureInitialized(DocumentsWriterPerThreadPool.ThreadState state)
private void	ensureOpen()
java.util.Queue<IndexWriter.Event>	eventQueue()
(package private) void	finishFullFlush(IndexWriter indexWriter, boolean success)
(package private) boolean	flushAllThreads()
int	getBufferedDeleteTermsSize()
LiveIndexWriterConfig	getIndexWriterConfig()
int	getNumBufferedDeleteTerms()
(package private) int	getNumDocs() Returns how many docs are currently buffered in RAM.
(package private) long	lockAndAbortAll(IndexWriter indexWriter) Returns how many documents were aborted.
private boolean	postUpdate(DocumentsWriterPerThread flushingDWPT, boolean hasEvents)
private boolean	preUpdate()
(package private) int	purgeBuffer(IndexWriter writer, boolean forced)
private void	putEvent(IndexWriter.Event event)
long	ramBytesUsed() Return the memory usage of this object in bytes.
private boolean	setFlushingDeleteQueue(DocumentsWriterDeleteQueue session)
(package private) void	subtractFlushedNumDocs(int numFlushed)
(package private) void	unlockAllAfterAbortAll(IndexWriter indexWriter)
(package private) boolean	updateDocument(java.lang.Iterable<? extends IndexableField> doc, Analyzer analyzer, Term delTerm)
(package private) boolean	updateDocuments(java.lang.Iterable<? extends java.lang.Iterable<? extends IndexableField>> docs, Analyzer analyzer, Term delTerm)
(package private) boolean	updateDocValues(DocValuesUpdate... updates)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.lucene.util.Accountable

getChildResources

Field Detail

directoryOrig

private final Directory directoryOrig

directory

private final Directory directory

closed

private volatile boolean closed

infoStream

private final InfoStream infoStream

config

private final LiveIndexWriterConfig config

numDocsInRAM

private final java.util.concurrent.atomic.AtomicInteger numDocsInRAM

deleteQueue

volatile DocumentsWriterDeleteQueue deleteQueue

ticketQueue

private final DocumentsWriterFlushQueue ticketQueue

pendingChangesInCurrentFullFlush

private volatile boolean pendingChangesInCurrentFullFlush

perThreadPool

final DocumentsWriterPerThreadPool perThreadPool

flushPolicy

final FlushPolicy flushPolicy

flushControl

final DocumentsWriterFlushControl flushControl

writer

private final IndexWriter writer

events

private final java.util.Queue<IndexWriter.Event> events

currentFullFlushDelQueue

private volatile DocumentsWriterDeleteQueue currentFullFlushDelQueue

Constructor Detail

DocumentsWriter

DocumentsWriter(IndexWriter writer,
                LiveIndexWriterConfig config,
                Directory directoryOrig,
                Directory directory)

Method Detail

deleteQueries

boolean deleteQueries(Query... queries)
               throws java.io.IOException

Throws:

java.io.IOException

deleteTerms

boolean deleteTerms(Term... terms)
             throws java.io.IOException

Throws:

java.io.IOException

updateDocValues

boolean updateDocValues(DocValuesUpdate... updates)
                 throws java.io.IOException

Throws:

java.io.IOException

currentDeleteSession

DocumentsWriterDeleteQueue currentDeleteSession()

applyAllDeletes

private boolean applyAllDeletes(DocumentsWriterDeleteQueue deleteQueue)
                         throws java.io.IOException

Throws:

java.io.IOException

purgeBuffer

int purgeBuffer(IndexWriter writer,
                boolean forced)
         throws java.io.IOException

Throws:

java.io.IOException

getNumDocs

int getNumDocs()

Returns how many docs are currently buffered in RAM.

ensureOpen

private void ensureOpen()
                 throws AlreadyClosedException

Throws:

AlreadyClosedException

abort

void abort(IndexWriter writer)

Called if we hit an exception at a bad time (when updating the index files) and must discard all currently buffered docs. This resets our state, discarding any docs added since last flush.

lockAndAbortAll

long lockAndAbortAll(IndexWriter indexWriter)

Returns how many documents were aborted.

abortThreadState

private int abortThreadState(DocumentsWriterPerThreadPool.ThreadState perThread)

Returns how many documents were aborted.

unlockAllAfterAbortAll

void unlockAllAfterAbortAll(IndexWriter indexWriter)

anyChanges

boolean anyChanges()

getBufferedDeleteTermsSize

public int getBufferedDeleteTermsSize()

getNumBufferedDeleteTerms

public int getNumBufferedDeleteTerms()

anyDeletions

public boolean anyDeletions()

close

public void close()

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

preUpdate

private boolean preUpdate()
                   throws java.io.IOException,
                          AbortingException

Throws:

java.io.IOException

AbortingException

postUpdate

private boolean postUpdate(DocumentsWriterPerThread flushingDWPT,
                           boolean hasEvents)
                    throws java.io.IOException,
                           AbortingException

Throws:

java.io.IOException

AbortingException

ensureInitialized

private void ensureInitialized(DocumentsWriterPerThreadPool.ThreadState state)
                        throws java.io.IOException

Throws:

java.io.IOException

updateDocuments

boolean updateDocuments(java.lang.Iterable<? extends java.lang.Iterable<? extends IndexableField>> docs,
                        Analyzer analyzer,
                        Term delTerm)
                 throws java.io.IOException,
                        AbortingException

Throws:

java.io.IOException

AbortingException

updateDocument

boolean updateDocument(java.lang.Iterable<? extends IndexableField> doc,
                       Analyzer analyzer,
                       Term delTerm)
                throws java.io.IOException,
                       AbortingException

Throws:

java.io.IOException

AbortingException

doFlush

private boolean doFlush(DocumentsWriterPerThread flushingDWPT)
                 throws java.io.IOException,
                        AbortingException

Throws:

java.io.IOException

AbortingException

subtractFlushedNumDocs

void subtractFlushedNumDocs(int numFlushed)

setFlushingDeleteQueue

private boolean setFlushingDeleteQueue(DocumentsWriterDeleteQueue session)

flushAllThreads

boolean flushAllThreads()
                 throws java.io.IOException,
                        AbortingException

Throws:

java.io.IOException

AbortingException

finishFullFlush

void finishFullFlush(IndexWriter indexWriter,
                     boolean success)

getIndexWriterConfig

public LiveIndexWriterConfig getIndexWriterConfig()

putEvent

private void putEvent(IndexWriter.Event event)

ramBytesUsed

public long ramBytesUsed()

Description copied from interface: Accountable

Return the memory usage of this object in bytes. Negative values are illegal.

Specified by:

ramBytesUsed in interface Accountable

eventQueue

public java.util.Queue<IndexWriter.Event> eventQueue()