org.apache.tika.batch

Class BatchProcess

java.lang.Object

org.apache.tika.batch.BatchProcess

All Implemented Interfaces:

java.util.concurrent.Callable<ParallelFileProcessingResult>

public class BatchProcess
extends java.lang.Object
implements java.util.concurrent.Callable<ParallelFileProcessingResult>

This is the main processor class for a single process. This class can only be run once.

It requires a FileResourceCrawler and FileResourceConsumers, and it can also support a StatusReporter and an Interrupter.

This is designed to shutdown if a parser has timed out or if there is an OutOfMemoryError. Consider using BatchProcessDriverCLI as a daemon/watchdog that monitors and can restart this batch process;

Note that this classs redirects stderr to stdout so that it can communicate without interference with the parent process on stderr.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BatchProcess.BATCH_CONSTANTS
private static class	BatchProcess.CAUSE_FOR_TERMINATION
private class	BatchProcess.State
private class	BatchProcess.TimeoutChecker
private class	BatchProcess.TimeoutFutureResult

Field Summary

Fields

Modifier and Type	Field and Description
private boolean	alreadyExecuted
private ConsumersManager	consumersManager
private long	consumersManagerMaxMillis
private FileResourceCrawler	fileResourceCrawler
private Interrupter	interrupter
private static org.slf4j.Logger	LOG
private int	maxAliveTimeSeconds
private java.io.PrintStream	outputStreamWriter
private long	pauseOnEarlyTerminationMillis
private StatusReporter	reporter
private java.util.concurrent.ArrayBlockingQueue<FileStarted>	timedOuts
private long	timeoutCheckPulseMillis
private long	timeoutThresholdMillis

Constructor Summary

Constructors

Constructor and Description
BatchProcess(FileResourceCrawler fileResourceCrawler, ConsumersManager consumersManager, StatusReporter reporter, Interrupter interrupter)

Method Summary

Modifier and Type	Method and Description
private boolean	aliveTooLong(long started)
private boolean	areResourcesPotentiallyRemaining()
ParallelFileProcessingResult	call() Runs main execution loop.
private int	countActiveConsumers()
private int	getExitStatus(BatchProcess.CAUSE_FOR_TERMINATION causeForTermination, java.lang.String restartMsg)
private boolean	isNonRestart(java.lang.Throwable e)
private BatchProcess.State	mainLoop(java.util.concurrent.CompletionService<IFileProcessorFutureResult> completionService, BatchProcess.TimeoutChecker timeoutChecker)
private void	politelyAwaitTermination(BatchProcess.CAUSE_FOR_TERMINATION causeForTermination) This is used instead of awaitTermination(), because that interrupts the thread and then waits for its termination.
void	setMaxAliveTimeSeconds(int maxAliveTimeSeconds) The maximum amount of time that this process can be alive.
void	setPauseOnEarlyTerminationMillis(long pauseOnEarlyTerminationMillis) If there is an early termination via an interrupt or too many timed out consumers or because a consumer or other Runnable threw a Throwable, pause this long before killing the consumers and other threads.
void	setTimeoutCheckPulseMillis(long timeoutCheckPulseMillis)
void	setTimeoutThresholdMillis(long timeoutThresholdMillis) The amount of time allowed before a consumer should be timed out.
private ParallelFileProcessingResult	shutdown(java.util.concurrent.ExecutorService ex, java.util.concurrent.CompletionService<IFileProcessorFutureResult> completionService, BatchProcess.TimeoutChecker timeoutChecker, BatchProcess.State state)
private void	shutdownConsumersManager()
private void	startConsumersManager()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

outputStreamWriter

private java.io.PrintStream outputStreamWriter

timeoutThresholdMillis

private long timeoutThresholdMillis

timeoutCheckPulseMillis

private long timeoutCheckPulseMillis

pauseOnEarlyTerminationMillis

private long pauseOnEarlyTerminationMillis

consumersManagerMaxMillis

private final long consumersManagerMaxMillis

maxAliveTimeSeconds

private int maxAliveTimeSeconds

fileResourceCrawler

private final FileResourceCrawler fileResourceCrawler

consumersManager

private final ConsumersManager consumersManager

reporter

private final StatusReporter reporter

interrupter

private final Interrupter interrupter

timedOuts

private final java.util.concurrent.ArrayBlockingQueue<FileStarted> timedOuts

alreadyExecuted

private boolean alreadyExecuted

Constructor Detail

BatchProcess

public BatchProcess(FileResourceCrawler fileResourceCrawler,
                    ConsumersManager consumersManager,
                    StatusReporter reporter,
                    Interrupter interrupter)

Method Detail

call

public ParallelFileProcessingResult call()
                                  throws java.lang.InterruptedException

Runs main execution loop.

Redirects stdout to stderr to keep clean communications over stdout with parent process

Specified by:

call in interface java.util.concurrent.Callable<ParallelFileProcessingResult>

Returns:

result of the processing

Throws:

java.lang.InterruptedException

mainLoop

private BatchProcess.State mainLoop(java.util.concurrent.CompletionService<IFileProcessorFutureResult> completionService,
                                    BatchProcess.TimeoutChecker timeoutChecker)

shutdown

private ParallelFileProcessingResult shutdown(java.util.concurrent.ExecutorService ex,
                                              java.util.concurrent.CompletionService<IFileProcessorFutureResult> completionService,
                                              BatchProcess.TimeoutChecker timeoutChecker,
                                              BatchProcess.State state)

startConsumersManager

private void startConsumersManager()

shutdownConsumersManager

private void shutdownConsumersManager()

politelyAwaitTermination

private void politelyAwaitTermination(BatchProcess.CAUSE_FOR_TERMINATION causeForTermination)

This is used instead of awaitTermination(), because that interrupts the thread and then waits for its termination. This politely waits.

Parameters:

causeForTermination - reason for termination.

isNonRestart

private boolean isNonRestart(java.lang.Throwable e)

getExitStatus

private int getExitStatus(BatchProcess.CAUSE_FOR_TERMINATION causeForTermination,
                          java.lang.String restartMsg)

areResourcesPotentiallyRemaining

private boolean areResourcesPotentiallyRemaining()

aliveTooLong

private boolean aliveTooLong(long started)

countActiveConsumers

private int countActiveConsumers()

setPauseOnEarlyTerminationMillis

public void setPauseOnEarlyTerminationMillis(long pauseOnEarlyTerminationMillis)

If there is an early termination via an interrupt or too many timed out consumers or because a consumer or other Runnable threw a Throwable, pause this long before killing the consumers and other threads. Typically makes sense for this to be the same or slightly larger than timeoutThresholdMillis

Parameters:

pauseOnEarlyTerminationMillis - how long to pause if there is an early termination

setTimeoutThresholdMillis

public void setTimeoutThresholdMillis(long timeoutThresholdMillis)

The amount of time allowed before a consumer should be timed out.

Parameters:

timeoutThresholdMillis - threshold in milliseconds before declaring a consumer timed out

setTimeoutCheckPulseMillis

public void setTimeoutCheckPulseMillis(long timeoutCheckPulseMillis)

setMaxAliveTimeSeconds

public void setMaxAliveTimeSeconds(int maxAliveTimeSeconds)

The maximum amount of time that this process can be alive. To avoid memory leaks, it is sometimes beneficial to shutdown (and restart) the process periodically.

If the value is < 0, the process will run until completion, interruption or exception.

Parameters:

maxAliveTimeSeconds - maximum amount of time in seconds to remain alive