org.apache.tika.batch

Class FileResourceConsumer

java.lang.Object

org.apache.tika.batch.FileResourceConsumer

All Implemented Interfaces:

java.util.concurrent.Callable<IFileProcessorFutureResult>

Direct Known Subclasses:

AbstractFSConsumer

public abstract class FileResourceConsumer
extends java.lang.Object
implements java.util.concurrent.Callable<IFileProcessorFutureResult>

This is a base class for file consumers. The goal of this class is to abstract out the multithreading and recordkeeping components.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	FileResourceConsumer.STATE

Field Summary

Fields

Modifier and Type	Field and Description
private int	consumerId
private FileStarted	currentFile
private FileResourceConsumer.STATE	currentState
static java.lang.String	ELAPSED_MILLIS
private java.util.concurrent.ArrayBlockingQueue<FileResource>	fileQueue
static java.lang.String	IO_IS
static java.lang.String	IO_OS
private java.lang.Object	lock
protected static org.slf4j.Logger	LOG
private long	maxConsecWaitInMillis
private static java.util.concurrent.atomic.AtomicInteger	numConsumers
private int	numHandledExceptions
private int	numResourcesConsumed
static java.lang.String	OOM
static java.lang.String	PARSE_ERR
static java.lang.String	PARSE_EX
static java.lang.String	TIMED_OUT

Constructor Summary

Constructors

Constructor and Description
FileResourceConsumer(java.util.concurrent.ArrayBlockingQueue<FileResource> fileQueue)

Method Summary

Modifier and Type	Method and Description
private boolean	_processFileResource(FileResource fileResource)
IFileProcessorFutureResult	call()
FileStarted	checkForTimedOutMillis(long staleThresholdMillis) Checks to see if the currentFile being processed (if there is one) should be timed out (still being worked on after staleThresholdMillis).
protected void	close(java.io.Closeable closeable)
protected void	flushAndClose(java.io.Closeable closeable)
FileStarted	getCurrentFile() Returns the name and start time of a file that is currently being processed.
private FileResource	getNextFileResource()
int	getNumHandledExceptions()
int	getNumResourcesConsumed()
protected java.lang.String	getXMLifiedLogMsg(java.lang.String type, java.lang.String resourceId, java.lang.String... attrs)
protected java.lang.String	getXMLifiedLogMsg(java.lang.String type, java.lang.String resourceId, java.lang.Throwable t, java.lang.String... attrs) Use this for structured output that captures resourceId and other attributes.
protected void	incrementHandledExceptions() Make sure to call this appropriately!
boolean	isStillActive() Returns whether or not the consumer is still could process a file or is still processing a file (ACTIVELY_CONSUMING or ASKED_TO_SHUTDOWN)
protected void	parse(java.lang.String resourceId, Parser parser, java.io.InputStream is, org.xml.sax.ContentHandler handler, Metadata m, ParseContext parseContext) Utility method to handle logging equivalently among all implementing classes.
void	pleaseShutdown() This politely asks the consumer to shutdown.
abstract boolean	processFileResource(FileResource fileResource) Main piece of code that needs to be implemented.
private void	setEndedState(FileResourceConsumer.STATE cause)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.slf4j.Logger LOG

TIMED_OUT

public static java.lang.String TIMED_OUT

OOM

public static java.lang.String OOM

IO_IS

public static java.lang.String IO_IS

IO_OS

public static java.lang.String IO_OS

PARSE_ERR

public static java.lang.String PARSE_ERR

PARSE_EX

public static java.lang.String PARSE_EX

ELAPSED_MILLIS

public static java.lang.String ELAPSED_MILLIS

numConsumers

private static java.util.concurrent.atomic.AtomicInteger numConsumers

maxConsecWaitInMillis

private long maxConsecWaitInMillis

fileQueue

private final java.util.concurrent.ArrayBlockingQueue<FileResource> fileQueue

consumerId

private final int consumerId

lock

private final java.lang.Object lock

currentFile

private FileStarted currentFile

numResourcesConsumed

private volatile int numResourcesConsumed

numHandledExceptions

private volatile int numHandledExceptions

currentState

private volatile FileResourceConsumer.STATE currentState

Constructor Detail

FileResourceConsumer

public FileResourceConsumer(java.util.concurrent.ArrayBlockingQueue<FileResource> fileQueue)

Method Detail

call

public IFileProcessorFutureResult call()

Specified by:

call in interface java.util.concurrent.Callable<IFileProcessorFutureResult>

processFileResource

public abstract boolean processFileResource(FileResource fileResource)

Main piece of code that needs to be implemented. Clients are responsible for closing streams and handling the exceptions that they'd like to handle.

Unchecked throwables can be thrown past this, of course. When an unchecked throwable is thrown, this logs the error, and then rethrows the exception. Clients/subclasses should make sure to catch and handle everything they can.

The design goal is that the whole process should close up and shutdown soon after an unchecked exception or error is thrown.

Make sure to call incrementHandledExceptions() appropriately in your implementation of this method.

Parameters:

fileResource - resource to process

Returns:

whether or not a file was successfully processed

incrementHandledExceptions

protected void incrementHandledExceptions()

Make sure to call this appropriately!

isStillActive

public boolean isStillActive()

Returns whether or not the consumer is still could process a file or is still processing a file (ACTIVELY_CONSUMING or ASKED_TO_SHUTDOWN)

Returns:

whether this consumer is still active

_processFileResource

private boolean _processFileResource(FileResource fileResource)

pleaseShutdown

public void pleaseShutdown()

This politely asks the consumer to shutdown. Before processing another file, the consumer will check to see if it has been asked to terminate.

This offers another method for politely requesting that a FileResourceConsumer stop processing besides passing it PoisonFileResource.

getCurrentFile

public FileStarted getCurrentFile()

Returns the name and start time of a file that is currently being processed. If no file is currently being processed, this will return null.

Returns:

FileStarted or null

getNumResourcesConsumed

public int getNumResourcesConsumed()

getNumHandledExceptions

public int getNumHandledExceptions()

checkForTimedOutMillis

public FileStarted checkForTimedOutMillis(long staleThresholdMillis)

Checks to see if the currentFile being processed (if there is one) should be timed out (still being worked on after staleThresholdMillis).

If the consumer should be timed out, this will return the currentFile and set the state to TIMED_OUT.

If the consumer was already timed out earlier or is not processing a file or has been working on a file for less than #staleThresholdMillis, then this will return null.

Parameters:

staleThresholdMillis - threshold to determine whether the consumer has gone stale.

Returns:

null or the file started that triggered the stale condition

getXMLifiedLogMsg

protected java.lang.String getXMLifiedLogMsg(java.lang.String type,
                                             java.lang.String resourceId,
                                             java.lang.String... attrs)

getXMLifiedLogMsg

protected java.lang.String getXMLifiedLogMsg(java.lang.String type,
                                             java.lang.String resourceId,
                                             java.lang.Throwable t,
                                             java.lang.String... attrs)

Use this for structured output that captures resourceId and other attributes.

Parameters:

type - entity name for exception

resourceId - resourceId string

t - throwable can be null

attrs - (array of key0, value0, key1, value1, etc.)

getNextFileResource

private FileResource getNextFileResource()
                                  throws java.lang.InterruptedException

Throws:

java.lang.InterruptedException

close

protected void close(java.io.Closeable closeable)

flushAndClose

protected void flushAndClose(java.io.Closeable closeable)

setEndedState

private void setEndedState(FileResourceConsumer.STATE cause)

parse

protected void parse(java.lang.String resourceId,
                     Parser parser,
                     java.io.InputStream is,
                     org.xml.sax.ContentHandler handler,
                     Metadata m,
                     ParseContext parseContext)
              throws java.lang.Throwable

Utility method to handle logging equivalently among all implementing classes. Use, override or avoid as desired.

Parameters:

resourceId - resourceId

parser - parser to use

is - inputStream (will be closed by this method!)

handler - handler for the content

m - metadata

parseContext - parse context

Throws:

java.lang.Throwable - (logs and then throws whatever was thrown (if anything)