FileResourceCrawler

java.lang.Object
- org.apache.tika.batch.FileResourceCrawler

All Implemented Interfaces:

java.util.concurrent.Callable<IFileProcessorFutureResult>

Direct Known Subclasses:

FSDirectoryCrawler, FSListCrawler
```
public abstract class FileResourceCrawler
extends java.lang.Object
implements java.util.concurrent.Callable<IFileProcessorFutureResult>
```

Field Summary

Fields
Modifier and Type	Field and Description
`private int`	`added`
`protected static int`	`ADDED`
`private int`	`considered`
`private DocumentSelector`	`documentSelector`
`private boolean`	`hasCompletedCrawling`
`private boolean`	`isActive`
`protected static org.slf4j.Logger`	`LOG`
`private long`	`maxConsecWaitInMillis`
`private int`	`maxFilesToAdd`
`private int`	`maxFilesToConsider`
`private int`	`numConsumers`
`private static long`	`PAUSE_INCREMENT_MILLIS`
`private java.util.concurrent.ArrayBlockingQueue<FileResource>`	`queue`
`private boolean`	`shutDownNoPoison`
`protected static int`	`SKIPPED`
`protected static int`	`STOP_NOW`
`private boolean`	`timedOut`

Constructor Summary

Constructors
Constructor and Description

FileResourceCrawler(java.util.concurrent.ArrayBlockingQueue<FileResource> queue, int numConsumers)

Constructors
Constructor and Description
`FileResourceCrawler(java.util.concurrent.ArrayBlockingQueue<FileResource> queue, int numConsumers)`

Method Summary

All Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`FileResourceCrawlerFutureResult`	`call()`
`int`	`getAdded()`
`int`	`getConsidered()`
`boolean`	`isActive()` If the crawler stops for any reason, it is no longer active.
`boolean`	`isQueueEmpty()` Use sparingly.
`protected boolean`	`select(Metadata m)`
`void`	`setDocumentSelector(DocumentSelector documentSelector)`
`void`	`setMaxConsecWaitInMillis(long maxConsecWaitInMillis)`
`void`	`setMaxFilesToAdd(int maxFilesToAdd)` Maximum number of files to add.
`void`	`setMaxFilesToConsider(int maxFilesToConsider)` Maximum number of files to consider.
`private void`	`shutdown()`
`void`	`shutDownNoPoison()` Set to true to shut down the FileResourceCrawler without adding poison.
`abstract void`	`start()` Implement this to control the addition of FileResources.
`protected int`	`tryToAdd(FileResource fileResource)`
`boolean`	`wasTimedOut()` Returns whether the crawler timed out while trying to add a resource to the queue.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - LOG
```
protected static final org.slf4j.Logger LOG
```
  - SKIPPED
```
protected static final int SKIPPED
```
    See Also:
    
    Constant Field Values
  - ADDED
```
protected static final int ADDED
```
    See Also:
    
    Constant Field Values
  - STOP_NOW
```
protected static final int STOP_NOW
```
    See Also:
    
    Constant Field Values
  - hasCompletedCrawling
```
private volatile boolean hasCompletedCrawling
```
  - shutDownNoPoison
```
private volatile boolean shutDownNoPoison
```
  - isActive
```
private volatile boolean isActive
```
  - timedOut
```
private volatile boolean timedOut
```
  - PAUSE_INCREMENT_MILLIS
```
private static final long PAUSE_INCREMENT_MILLIS
```
    See Also:
    
    Constant Field Values
  - maxFilesToAdd
```
private int maxFilesToAdd
```
  - maxFilesToConsider
```
private int maxFilesToConsider
```
  - queue
```
private final java.util.concurrent.ArrayBlockingQueue<FileResource> queue
```
  - numConsumers
```
private final int numConsumers
```
  - maxConsecWaitInMillis
```
private long maxConsecWaitInMillis
```
  - documentSelector
```
private DocumentSelector documentSelector
```
  - added
```
private int added
```
  - considered
```
private int considered
```
- Constructor Detail
  - FileResourceCrawler
```
public FileResourceCrawler(java.util.concurrent.ArrayBlockingQueue<FileResource> queue,
                           int numConsumers)
```
    Parameters:
    
    queue - shared queue
    
    numConsumers - number of consumers (needs to know how many poisons to add when done)
- Method Detail
  - start
```
public abstract void start()
                    throws java.lang.InterruptedException
```
    Implement this to control the addition of FileResources. Call tryToAdd(org.apache.tika.batch.FileResource) to add FileResources to the queue.
    
    Throws:
    
    java.lang.InterruptedException
  - call
```
public FileResourceCrawlerFutureResult call()
```
    Specified by:
    
    call in interface java.util.concurrent.Callable<IFileProcessorFutureResult>
  - tryToAdd
```
protected int tryToAdd(FileResource fileResource)
                throws java.lang.InterruptedException
```
    Parameters:
    
    fileResource - resource to add
    
    Returns:
    
    int status of the attempt (SKIPPED, ADDED, STOP_NOW) to add the resource to the queue.
    
    Throws:
    
    java.lang.InterruptedException
  - shutdown
```
private void shutdown()
               throws java.lang.InterruptedException
```
    Throws:
    
    java.lang.InterruptedException
  - isActive
```
public boolean isActive()
```
    If the crawler stops for any reason, it is no longer active.
    
    Returns:
    
    whether crawler is active or not
  - setMaxConsecWaitInMillis
```
public void setMaxConsecWaitInMillis(long maxConsecWaitInMillis)
```
  - setDocumentSelector
```
public void setDocumentSelector(DocumentSelector documentSelector)
```
  - getConsidered
```
public int getConsidered()
```
  - select
```
protected boolean select(Metadata m)
```
  - setMaxFilesToAdd
```
public void setMaxFilesToAdd(int maxFilesToAdd)
```
    Maximum number of files to add. If maxFilesToAdd < 0 (default), then this crawler will add all documents.
    
    Parameters:
    
    maxFilesToAdd - maximum number of files to add to the queue
  - setMaxFilesToConsider
```
public void setMaxFilesToConsider(int maxFilesToConsider)
```
    Maximum number of files to consider. A file is considered whether or not the DocumentSelector selects a document.
    If maxFilesToConsider < 0 (default), then this crawler will add all documents.
    
    Parameters:
    
    maxFilesToConsider - maximum number of files to consider adding to the queue
  - isQueueEmpty
```
public boolean isQueueEmpty()
```
    Use sparingly. This synchronizes on the queue!
    
    Returns:
    
    whether this queue contains any non-poison file resources
  - wasTimedOut
```
public boolean wasTimedOut()
```
    Returns whether the crawler timed out while trying to add a resource to the queue.
    If the crawler timed out while trying to add poison, this is not set to true.
    
    Returns:
    
    whether this was timed out or not
  - getAdded
```
public int getAdded()
```
    Returns:
    
    number of files that this crawler added to the queue
  - shutDownNoPoison
```
public void shutDownNoPoison()
```
    Set to true to shut down the FileResourceCrawler without adding poison. Do this only if you've already called another mechanism to request that consumers shut down. This prevents a potential deadlock issue where the crawler is trying to add to the queue, but it is full.

Class FileResourceCrawler

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

LOG

SKIPPED

ADDED

STOP_NOW

hasCompletedCrawling

shutDownNoPoison

isActive

timedOut

PAUSE_INCREMENT_MILLIS

maxFilesToAdd

maxFilesToConsider

queue

numConsumers

maxConsecWaitInMillis

documentSelector

added

considered

Constructor Detail

FileResourceCrawler

Method Detail

start

call

tryToAdd

shutdown

isActive

setMaxConsecWaitInMillis

setDocumentSelector

getConsidered

select

setMaxFilesToAdd

setMaxFilesToConsider

isQueueEmpty

wasTimedOut

getAdded

shutDownNoPoison