org.apache.tika.language

Class LanguageProfilerBuilder

java.lang.Object

org.apache.tika.language.LanguageProfilerBuilder

Deprecated.

@Deprecated
public class LanguageProfilerBuilder
extends java.lang.Object

This class runs a ngram analysis over submitted text, results might be used for automatic language identification. The similarity calculation is at experimental level. You have been warned. Methods are provided to build new NGramProfiles profiles.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) static class	LanguageProfilerBuilder.NGramEntry Deprecated. Inner class that describes a NGram
private static class	LanguageProfilerBuilder.QuickStringBuffer Deprecated.

Field Summary

Fields

Modifier and Type	Field and Description
(package private) static int	ABSOLUTE_MAX_NGRAM_LENGTH Deprecated. The maximum length allowed for a ngram.
(package private) static int	ABSOLUTE_MIN_NGRAM_LENGTH Deprecated. The minimum length allowed for a ngram.
(package private) static int	DEFAULT_MAX_NGRAM_LENGTH Deprecated. The default max length of ngram
(package private) static int	DEFAULT_MIN_NGRAM_LENGTH Deprecated. The default min length of ngram
(package private) static java.lang.String	FILE_EXTENSION Deprecated. The ngram profile file extension
(package private) static int	MAX_SIZE Deprecated. The profile max size (number of ngrams of the same size)
private int	maxLength Deprecated. The max length of ngram
private int	minLength Deprecated. The min length of ngram
private java.lang.String	name Deprecated. The profile's name
private int[]	ngramcounts Deprecated. The total number of ngrams occurences
private java.util.Map<java.lang.CharSequence,LanguageProfilerBuilder.NGramEntry>	ngrams Deprecated. An index of the ngrams of the profile
private static java.lang.String	SEP_CHARSEQ Deprecated. The String form of the separator char
(package private) static char	SEPARATOR Deprecated. separator char
private java.util.List<LanguageProfilerBuilder.NGramEntry>	sorted Deprecated. The NGrams of this profile sorted on the number of occurrences
private LanguageProfilerBuilder.QuickStringBuffer	word Deprecated. A StringBuffer used during analysis

Constructor Summary

Constructors

Constructor and Description
LanguageProfilerBuilder(java.lang.String name) Deprecated. Constructs a new ngram profile where minlen=3, maxlen=3
LanguageProfilerBuilder(java.lang.String name, int minlen, int maxlen) Deprecated. Constructs a new ngram profile

Method Summary

Modifier and Type	Method and Description
private void	add(java.lang.CharSequence cs) Deprecated. Adds ngrams from a single word in this profile
private void	add(LanguageProfilerBuilder.QuickStringBuffer word) Deprecated. Adds the last NGrams from the specified word.
void	add(java.lang.StringBuffer word) Deprecated. Adds ngrams from a single word to this profile
private void	add(java.lang.StringBuffer word, int n) Deprecated.
void	analyze(java.lang.StringBuilder text) Deprecated. Analyzes a piece of text
static LanguageProfilerBuilder	create(java.lang.String name, java.io.InputStream is, java.lang.String encoding) Deprecated. Creates a new Language profile from (preferably quite large - 5-10k of lines) text file
java.lang.String	getName() Deprecated.
float	getSimilarity(LanguageProfilerBuilder another) Deprecated. Calculates a score how well NGramProfiles match each other
java.util.List<LanguageProfilerBuilder.NGramEntry>	getSorted() Deprecated. Returns a sorted list of ngrams (sort done by 1.
void	load(java.io.InputStream is) Deprecated. Loads a ngram profile from an InputStream (assumes UTF-8 encoded content)
static void	main(java.lang.String[] args) Deprecated. main method used for testing only
protected void	normalize() Deprecated. Normalizes the profile (calculates the ngrams frequencies)
void	save(java.io.OutputStream os) Deprecated. Writes NGramProfile content into OutputStream, content is outputted with UTF-8 encoding
java.lang.String	toString() Deprecated.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

ABSOLUTE_MIN_NGRAM_LENGTH

static final int ABSOLUTE_MIN_NGRAM_LENGTH

Deprecated.

The minimum length allowed for a ngram.

See Also:

Constant Field Values

ABSOLUTE_MAX_NGRAM_LENGTH

static final int ABSOLUTE_MAX_NGRAM_LENGTH

Deprecated.

The maximum length allowed for a ngram.

See Also:

Constant Field Values

DEFAULT_MIN_NGRAM_LENGTH

static final int DEFAULT_MIN_NGRAM_LENGTH

Deprecated.

The default min length of ngram

See Also:

Constant Field Values

DEFAULT_MAX_NGRAM_LENGTH

static final int DEFAULT_MAX_NGRAM_LENGTH

Deprecated.

The default max length of ngram

See Also:

Constant Field Values

FILE_EXTENSION

static final java.lang.String FILE_EXTENSION

Deprecated.

The ngram profile file extension

See Also:

Constant Field Values

MAX_SIZE

static final int MAX_SIZE

Deprecated.

The profile max size (number of ngrams of the same size)

See Also:

Constant Field Values

SEPARATOR

static final char SEPARATOR

Deprecated.

separator char

See Also:

Constant Field Values

SEP_CHARSEQ

private static final java.lang.String SEP_CHARSEQ

Deprecated.

The String form of the separator char

name

private java.lang.String name

Deprecated.

The profile's name

sorted

private java.util.List<LanguageProfilerBuilder.NGramEntry> sorted

Deprecated.

The NGrams of this profile sorted on the number of occurrences

minLength

private int minLength

Deprecated.

The min length of ngram

maxLength

private int maxLength

Deprecated.

The max length of ngram

ngramcounts

private int[] ngramcounts

Deprecated.

The total number of ngrams occurences

ngrams

private java.util.Map<java.lang.CharSequence,LanguageProfilerBuilder.NGramEntry> ngrams

Deprecated.

An index of the ngrams of the profile

word

private LanguageProfilerBuilder.QuickStringBuffer word

Deprecated.

A StringBuffer used during analysis

Constructor Detail

LanguageProfilerBuilder

public LanguageProfilerBuilder(java.lang.String name,
                               int minlen,
                               int maxlen)

Deprecated.

Constructs a new ngram profile

Parameters:

name - is the name of the profile

minlen - is the min length of ngram sequences

maxlen - is the max length of ngram sequences

LanguageProfilerBuilder

public LanguageProfilerBuilder(java.lang.String name)

Deprecated.

Constructs a new ngram profile where minlen=3, maxlen=3

Parameters:

name - is a name of profile, usually two length string

Since:

Tika 1.0

Method Detail

getName

public java.lang.String getName()

Deprecated.

Returns:

Returns the name.

add

public void add(java.lang.StringBuffer word)

Deprecated.

Adds ngrams from a single word to this profile

Parameters:

word - is the word to add

add

private void add(LanguageProfilerBuilder.QuickStringBuffer word)

Deprecated.

Adds the last NGrams from the specified word.

add

private void add(java.lang.CharSequence cs)

Deprecated.

Adds ngrams from a single word in this profile

Parameters:

cs - char sequence to add

analyze

public void analyze(java.lang.StringBuilder text)

Deprecated.

Analyzes a piece of text

Parameters:

text - the text to be analyzed

add

private void add(java.lang.StringBuffer word,
                 int n)

Deprecated.

Parameters:

word -

n - sequence length

normalize

protected void normalize()

Deprecated.

Normalizes the profile (calculates the ngrams frequencies)

getSorted

public java.util.List<LanguageProfilerBuilder.NGramEntry> getSorted()

Deprecated.

Returns a sorted list of ngrams (sort done by 1. frequency 2. sequence)

Returns:

sorted vector of ngrams

toString

public java.lang.String toString()

Deprecated.

Overrides:

toString in class java.lang.Object

getSimilarity

public float getSimilarity(LanguageProfilerBuilder another)
                    throws TikaException

Deprecated.

Calculates a score how well NGramProfiles match each other

Parameters:

another - ngram profile to compare against

Returns:

similarity 0=exact match

Throws:

TikaException - if could not calculate a score

load

public void load(java.io.InputStream is)
          throws java.io.IOException

Deprecated.

Loads a ngram profile from an InputStream (assumes UTF-8 encoded content)

Parameters:

is - the InputStream to read

Throws:

java.io.IOException

create

public static LanguageProfilerBuilder create(java.lang.String name,
                                             java.io.InputStream is,
                                             java.lang.String encoding)
                                      throws TikaException

Deprecated.

Creates a new Language profile from (preferably quite large - 5-10k of lines) text file

Parameters:

name - to be given for the profile

is - a stream to be read

encoding - is the encoding of stream

Throws:

TikaException - if could not create a language profile

save

public void save(java.io.OutputStream os)
          throws java.io.IOException

Deprecated.

Writes NGramProfile content into OutputStream, content is outputted with UTF-8 encoding

Parameters:

os - the Stream to output to

Throws:

java.io.IOException

main

public static void main(java.lang.String[] args)

Deprecated.

main method used for testing only

Parameters:

args -