Package opennlp.tools.tokenize

# Class TokenizerME

java.lang.Object
opennlp.tools.tokenize.TokenizerME

All Implemented Interfaces:
  `opennlp.tools.ml.Probabilistic`, `opennlp.tools.tokenize.Tokenizer`

@ThreadSafe public class TokenizerME extends Object implements opennlp.tools.ml.Probabilistic
A `Tokenizer` for converting raw text into separated tokens. It uses Maximum Entropy to make its decisions. The features are loosely based off of Jeff Reynar's UPenn thesis "Topic Segmentation: Algorithms and Applications.", which is available from his homepage: http://www.cis.upenn.edu/~jcreynar.

This implementation needs a statistical model to tokenize a text which reproduces the tokenization observed in the training data used to create the model. The `TokenizerModel` class encapsulates that model and provides methods to create it from the binary representation.

A tokenizer instance is thread-safe. One tokenizer can be shared across multiple threads to save memory.

Note: In container environments with classloader isolation (e.g. Jakarta EE), ensure instances do not outlive the application's lifecycle, as underlying components use `ThreadLocal` state that may pin the classloader.

To train a new model, use `train(ObjectStream, TokenizerFactory, TrainingParameters)`.

Sample usage:

`InputStream modelIn;
...
TokenizerModel model = TokenizerModel(modelIn);
Tokenizer tokenizer = new TokenizerME(model);
String tokens[] = tokenizer.tokenize("A sentence to be tokenized.");`

See Also:
  - `Tokenizer`
  - `TokenizerModel`
  - `TokenSample`
  - `Probabilistic`

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final String`
  `NO_SPLIT`
  Constant indicates no token split.
  `static final String`
  `SPLIT`
  Constant indicates a token split.

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `TokenizerME(String language)`
  Initializes a `TokenizerME` by downloading a default model.
  `TokenizerME(TokenizerModel model)`
  Instantiates a `TokenizerME` with an existing `TokenizerModel`.
  `TokenizerME(TokenizerModel model, Dictionary abbDict)`
  Instantiates a `TokenizerME` with an existing `TokenizerModel`.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `clearThreadLocalState()`
  Removes thread-local state to prevent classloader leaks in container environments.
  `double[]`
  `getTokenProbabilities()`
  Deprecated, for removal: This API element is subject to removal in a future version.
  Use `probs()` instead.
  `double[]`
  `probs()`
  The sequence was determined based on the previous call to `tokenizePos(String)`.
  `void`
  `setKeepNewLines(boolean arg0)`
  `String[]`
  `tokenize(String arg0)`
  `opennlp.tools.util.Span[]`
  `tokenizePos(String d)`
  Tokenizes the string.
  `static TokenizerModel`
  `train(opennlp.tools.util.ObjectStream<opennlp.tools.tokenize.TokenSample> samples, TokenizerFactory factory, opennlp.tools.util.TrainingParameters mlParams)`
  Trains a model for the `TokenizerME`.
  `boolean`
  `useAlphaNumericOptimization()`

  ### Methods inherited from class java.lang.Object

  `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### SPLIT

    public static final String SPLIT
    Constant indicates a token split.
    See Also:
      - Constant Field Values

  - ### NO_SPLIT

    public static final String NO_SPLIT
    Constant indicates no token split.
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### TokenizerME

    public TokenizerME(String language) throws IOException
    Initializes a `TokenizerME` by downloading a default model.
    Parameters:
      `language` - The language of the tokenizer.
    Throws:
      `IOException` - Thrown if the model cannot be downloaded or saved.

  - ### TokenizerME

    public TokenizerME(TokenizerModel model)
    Instantiates a `TokenizerME` with an existing `TokenizerModel`.
    Parameters:
      `model` - The `TokenizerModel` to be used.

  - ### TokenizerME

    public TokenizerME(TokenizerModel model, Dictionary abbDict)
    Instantiates a `TokenizerME` with an existing `TokenizerModel`.
    Parameters:
      `model` - The `TokenizerModel` to be used.
      `abbDict` - The `Dictionary` to be used. It must fit the language of the `model`.

- ## Method Details

  - ### probs

    public double[] probs()
    The sequence was determined based on the previous call to `tokenizePos(String)`.
    Specified by:
      `probs` in interface `opennlp.tools.ml.Probabilistic`
    Returns:
      an array with the same number of probabilities as tokens were sent to the computational method when `tokenizePos(String)` was last called; if not applicable, an empty array

  - ### getTokenProbabilities

    @Deprecated(forRemoval=true, since="2.5.5") public double[] getTokenProbabilities()
    Deprecated, for removal: This API element is subject to removal in a future version.
    Use `probs()` instead.
    Returns:
      the probabilities associated with the most recent calls to `tokenizePos(String)`; if not applicable, an empty array

  - ### tokenizePos

    public opennlp.tools.util.Span[] tokenizePos(String d)
    Tokenizes the string.
    Specified by:
      `tokenizePos` in interface `opennlp.tools.tokenize.Tokenizer`
    Parameters:
      `d` - The string to be tokenized.
    Returns:
      A `Span` array containing individual tokens as elements.

  - ### clearThreadLocalState

    public void clearThreadLocalState()
    Removes thread-local state to prevent classloader leaks in container environments. Call when the thread is returned to a pool or the tokenizer is no longer needed.

  - ### train

    public static TokenizerModel train(opennlp.tools.util.ObjectStream<opennlp.tools.tokenize.TokenSample> samples, TokenizerFactory factory, opennlp.tools.util.TrainingParameters mlParams) throws IOException
    Trains a model for the `TokenizerME`.
    Parameters:
      `samples` - The samples used for the training.
      `factory` - A `TokenizerFactory` to get resources from.
      `mlParams` - The machine learning `train parameters`.
    Returns:
      A trained `TokenizerModel`.
    Throws:
      `IOException` - Thrown during IO on a temp file created during training, or if reading from the `ObjectStream` fails.

  - ### useAlphaNumericOptimization

    public boolean useAlphaNumericOptimization()
    Returns:
      `true` if the tokenizer uses alphanumeric optimization, `false` otherwise.

  - ### tokenize

    public String[] tokenize(String arg0)
    Specified by:
      `tokenize` in interface `opennlp.tools.tokenize.Tokenizer`

  - ### setKeepNewLines

    public void setKeepNewLines(boolean arg0)