Package opennlp.tools.tokenize

# Class SimpleTokenizer

java.lang.Object
opennlp.tools.tokenize.SimpleTokenizer

All Implemented Interfaces:
  `Tokenizer`

public class SimpleTokenizer extends Object
A basic `Tokenizer` implementation which performs tokenization using character classes.

To obtain an instance of this tokenizer use the static final `INSTANCE` field.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final SimpleTokenizer`
  `INSTANCE`
  Use this static reference to retrieve an instance of the `SimpleTokenizer`.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `setKeepNewLines(boolean keepNewLines)`
  Switches whether to keep new lines or not.
  `String[]`
  `tokenize(String s)`
  Splits a string into its atomic parts.
  `Span[]`
  `tokenizePos(String s)`
  Finds the boundaries of atomic parts in a string.

  ### Methods inherited from class java.lang.Object

  `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### INSTANCE

    public static final SimpleTokenizer INSTANCE
    Use this static reference to retrieve an instance of the `SimpleTokenizer`.

- ## Method Details

  - ### tokenizePos

    public Span[] tokenizePos(String s)
    Description copied from interface: `Tokenizer`
    Finds the boundaries of atomic parts in a string.
    Parameters:
      `s` - The string to be tokenized.
    Returns:
      The `spans (offsets into `s`)` for each token as the individuals array elements.

  - ### tokenize

    public String[] tokenize(String s)
    Description copied from interface: `Tokenizer`
    Splits a string into its atomic parts.
    Specified by:
      `tokenize` in interface `Tokenizer`
    Parameters:
      `s` - The string to be tokenized.
    Returns:
      The String[] with the individual tokens as the array elements.

  - ### setKeepNewLines

    public void setKeepNewLines(boolean keepNewLines)
    Switches whether to keep new lines or not.
    Parameters:
      `keepNewLines` - `True` if new lines are kept, `false` otherwise.