Package opennlp.tools.tokenize

# Class SimpleTokenizer

java.lang.Object
opennlp.tools.tokenize.SimpleTokenizer

All Implemented Interfaces:
  `opennlp.tools.tokenize.Tokenizer`

public class SimpleTokenizer extends Object
A basic `Tokenizer` implementation which performs tokenization using character classes.

To obtain an instance of this tokenizer use the static final `INSTANCE` field.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final SimpleTokenizer`
  `INSTANCE`
  Use this static reference to retrieve an instance of the `SimpleTokenizer`.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `setKeepNewLines(boolean arg0)`
  `String[]`
  `tokenize(String arg0)`
  `opennlp.tools.util.Span[]`
  `tokenizePos(String s)`

  ### Methods inherited from class java.lang.Object

  `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### INSTANCE

    public static final SimpleTokenizer INSTANCE
    Use this static reference to retrieve an instance of the `SimpleTokenizer`.

- ## Method Details

  - ### tokenizePos

    public opennlp.tools.util.Span[] tokenizePos(String s)

  - ### tokenize

    public String[] tokenize(String arg0)
    Specified by:
      `tokenize` in interface `opennlp.tools.tokenize.Tokenizer`

  - ### setKeepNewLines

    public void setKeepNewLines(boolean arg0)