Package opennlp.tools.tokenize

# Class TokenSampleStream

java.lang.Object
opennlp.tools.util.FilterObjectStream<String,opennlp.tools.tokenize.TokenSample>
opennlp.tools.tokenize.TokenSampleStream

All Implemented Interfaces:
  `AutoCloseable`, `opennlp.tools.util.ObjectStream<opennlp.tools.tokenize.TokenSample>`

public class TokenSampleStream extends FilterObjectStream<String,opennlp.tools.tokenize.TokenSample>
This class is a `stream filter` which reads in string encoded samples and creates `samples` out of them. The input string sample is tokenized if a whitespace or the special separator chars occur.

Sample:
"token1 token2 token3<SPLIT>token4"
The tokens `token1` and `token2` are separated by a whitespace, `token3` and `token4` are separated by the special character sequence. In this case, the default split sequence applies.

Note: The sequence must be unique in the input string and is not escaped.

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `TokenSampleStream(opennlp.tools.util.ObjectStream<String> sentences)`
  Initializes a `instance`.
  `TokenSampleStream(opennlp.tools.util.ObjectStream<String> samples, String separatorChars)`
  Initializes a `instance`.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `opennlp.tools.tokenize.TokenSample`
  `read()`

  ### Methods inherited from class opennlp.tools.util.FilterObjectStream

  `close, reset`

  ### Methods inherited from class java.lang.Object

  `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Constructor Details

  - ### TokenSampleStream

    public TokenSampleStream(opennlp.tools.util.ObjectStream<String> samples, String separatorChars)
    Initializes a `instance`.
    Parameters:
      `samples` - A plain text `line stream`. Must not be `null`.
      `separatorChars` - The characters to be considered separators. See `TokenSample.DEFAULT_SEPARATOR_CHARS`. Must not be `null`.

  - ### TokenSampleStream

    public TokenSampleStream(opennlp.tools.util.ObjectStream<String> sentences)
    Initializes a `instance`.
    Parameters:
      `sentences` - A plain text `line stream`. Must not be `null`.

- ## Method Details

  - ### read

    public opennlp.tools.tokenize.TokenSample read() throws IOException
    Throws:
      `IOException`