Package opennlp.tools.stopword

# Class StopwordFilteringTokenizer

java.lang.Object
opennlp.tools.stopword.StopwordFilteringTokenizer

All Implemented Interfaces:
  `opennlp.tools.tokenize.Tokenizer`

@ThreadSafe public final class StopwordFilteringTokenizer extends Object implements opennlp.tools.tokenize.Tokenizer
A `Tokenizer` decorator which delegates tokenization to a wrapped `Tokenizer` and then removes any tokens identified as stopwords by the supplied `StopwordFilter`.

Both `tokenize(String)` and `tokenizePos(String)` apply the filter using the same greedy longest-match window scan, so single-token (1-gram) and multi-token (n-gram) stopword entries are dropped identically across `tokenize(String)`, `tokenizePos(String)` and `StopwordFilterStream`. For `tokenizePos(String)` the `Spans` covering a matched entry are dropped while the offsets of the remaining spans are kept intact (they continue to refer to positions in the original input string).

Instances are immutable and therefore safe for concurrent use provided that both the wrapped `Tokenizer` and the `StopwordFilter` are thread-safe. `DictionaryStopwordFilter` is unconditionally thread-safe; combined with a thread-safe delegate tokenizer (e.g. `SimpleTokenizer.INSTANCE`) the resulting decorator is thread-safe with no further synchronization required.

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `StopwordFilteringTokenizer(opennlp.tools.tokenize.Tokenizer delegate, opennlp.tools.stopword.StopwordFilter filter)`
  Initializes a `StopwordFilteringTokenizer`.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `String[]`
  `tokenize(String s)`
  Tokenizes the supplied string with the wrapped `Tokenizer` and then removes any tokens which the `StopwordFilter` considers a stopword.
  `opennlp.tools.util.Span[]`
  `tokenizePos(String s)`
  Computes token spans with the wrapped `Tokenizer` and then drops the spans covering any stopword entry according to the `StopwordFilter`.

  ### Methods inherited from class java.lang.Object

  `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Constructor Details

  - ### StopwordFilteringTokenizer

    public StopwordFilteringTokenizer(opennlp.tools.tokenize.Tokenizer delegate, opennlp.tools.stopword.StopwordFilter filter)
    Initializes a `StopwordFilteringTokenizer`.
    Parameters:
      `delegate` - The underlying `Tokenizer` that produces the raw tokens. Must not be `null`.
      `filter` - The `StopwordFilter` which decides whether a token is a stopword. Must not be `null`.
    Throws:
      `IllegalArgumentException` - if `delegate` or `filter` is `null`.

- ## Method Details

  - ### tokenize

    public String[] tokenize(String s)
    Tokenizes the supplied string with the wrapped `Tokenizer` and then removes any tokens which the `StopwordFilter` considers a stopword.
    Specified by:
      `tokenize` in interface `opennlp.tools.tokenize.Tokenizer`
    Parameters:
      `s` - The string to be tokenized.
    Returns:
      The remaining tokens in their original order.

  - ### tokenizePos

    public opennlp.tools.util.Span[] tokenizePos(String s)
    Computes token spans with the wrapped `Tokenizer` and then drops the spans covering any stopword entry according to the `StopwordFilter`. A greedy left-to-right window scan mirrors `StopwordFilter.filter(String[])`: at each position the longest window of consecutive spans whose covered texts form a registered entry is removed; otherwise the current span is kept and the scan advances by one. This way multi-word (n-gram) entries are dropped here exactly as they are by `tokenize(String)`. The relative order and the offsets of the surviving spans are preserved.
    Specified by:
      `tokenizePos` in interface `opennlp.tools.tokenize.Tokenizer`
    Parameters:
      `s` - The string to be tokenized.
    Returns:
      The remaining `Spans` in their original order.