Package opennlp.tools.tokenize

# Class BertTokenizer

java.lang.Object
opennlp.tools.tokenize.BertTokenizer

All Implemented Interfaces:
  `Tokenizer`

public class BertTokenizer extends Object implements Tokenizer
A `Tokenizer` implementation of the full BERT tokenization pipeline: basic tokenization (text normalization) followed by wordpiece tokenization.

The basic tokenization stage reproduces the reference BERT `BasicTokenizer`:

1. Removal of control characters and normalization of all whitespace to single spaces.
2. Whitespace isolation of CJK ideographs.
3. For uncased models: lower casing and accent stripping (Unicode NFD decomposition with removal of combining marks).
4. Isolation of every punctuation character as its own token.

The normalized text is then split into subwords by a `WordpieceTokenizer` sharing the same vocabulary and special tokens.

This pipeline is required for correct results with BERT-style models: feeding raw text directly to `WordpieceTokenizer` maps every token that does not literally appear in the vocabulary - for uncased models that includes every capitalized word - to the unknown token.

Whether to use the lower casing variant is a property of the model: uncased models (for example `bert-base-uncased` and the `sentence-transformers` models derived from it) require it, cased models must not use it. Accent stripping is coupled to lower casing, as in the reference implementation's default (`strip_accents` follows `do_lower_case` unless overridden).

For reference see:

- https://github.com/google-research/bert (`tokenization.py`)

See Also:
  - `WordpieceTokenizer`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `BertTokenizer(Set<String> vocabulary)`
  Initializes a `BertTokenizer` for an uncased BERT model, with lower casing and accent stripping enabled.
  `BertTokenizer(Set<String> vocabulary, boolean lowerCase)`
  Initializes a `BertTokenizer` with BERT special tokens.
  `BertTokenizer(Set<String> vocabulary, boolean lowerCase, String classificationToken, String separatorToken, String unknownToken)`
  Initializes a `BertTokenizer` with custom special tokens, for models like RoBERTa that do not use the BERT defaults.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `String[]`
  `tokenize(String text)`
  Tokenizes the given text into wordpieces, surrounded by the classification and separator tokens.
  `Span[]`
  `tokenizePos(String text)`
  Not supported: wordpiece tokens (subwords, `##` continuations and special tokens) have no faithful character spans in the original text.

  ### Methods inherited from class java.lang.Object

  `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Constructor Details

  - ### BertTokenizer

    public BertTokenizer(Set<String> vocabulary)
    Initializes a `BertTokenizer` for an uncased BERT model, with lower casing and accent stripping enabled.
    Parameters:
      `vocabulary` - The wordpiece vocabulary. Must not be `null`.

  - ### BertTokenizer

    public BertTokenizer(Set<String> vocabulary, boolean lowerCase)
    Initializes a `BertTokenizer` with BERT special tokens.
    Parameters:
      `vocabulary` - The wordpiece vocabulary. Must not be `null`.
      `lowerCase` - `true` for uncased models (lower casing and accent stripping), `false` for cased models.

  - ### BertTokenizer

    public BertTokenizer(Set<String> vocabulary, boolean lowerCase, String classificationToken, String separatorToken, String unknownToken)
    Initializes a `BertTokenizer` with custom special tokens, for models like RoBERTa that do not use the BERT defaults.
    Parameters:
      `vocabulary` - The wordpiece vocabulary. Must not be `null`.
      `lowerCase` - `true` for uncased models (lower casing and accent stripping), `false` for cased models.
      `classificationToken` - The CLS token.
      `separatorToken` - The SEP token.
      `unknownToken` - The UNK token.

- ## Method Details

  - ### tokenize

    public String[] tokenize(String text)
    Tokenizes the given text into wordpieces, surrounded by the classification and separator tokens.
    Specified by:
      `tokenize` in interface `Tokenizer`
    Parameters:
      `text` - The text to tokenize. Must not be `null`.
    Returns:
      The wordpiece tokens.

  - ### tokenizePos

    public Span[] tokenizePos(String text)
    Not supported: wordpiece tokens (subwords, `##` continuations and special tokens) have no faithful character spans in the original text.
    Specified by:
      `tokenizePos` in interface `Tokenizer`
    Parameters:
      `text` - The string to be tokenized.
    Returns:
      The `spans (offsets into `s`)` for each token as the individuals array elements.
    Throws:
      `UnsupportedOperationException` - Always.