Package opennlp.tools.tokenize

# Class BPEModel

java.lang.Object
opennlp.tools.util.model.BaseModel
opennlp.tools.tokenize.BPEModel

All Implemented Interfaces:
  `Serializable`, `opennlp.tools.util.model.ArtifactProvider`

public final class BPEModel extends BaseModel
The `BPEModel` stores learned BPE merge operations and can be serialized and deserialized for reuse.

A model is created by the `BPETokenizerTrainer` and contains an ordered list of `BPETokenizer.SymbolPair` merge operations that define the BPE vocabulary. The model is persisted as a standard OpenNLP ZIP package with a `bpe.merges` artifact containing the merge rules.

Usage:

     // Create via training
     BPETokenizerTrainer trainer = new BPETokenizerTrainer();
     BPEModel model = trainer.train(corpus, 10000, "en");

     // Save to disk
     model.serialize(Path.of("bpe-en.bin"));

     // Load from disk
     BPEModel loaded = new BPEModel(Path.of("bpe-en.bin"));

     // Use for tokenization
     BPETokenizer tokenizer = new BPETokenizer(loaded);

See Also:
  - `BPETokenizer`
  - `BPETokenizerTrainer`
  - `BPETokenizerFactory`
  - Serialized Form

- ## Field Summary

  ### Fields inherited from class opennlp.tools.util.model.BaseModel

  `TRAINING_CUTOFF_PROPERTY, TRAINING_EVENTHASH_PROPERTY, TRAINING_ITERATIONS_PROPERTY`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `BPEModel(File modelFile)`
  Initializes a `BPEModel` from a `File`.
  `BPEModel(InputStream in)`
  Initializes a `BPEModel` from an `InputStream`.
  `BPEModel(URL modelURL)`
  Initializes a `BPEModel` from a `URL`.
  `BPEModel(Path modelPath)`
  Initializes a `BPEModel` from a `Path`.
  `BPEModel(List<BPETokenizer.SymbolPair> merges, Map<String,String> manifestInfoEntries, BPETokenizerFactory factory)`
  Creates a `BPEModel` from trained merge rules.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `BPETokenizerFactory`
  `getFactory()`
  `List<BPETokenizer.SymbolPair>`
  `getMerges()`

  ### Methods inherited from class opennlp.tools.util.model.BaseModel

  `getArtifact, getLanguage, getManifestProperty, getVersion, isLoadedFromSerialized, serialize, serialize, serialize`

  ### Methods inherited from class java.lang.Object

  `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Constructor Details

  - ### BPEModel

    public BPEModel(List<BPETokenizer.SymbolPair> merges, Map<String,String> manifestInfoEntries, BPETokenizerFactory factory)
    Creates a `BPEModel` from trained merge rules.
    Parameters:
      `merges` - The ordered list of merge operations. Must not be `null`.
      `manifestInfoEntries` - Additional manifest info.
      `factory` - The `BPETokenizerFactory`.

  - ### BPEModel

    public BPEModel(InputStream in) throws IOException
    Initializes a `BPEModel` from an `InputStream`.
    Parameters:
      `in` - The `InputStream` for loading the model.
    Throws:
      `IOException` - Thrown if IO errors occurred.

  - ### BPEModel

    public BPEModel(File modelFile) throws IOException
    Initializes a `BPEModel` from a `File`.
    Parameters:
      `modelFile` - The `File` for loading the model.
    Throws:
      `IOException` - Thrown if IO errors occurred.

  - ### BPEModel

    public BPEModel(Path modelPath) throws IOException
    Initializes a `BPEModel` from a `Path`.
    Parameters:
      `modelPath` - The `Path` for loading the model.
    Throws:
      `IOException` - Thrown if IO errors occurred.

  - ### BPEModel

    public BPEModel(URL modelURL) throws IOException
    Initializes a `BPEModel` from a `URL`.
    Parameters:
      `modelURL` - The `URL` for loading the model.
    Throws:
      `IOException` - Thrown if IO errors occurred.

- ## Method Details

  - ### getFactory

    public BPETokenizerFactory getFactory()
    Returns:
      The active `BPETokenizerFactory`.

  - ### getMerges

    public List<BPETokenizer.SymbolPair> getMerges()
    Returns:
      An unmodifiable, ordered list of BPE merge operations stored in this model.