Mila::Data::BpeTokenizer Class Reference

Unified BPE tokenizer targeting GPT-2, Llama 3.x, and Mistral model families. More...

Inheritance diagram for Mila::Data::BpeTokenizer:

Public Member Functions
	BpeTokenizer (BpeVocabulary vocab)
std::string	decode (std::span< const TokenId > tokens) override
	Decode a sequence of token IDs back to a UTF-8 string.
std::vector< TokenId >	encode (const std::string &text) override
	Encode text to a sequence of token IDs.
std::optional< TokenId >	getBosTokenId () const override
std::optional< TokenId >	getEosTokenId () const override
std::optional< TokenId >	getPadTokenId () const override
const BpeVocabulary &	getVocab () const
size_t	getVocabSize () const override
bool	isValidToken (TokenId tokenId) const override
std::string	tokenToString (TokenId tokenId) const override

Static Public Member Functions
static BpeTokenizer	load (const std::filesystem::path &path)
	Load a tokenizer from a Mila binary vocabulary file.
static std::shared_ptr< BpeTokenizer >	loadGemma (const std::filesystem::path &path)
	Load a Gemma 4 tokenizer from the binary produced by convert_tokenizer.py.
static std::shared_ptr< BpeTokenizer >	loadGpt2 (const std::filesystem::path &path)
	Load a GPT-2 tokenizer from the binary produced by convert_gpt2_tokenizer.py.
static std::shared_ptr< BpeTokenizer >	loadLlama32 (const std::filesystem::path &path)
	Load a Llama 3.2 tokenizer from the binary produced by convert_llama_tokenizer.py.
static std::shared_ptr< BpeTokenizer >	loadMistral (const std::filesystem::path &vocab_path, const std::filesystem::path &merges_path)
	Load a Mistral tokenizer.

Detailed Description

Unified BPE tokenizer targeting GPT-2, Llama 3.x, and Mistral model families.

Construct from a pre-built vocabulary or via the convenience factory methods:

// GPT-2
auto tok = BpeTokenizer::loadGpt2( "gpt2_tokenizer.bin" );
auto ids = tok->encode( "Hello, world!" );
 
// Llama 3.2
auto tok = BpeTokenizer::loadLlama32( "llama32_tokenizer.bin" );
auto ids = tok->encode( "<|begin_of_text|>Hello, world!" );

The special token pre-pass is enabled whenever the vocabulary registers at least one special token. For GPT-2, this means "<|endoftext|>" is intercepted before BPE runs; for Llama 3.x, the full set of named and extended tokens is intercepted.

Member Function Documentation

decode()

std::string Mila::Data::BpeTokenizer::decode ( std::span< const TokenId > tokens )

inlineoverridevirtual

Decode a sequence of token IDs back to a UTF-8 string.

Each token string is byte-decoded using the GPT-2 style byte mapping. IDs with no vocabulary entry emit a '?' placeholder.

Parameters

tokens

Sequence of token IDs.

Returns

Decoded UTF-8 string.

Implements Mila::Data::Tokenizer.

encode()

std::vector< TokenId > Mila::Data::BpeTokenizer::encode ( const std::string & text )

inlineoverridevirtual

Encode text to a sequence of token IDs.

Performs the special token pre-pass first when the vocabulary has registered special tokens. Plain text segments between special tokens are processed through the standard pre-tokenization and BPE merge pipeline.

Parameters

text	Input text (UTF-8).

Returns

Sequence of token IDs.

Implements Mila::Data::Tokenizer.

getBosTokenId()

std::optional< TokenId > Mila::Data::BpeTokenizer::getBosTokenId ( ) const

inlineoverridevirtual

Implements Mila::Data::Tokenizer.

getEosTokenId()

std::optional< TokenId > Mila::Data::BpeTokenizer::getEosTokenId ( ) const

inlineoverridevirtual

Implements Mila::Data::Tokenizer.

getPadTokenId()

std::optional< TokenId > Mila::Data::BpeTokenizer::getPadTokenId ( ) const

inlineoverridevirtual

Implements Mila::Data::Tokenizer.

getVocabSize()

size_t Mila::Data::BpeTokenizer::getVocabSize ( ) const

inlineoverridevirtual

Implements Mila::Data::Tokenizer.

isValidToken()

bool Mila::Data::BpeTokenizer::isValidToken ( TokenId tokenId ) const

inlineoverridevirtual

Implements Mila::Data::Tokenizer.

load()

BpeTokenizer Mila::Data::BpeTokenizer::load ( const std::filesystem::path & path )

inlinestatic

Load a tokenizer from a Mila binary vocabulary file.

Parameters

path	Path to a vocabulary file written by BpeVocabulary::save().

Returns

Loaded BpeTokenizer instance.

Exceptions

std::runtime_error

on I/O or format errors.

loadGemma()

std::shared_ptr< BpeTokenizer > Mila::Data::BpeTokenizer::loadGemma ( const std::filesystem::path & path )

inlinestatic

Load a Gemma 4 tokenizer from the binary produced by convert_tokenizer.py.

Gemma uses SentencePiece BPE: byte_level=false (raw UTF-8 pieces, U+2581 for spaces) with a Metaspace pre-tokenization and byte fallback.

Parameters

path	Path to the Gemma tokenizer binary.

Returns

Shared tokenizer instance.

Exceptions

std::runtime_error

on I/O or format errors.

loadGpt2()

std::shared_ptr< BpeTokenizer > Mila::Data::BpeTokenizer::loadGpt2 ( const std::filesystem::path & path )

inlinestatic

Load a GPT-2 tokenizer from the binary produced by convert_gpt2_tokenizer.py.

Parameters

path	Path to the GPT-2 tokenizer binary.

Returns

Shared tokenizer instance.

Exceptions

std::runtime_error

on I/O or format errors.

loadLlama32()

std::shared_ptr< BpeTokenizer > Mila::Data::BpeTokenizer::loadLlama32 ( const std::filesystem::path & path )

inlinestatic

Load a Llama 3.2 tokenizer from the binary produced by convert_llama_tokenizer.py.

Parameters

path	Path to the Llama 3.2 tokenizer binary.

Returns

Shared tokenizer instance.

Exceptions

std::runtime_error

on I/O or format errors.

loadMistral()

std::shared_ptr< BpeTokenizer > Mila::Data::BpeTokenizer::loadMistral ( const std::filesystem::path & vocab_path, const std::filesystem::path & merges_path )

inlinestatic

Load a Mistral tokenizer.

Note

Not yet implemented. Provide a Mila binary produced by save() as a workaround.

Exceptions

std::runtime_error

always.

tokenToString()

std::string Mila::Data::BpeTokenizer::tokenToString ( TokenId tokenId ) const

inlineoverridevirtual

Implements Mila::Data::Tokenizer.

---

The documentation for this class was generated from the following file:

• Mila/Src/Data/Tokenizers/Bpe/BpeTokenizer.ixx