Unified Byte Pair Encoding (BPE) vocabulary. More...

Inheritance diagram for Mila::Data::BpeVocabulary:

Public Member Functions
bool	containsToken (const std::string &token) const
	Raw vocabulary membership test (no UNK fallback).
const BpeVocabularyConfig &	getConfig () const
std::optional< size_t >	getMergePriority (const std::string &left, const std::string &right) const
const std::vector< std::pair< std::string, std::string > > &	getMergeRules () const
size_t	getSize () const override
	Get the number of tokens in the vocabulary.
std::optional< TokenId >	getSpecialTokenId (const std::string &token_str) const
	Look up a special token ID by its string representation.
const std::vector< std::pair< std::string, TokenId > > &	getSpecialTokenList () const
	Return the special token list sorted longest-first.
std::optional< std::string >	idToToken (TokenId id) const override
	Map a numeric id back to its token string.
bool	isByteLevel () const
void	save (const fs::path &path) const override
	Serialize the vocabulary to Mila binary format (content version 2).
std::optional< TokenId >	tokenToId (const std::string &token) const override
	Convert a token string to its ID.
Public Member Functions inherited from Mila::Data::TokenizerVocabulary
virtual	~TokenizerVocabulary ()=default
	Virtual destructor.
virtual void	save (const std::filesystem::path &path) const =0
	Serialize the vocabulary to disk at the given path.

Static Public Member Functions
static const std::unordered_map< std::string, unsigned char > &	getByteDecoder ()
static const std::unordered_map< unsigned char, std::string > &	getByteEncoder ()
static BpeVocabulary	load (const fs::path &path)
	Load a vocabulary from Mila binary format (version 2).
static BpeVocabulary	loadGemma (const fs::path &path)
	Load a pretrained Gemma 4 (SentencePiece BPE) vocabulary.
static BpeVocabulary	loadGpt2 (const fs::path &tokenizer_path)
	Load a pretrained GPT-2 vocabulary.
static BpeVocabulary	loadLlama32 (const fs::path &path)
	Load a pretrained Llama 3.2 vocabulary.
static BpeVocabulary	loadMistral (const fs::path &vocab_path, const fs::path &merges_path)
	Load a pretrained Mistral vocabulary.
static BpeVocabulary	loadQwen (const fs::path &path)
	Load a pretrained Qwen 3.x (byte-level BPE) vocabulary.
static BpeVocabulary	train (const std::string &corpus, const BpeVocabularyConfig &config)
	Train a BPE vocabulary from a text corpus.
static BpeVocabulary	trainFromFile (const fs::path &corpus_path, const BpeVocabularyConfig &config)
	Train a BPE vocabulary from a corpus file.

Detailed Description

Unified Byte Pair Encoding (BPE) vocabulary.

Immutable after construction; safe for concurrent reads. Supports training from scratch via BpeTrainer, or loading pretrained vocabularies from:

Mila binary format produced by save() (load)
GPT-2 binary produced by convert_gpt2_tokenizer.py (loadGpt2)
Llama 3.2 binary produced by convert_llama_tokenizer.py (loadLlama32)
Gemma 4 binary produced by Gemma/convert_tokenizer.py (loadGemma)
Qwen 3.x binary produced by Qwen/convert_tokenizer.py (loadQwen)

Special tokens are keyed on their string representation (e.g., "<|endoftext|>", "<|begin_of_text|>") and exposed via getSpecialTokenList() for O(n) pre-pass scanning in BpeTokenizer. Extended special tokens from SpecialTokens are registered automatically.

Member Function Documentation

◆ containsToken()

bool Mila::Data::BpeVocabulary::containsToken ( const std::string & token ) const

inline

Raw vocabulary membership test (no UNK fallback).

Unlike tokenToId(), this never substitutes the UNK id, so the SentencePiece encode path can ask "is this exact piece in the vocab?" to decide between using a character directly and byte-fallback.

◆ getSize()

size_t Mila::Data::BpeVocabulary::getSize ( ) const

inlineoverridevirtual

Get the number of tokens in the vocabulary.

Returns: size_t Number of entries (tokens) present in the vocabulary.

Implements Mila::Data::TokenizerVocabulary.

◆ getSpecialTokenId()

std::optional< TokenId > Mila::Data::BpeVocabulary::getSpecialTokenId ( const std::string & token_str ) const

inline

Look up a special token ID by its string representation.

Used by BpeTokenizer's encode pre-pass to resolve tokens such as "<|endoftext|>" or "<|begin_of_text|>" directly to IDs before BPE runs.

Parameters

token_str Token string to look up.

Returns: Token ID if registered as special, nullopt otherwise.

◆ getSpecialTokenList()

const std::vector< std::pair< std::string, TokenId > > & Mila::Data::BpeVocabulary::getSpecialTokenList ( ) const

inline

Return the special token list sorted longest-first.

Ordered longest-first so BpeTokenizer's linear scan matches longer tokens before any of their prefixes (e.g., "<|begin_of_text|>" before "<|").

Returns: Vector of (token_string, token_id) pairs.

◆ idToToken()

std::optional< std::string > Mila::Data::BpeVocabulary::idToToken ( TokenId id ) const

inlineoverridevirtual

Map a numeric id back to its token string.

Returns an empty optional if the id is out of range or not defined.

Parameters

id	Token id to convert.

Returns: std::optional<std::string> The token string if present, otherwise empty.

Implements Mila::Data::TokenizerVocabulary.

◆ load()

BpeVocabulary Mila::Data::BpeVocabulary::load ( const fs::path & path )

inlinestatic

Load a vocabulary from Mila binary format (version 2).

Reads a file written by save(). Special tokens are restored from the serialized (string, id) pairs and the special token list is rebuilt automatically.

Parameters

path	Input file path.

Returns: Loaded BpeVocabulary instance.

Exceptions

std::runtime_error on I/O errors or format mismatch.

◆ loadGemma()

BpeVocabulary Mila::Data::BpeVocabulary::loadGemma ( const fs::path & path )

static

Load a pretrained Gemma 4 (SentencePiece BPE) vocabulary.

Reads the Gemma-extended binary produced by convert_tokenizer.py:

Header: vocab_size (uint32), use_byte_fallback (uint8),
        model_type (uint8: 1=BPE, 2=Unigram), num_merges (uint32)
For each token: token_length (uint32), token_bytes, score (float32), token_id (uint32)
For each merge:  left_length (uint32), left, right_length (uint32), right
has_bos/eos/pad/unk (uint32) + id (uint32, conditional)

Configures the SentencePiece runtime: byte_level=false (pieces are raw UTF-8 with U+2581 for spaces) and PreTokenizationMode::SentencePiece. The instruct turn-boundary tokens are registered from the loaded vocab.

Parameters

path	Path to the converted Gemma tokenizer binary.

Exceptions

std::runtime_error on I/O errors or an unsupported (Unigram) model type.

◆ loadGpt2()

BpeVocabulary Mila::Data::BpeVocabulary::loadGpt2 ( const fs::path & tokenizer_path )

static

Load a pretrained GPT-2 vocabulary.

Reads the binary format produced by convert_gpt2_tokenizer.py:

vocab_size   (uint32)
num_merges   (uint32)
For each token: token_length (uint32), token_bytes (utf-8), token_id (uint32)
For each merge: left_length (uint32), left, right_length (uint32), right
has_eos (uint32), eos_id (uint32, conditional)
has_bos (uint32), bos_id (uint32, conditional)
has_pad (uint32), pad_id (uint32, conditional)

Parameters

tokenizer_path Path to the converted GPT-2 tokenizer binary.

Returns: Loaded BpeVocabulary instance.

Exceptions

std::runtime_error on I/O or format errors.

◆ loadLlama32()

BpeVocabulary Mila::Data::BpeVocabulary::loadLlama32 ( const fs::path & path )

static

Load a pretrained Llama 3.2 vocabulary.

Reads the binary format produced by convert_llama_tokenizer.py:

Header: vocab_size (uint32), use_byte_fallback (uint8)
For each token: token_length (uint32), token_bytes, score (float32), token_id (uint32)
has_bos (uint32), bos_id (uint32, conditional)  -- 128000
has_eos (uint32), eos_id (uint32, conditional)  -- 128001
has_pad (uint32), pad_id (uint32, conditional)
has_unk (uint32), unk_id (uint32, conditional)  -- absent for Llama 3.2

Llama 3.x vocabularies carry no explicit BPE merges; the merge order is encoded implicitly in the token ID assignment.

Parameters

path	Path to the converted Llama 3.2 tokenizer binary.

Returns: Loaded BpeVocabulary instance.

Exceptions

std::runtime_error on I/O or format errors.

◆ loadMistral()

BpeVocabulary Mila::Data::BpeVocabulary::loadMistral	(	const fs::path &	vocab_path,
		const fs::path &	merges_path )

static

Load a pretrained Mistral vocabulary.

Note: Not yet implemented for external Mistral formats. Provide a Mila binary produced by save() as a workaround.

Exceptions

std::runtime_error always.

◆ loadQwen()

BpeVocabulary Mila::Data::BpeVocabulary::loadQwen ( const fs::path & path )

static

Load a pretrained Qwen 3.x (byte-level BPE) vocabulary.

Reads the same binary layout loadGemma does, produced by Qwen/convert_tokenizer.py:

Header: vocab_size (uint32), use_byte_fallback (uint8),
        model_type (uint8: 1=BPE), num_merges (uint32)
For each token: token_length (uint32), token_bytes, score (float32), token_id (uint32)
For each merge:  left_length (uint32), left, right_length (uint32), right
has_bos/eos/pad/unk (uint32) + id (uint32, conditional)

Configures the GPT-2 byte-level runtime (byte_level=true, the GPT-2 byte encoder) with Qwen's own split pattern, and takes the merge-by-rank path rather than Llama's max-munch: Qwen's ignore_merges is false and its vocabulary is not max-munch equivalent. Chat, thinking and tool-calling markers are registered from the loaded vocabulary.

Parameters

path	Path to the converted Qwen tokenizer binary.

Exceptions

std::runtime_error on I/O errors or a non-BPE model type.

◆ save()

void Mila::Data::BpeVocabulary::save ( const fs::path & path ) const

inlineoverride

Serialize the vocabulary to Mila binary format (content version 2).

Writes a MilaFileHeader followed by the vocabulary content. Special tokens are stored as (string_length, string, token_id) triples, eliminating the char-key indirection used in the former GPT-2-only format.

Parameters

path	Output file path. Parent directory must exist.

Exceptions

std::runtime_error on I/O errors.

◆ tokenToId()

std::optional< TokenId > Mila::Data::BpeVocabulary::tokenToId ( const std::string & token ) const

inlineoverridevirtual

Convert a token string to its ID.

Falls back to the UNK token ID when the token is not found and use_unk is enabled (GPT-2 style). Llama 3.x vocabularies return nullopt on a miss because they rely on byte-level fallback rather than an UNK token.

Parameters

token UTF-8 encoded token string.

Returns: Token ID, UNK ID (if enabled), or nullopt on miss.

Implements Mila::Data::TokenizerVocabulary.

◆ train()

BpeVocabulary Mila::Data::BpeVocabulary::train	(	const std::string &	corpus,
		const BpeVocabularyConfig &	config )

inlinestatic

Train a BPE vocabulary from a text corpus.

Parameters

corpus	Training text.
config	Vocabulary configuration; config.validate() is called internally.

Returns: Trained BpeVocabulary instance.

Exceptions

std::invalid_argument if config fails validation.

◆ trainFromFile()

BpeVocabulary Mila::Data::BpeVocabulary::trainFromFile	(	const fs::path &	corpus_path,
		const BpeVocabularyConfig &	config )

inlinestatic

Train a BPE vocabulary from a corpus file.

Parameters

corpus_path	Path to a UTF-8 text corpus file.
config	Vocabulary configuration.

Returns: Trained BpeVocabulary instance.

Exceptions

std::runtime_error	if the file cannot be opened.
std::invalid_argument	if config fails validation.

The documentation for this class was generated from the following file:

Mila/Src/Data/Tokenizers/Bpe/BpeVocabulary.ixx

Public Member Functions

Static Public Member Functions

Detailed Description

Member Function Documentation

◆ containsToken()

◆ getSize()

◆ getSpecialTokenId()

◆ getSpecialTokenList()

◆ idToToken()

◆ load()

◆ loadGemma()

◆ loadGpt2()

◆ loadLlama32()

◆ loadMistral()

◆ loadQwen()

◆ save()

◆ tokenToId()

◆ train()

◆ trainFromFile()