Mila::Data::BpeVocabularyConfig Class Reference

Configuration for the BPE vocabulary. More...

Public Member Functions
void	fromMetadata (const SerializationMetadata &meta)
	Restore configuration from metadata.
size_t	getMaxMerges () const
size_t	getMinFrequency () const
PreTokenizationMode	getPreTokenizationMode () const
const std::string &	getPreTokenizationPattern () const
const SpecialTokens &	getSpecialTokens () const
size_t	getVocabSize () const
bool	isByteLevel () const
bool	isMergeCachingEnabled () const
SerializationMetadata	toMetadata () const
	Serialize configuration to metadata.
std::string	toString () const
void	validate () const
	Validate configuration for training.
BpeVocabularyConfig &	withByteLevel (bool byte_level)
BpeVocabularyConfig &	withMaxMerges (size_t max_merges)
BpeVocabularyConfig &	withMergeCaching (bool enable)
BpeVocabularyConfig &	withMinFrequency (size_t frequency)
BpeVocabularyConfig &	withPreTokenization (PreTokenizationMode mode)
BpeVocabularyConfig &	withPreTokenizationPattern (const std::string &pattern)
BpeVocabularyConfig &	withSpecialTokens (const SpecialTokens &tokens)
BpeVocabularyConfig &	withVocabSize (size_t size)

Detailed Description

Configuration for the BPE vocabulary.

Describes both training hyperparameters and the runtime properties (byte-level encoding, pre-tokenization pattern, special token set) that apply to all BPE families. Serialized with vocabulary files to provide full provenance and to enable validation of vocabulary compatibility.

Typical usage for pretrained models (no training validation needed):

auto config = BpeVocabularyConfig()
    .withVocabSize( 128256 )
    .withByteLevel( true )
    .withPreTokenization( PreTokenizationMode::Llama3Regex )
    .withPreTokenizationPattern( LLAMA3_PRETOKENIZATION_PATTERN )
    .withSpecialTokens( SpecialTokens::llamaStyle() );

Member Function Documentation

fromMetadata()

void Mila::Data::BpeVocabularyConfig::fromMetadata ( const SerializationMetadata & meta )

inline

Restore configuration from metadata.

All fields use tryGet* so that files produced by older builds without a given field fall back silently to the in-class defaults.

toMetadata()

SerializationMetadata Mila::Data::BpeVocabularyConfig::toMetadata ( ) const

inline

Serialize configuration to metadata.

Persists all fields including token strings so that round-tripped vocabularies reproduce the correct special token set on load.

validate()

void Mila::Data::BpeVocabularyConfig::validate ( ) const

inline

Validate configuration for training.

Called by BpeTrainer before training begins. Must not be called for pretrained vocabularies loaded via factory methods.

Exceptions

std::invalid_argument

on invalid training configuration.

---

The documentation for this class was generated from the following file:

• Mila/Src/Data/Tokenizers/Bpe/BpeVocabularyConfig.ixx