BpeVocabularyConfig.ixx File Reference

Unified configuration for BPE vocabulary construction and runtime properties. More...

#include <cstddef>
#include <cstdint>
#include <string>
#include <stdexcept>
#include <sstream>
import Serialization.Metadata;
import Data.SpecialTokens;
import Data.BpePreTokenizationMode;

Classes
class	Mila::Data::BpeVocabularyConfig
	Configuration for the BPE vocabulary. More...

Namespaces
namespace	Mila
	Mila main API namespace.

Detailed Description

Unified configuration for BPE vocabulary construction and runtime properties.

Covers training hyperparameters (used only by BpeTrainer) and vocabulary properties shared across the GPT-2, Llama 3.x, and Mistral BPE families. Training fields (min_frequency, max_merges, enable_merge_caching) are ignored when loading pretrained vocabularies; validate() enforces them only when called by BpeTrainer.