Bpe Directory Reference

Files
	BpePreTokenizationMode.ixx
	Pre-tokenization mode enumeration for GPT-4 style BPE tokenizers.
	BpeTokenizer.ixx
	Unified BPE tokenizer for GPT-2, Llama 3.x, Mistral, and Gemma model families.
	BpeTrainer.ixx
	BPE vocabulary trainer with incremental corpus accumulation.
	BpeVocabulary.ixx
	BPE vocabulary for GPT-2, Llama 3.x, and Mistral model families.
	BpeVocabularyConfig.ixx
	Unified configuration for BPE vocabulary construction and runtime properties.