Data.BpePreTokenizationMode Module Reference

Enumerations
enum class	Mila::Data::PreTokenizationMode {   None , Whitespace , Gpt2Regex , Llama3Regex ,   SentencePiece }
	Pre-tokenization strategies for GPT-4 style BPE tokenizers.

Variables
constexpr const char *	Mila::Data::GPT2_PRETOKENIZATION_PATTERN
constexpr const char *	Mila::Data::GPT2_PRETOKENIZATION_PATTERN_ASCII_FALLBACK
constexpr const char *	Mila::Data::LLAMA3_PRETOKENIZATION_PATTERN
constexpr const char *	Mila::Data::LLAMA3_PRETOKENIZATION_PATTERN_ASCII_FALLBACK

Files
file	Mila/Src/Data/Tokenizers/Bpe/BpePreTokenizationMode.ixx
	Pre-tokenization mode enumeration for GPT-4 style BPE tokenizers.