BpePreTokenizationMode.ixx File Reference

Pre-tokenization mode enumeration for GPT-4 style BPE tokenizers. More...

Namespaces
namespace	Mila
	Mila main API namespace.

Enumerations
enum class	Mila::Data::PreTokenizationMode {   None , Whitespace , Gpt2Regex , Llama3Regex ,   SentencePiece }
	Pre-tokenization strategies for GPT-4 style BPE tokenizers.

Variables
constexpr const char *	Mila::Data::GPT2_PRETOKENIZATION_PATTERN
constexpr const char *	Mila::Data::GPT2_PRETOKENIZATION_PATTERN_ASCII_FALLBACK
constexpr const char *	Mila::Data::LLAMA3_PRETOKENIZATION_PATTERN
constexpr const char *	Mila::Data::LLAMA3_PRETOKENIZATION_PATTERN_ASCII_FALLBACK

Detailed Description

Pre-tokenization mode enumeration for GPT-4 style BPE tokenizers.

Extends the GPT-2 BPE pre-tokenization modes with the GPT-4 / Llama 3.x regex pattern. This module is intentionally separate from the BpeTokenizer variant to avoid any risk of breaking validated GPT-2 tokenization.

Variable Documentation

GPT2_PRETOKENIZATION_PATTERN

const char* Mila::Data::GPT2_PRETOKENIZATION_PATTERN

constexprexport

Initial value:

                                              =
R"('s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+)"

GPT2_PRETOKENIZATION_PATTERN_ASCII_FALLBACK

const char* Mila::Data::GPT2_PRETOKENIZATION_PATTERN_ASCII_FALLBACK

constexprexport

Initial value:

                                                             =
R"('s|'t|'re|'ve|'m|'ll|'d| ?[A-Za-z]+| ?[0-9]+| ?[^\sA-Za-z0-9]+|\s+(?!\S)|\s+)"

LLAMA3_PRETOKENIZATION_PATTERN

const char* Mila::Data::LLAMA3_PRETOKENIZATION_PATTERN

constexprexport

Initial value:

                                                =
R"((?i:'[sdmt]|'ll|'ve|'re)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+)"

LLAMA3_PRETOKENIZATION_PATTERN_ASCII_FALLBACK

const char* Mila::Data::LLAMA3_PRETOKENIZATION_PATTERN_ASCII_FALLBACK

constexprexport

Initial value:

                                                               =
R"((?:'[sdmt]|'ll|'ve|'re)|[^\r\nA-Za-z0-9]?[A-Za-z]+|[0-9]{1,3}| ?[^\sA-Za-z0-9]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+)"