BpeTokenizer.ixx File Reference

Unified BPE tokenizer for GPT-2, Llama 3.x, Mistral, and Gemma model families. More...

#include <string>
#include <string_view>
#include <vector>
#include <span>
#include <memory>
#include <optional>
#include <filesystem>
#include <mutex>
#include <regex>
#include <limits>
#include <stdexcept>
import Logging.Logger;
import Data.TokenizerVocabulary;
import Data.Tokenizer;
import Data.BpePreTokenizationMode;
import Data.BpeVocabulary;

Classes
class	Mila::Data::BpeTokenizer
	Unified BPE tokenizer targeting GPT-2, Llama 3.x, and Mistral model families. More...

Namespaces
namespace	Mila
	Mila main API namespace.

Typedefs
using	Mila::Data::TokenId

Detailed Description

Unified BPE tokenizer for GPT-2, Llama 3.x, Mistral, and Gemma model families.

Encode pipeline:

1. Special token pre-pass: split input on registered special token strings (longest-first scan) and emit their IDs directly, bypassing BPE entirely. GPT-2 vocabularies with no registered special tokens skip this via fast path.
2. Pre-tokenize each plain text segment with the configured regex pattern.
3. Byte-encode each pre-token using the GPT-2 style byte encoder. 4a. BPE path (GPT-2 / trained vocabularies): apply explicit merge rules greedily, lowest priority index first. 4b. Max-munch path (Llama 3.x / TikToken): find the longest vocabulary match at each position in the encoded unit sequence. Used when no merge rules are present; the merge order is implicit in the token ID assignment. 4c. SentencePiece path (Gemma): Metaspace pre-tokenization (space -> U+2581), UTF-8 character initial units with <0xNN> byte fallback, then the merge rules by rank. Selected when the vocabulary uses PreTokenizationMode::SentencePiece.
4. Map final tokens to IDs; fall back to 0 on a miss.

Decode pipeline: Concatenate token strings and reverse the byte encoding back to UTF-8.