TokenizerTrainer.ixx File Reference

Abstract trainer interface for building tokenizers' vocabularies. More...

#include <string>
#include <memory>
#include <filesystem>
#include <istream>
#include <fstream>
#include <sstream>
import Data.TokenizerVocabulary;

Classes
class	Mila::Data::TokenizerTrainer
	Abstract interface for training tokenizer vocabularies from text corpora. More...

Namespaces
namespace	Mila
	Mila main API namespace.

Detailed Description

Abstract trainer interface for building tokenizers' vocabularies.

Defines the minimal lifecycle for constructing a tokenizer vocabulary from a text corpus. Tokenizer Trainers perform training and return the resulting vocabulary object; ownership is transferred to the caller.