TokenizerVocabulary.ixx File Reference

Abstract interface for tokenizer vocabularies used by data pipelines. More...

#include <string>
#include <vector>
#include <unordered_map>
#include <optional>
#include <filesystem>
import Data.Tokenizer;

Classes
class	Mila::Data::TokenizerVocabulary
	Generic tokenizer vocabulary interface. More...

Namespaces
namespace	Mila
	Mila main API namespace.

Detailed Description

Abstract interface for tokenizer vocabularies used by data pipelines.

Defines the minimal API for mapping between token strings and numeric ids, and for persisting/loading vocabulary contents.