Data.TokenizerType Module Reference

Enumerations
enum class	TokenizerType : uint8_t {   Unknown = 0 , Char , Bpe , SentencePiece ,   Word , Unigram }
	Tokenizer type discriminator used across tokenizer and vocabulary types.

Functions
TokenizerType	Mila::Data::stringToTokenizerType (std::string_view s) noexcept
	Parse a string into TokenizerType.
std::string_view	Mila::Data::tokenizerTypeToString (TokenizerType t) noexcept
	Convert TokenizerType to a stable string representation.

Files
file	Mila/Src/Data/Tokenizers/TokenizerType.ixx