Data.SpecialTokens Module Reference

Classes
struct	Mila::Data::SpecialTokens
	Configuration for special tokens across all tokenizer types. More...

Enumerations
enum class	Mila::Data::SpecialToken : uint8_t {   PAD = 0 , UNK , BOS , EOS ,   MASK , SEP , CLS }
	Special token types for tokenization.

Files
file	Mila/Src/Data/Tokenizers/SpecialTokens.ixx
	Configuration for special tokens used across all tokenizer types.