SpecialTokens.ixx File Reference

Configuration for special tokens used across all tokenizer types. More...

#include <string>
#include <string_view>
#include <vector>
#include <unordered_map>
#include <cstdint>

Classes
struct	Mila::Data::SpecialTokens
	Configuration for special tokens across all tokenizer types. More...

Namespaces
namespace	Mila
	Mila main API namespace.

Enumerations
enum class	Mila::Data::SpecialToken : uint8_t {   PAD = 0 , UNK , BOS , EOS ,   MASK , SEP , CLS }
	Special token types for tokenization.

Detailed Description

Configuration for special tokens used across all tokenizer types.