Mila::Data::CharVocabulary Class Reference

Character vocabulary for tokenization. More...

Inheritance diagram for Mila::Data::CharVocabulary:

Public Member Functions
TokenId	charToIndex (char c) const
const CharVocabularyConfig &	getConfig () const
	Get the configuration used to create this vocabulary.
size_t	getSize () const override
	Get the number of tokens in the vocabulary.
bool	hasSpecialTokens () const
std::optional< std::string >	idToToken (TokenId id) const override
	Map a numeric id back to its token string.
char	indexToChar (TokenId idx) const
TokenId	padTokenId () const
void	save (const fs::path &path) const override
	Serialize vocabulary to disk with configuration.
std::optional< TokenId >	tokenToId (const std::string &token) const override
	Map a token string to its numeric id.
TokenId	unkTokenId () const
Public Member Functions inherited from Mila::Data::TokenizerVocabulary
virtual	~TokenizerVocabulary ()=default
	Virtual destructor.
virtual void	save (const std::filesystem::path &path) const =0
	Serialize the vocabulary to disk at the given path.

Static Public Member Functions
static CharVocabulary	load (const fs::path &path)
	Load vocabulary from Mila binary format.
static CharVocabulary	train (const std::string &corpus, const CharVocabularyConfig &config)
	Build a character vocabulary from text corpus.
static CharVocabulary	trainFromFile (const fs::path &corpus_path, const CharVocabularyConfig &config)
	Build a character vocabulary from corpus file.

Detailed Description

Character vocabulary for tokenization.

Immutable vocabulary created via static factory methods. Stores configuration for full provenance tracking and serialization.

Thread safety: Immutable after construction, safe for concurrent reads.

Member Function Documentation

getConfig()

const CharVocabularyConfig & Mila::Data::CharVocabulary::getConfig ( ) const

inline

Get the configuration used to create this vocabulary.

Returns

const CharVocabularyConfig& Configuration reference.

getSize()

size_t Mila::Data::CharVocabulary::getSize ( ) const

inlineoverridevirtual

Get the number of tokens in the vocabulary.

Returns

size_t Number of entries (tokens) present in the vocabulary.

Implements Mila::Data::TokenizerVocabulary.

idToToken()

std::optional< std::string > Mila::Data::CharVocabulary::idToToken ( TokenId id ) const

inlineoverridevirtual

Map a numeric id back to its token string.

Returns an empty optional if the id is out of range or not defined.

Parameters

id	Token id to convert.

Returns

std::optional<std::string> The token string if present, otherwise empty.

Implements Mila::Data::TokenizerVocabulary.

load()

CharVocabulary Mila::Data::CharVocabulary::load ( const fs::path & path )

inlinestatic

Load vocabulary from Mila binary format.

Reads vocabulary and configuration from file written by save().

Parameters

path	Input file path.

Returns

Loaded CharVocabulary instance.

Exceptions

std::runtime_error

on I/O errors or format incompatibility.

save()

void Mila::Data::CharVocabulary::save ( const fs::path & path ) const

inlineoverride

Serialize vocabulary to disk with configuration.

File format includes MilaFileHeader with config metadata followed by binary vocabulary content.

Parameters

path	Output file path. Parent directory must exist.

Exceptions

std::runtime_error

on I/O errors.

tokenToId()

std::optional< TokenId > Mila::Data::CharVocabulary::tokenToId ( const std::string & token ) const

inlineoverridevirtual

Map a token string to its numeric id.

The lookup returns an empty optional when the token is not present in the vocabulary. Implementations may provide an explicit unknown-token id; callers should interpret an empty optional as "no mapping".

Parameters

token

UTF-8 encoded token string to look up.

Returns

std::optional<TokenId> The token id if present, otherwise empty.

Implements Mila::Data::TokenizerVocabulary.

train()

CharVocabulary Mila::Data::CharVocabulary::train ( const std::string & corpus, const CharVocabularyConfig & config )

inlinestatic

Build a character vocabulary from text corpus.

Parameters

corpus	Training text corpus.
config	Vocabulary configuration.

Returns

Trained CharVocabulary instance.

Exceptions

std::invalid_argument

if config is invalid.

trainFromFile()

CharVocabulary Mila::Data::CharVocabulary::trainFromFile ( const fs::path & corpus_path, const CharVocabularyConfig & config )

inlinestatic

Build a character vocabulary from corpus file.

Parameters

corpus_path	Path to training corpus text file.
config	Vocabulary configuration.

Returns

Trained CharVocabulary instance.

Exceptions

std::runtime_error	if file cannot be opened.
std::invalid_argument	if config is invalid.

---

The documentation for this class was generated from the following file:

• Mila/Src/Data/Tokenizers/Char/CharVocabulary.ixx