Mila::Data::CharVocabularyConfig Class Reference

Configuration for Character-level tokenizer training. More...

Public Member Functions
void	fromMetadata (const SerializationMetadata &meta)
	Populate configuration from metadata.
const SpecialTokens &	getSpecialTokens () const
bool	isByteLevel () const
bool	isCaseSensitive () const
bool	shouldNormalizeUnicode () const
SerializationMetadata	toMetadata () const
	Convert configuration to metadata for serialization.
std::string	toString () const
	Produce human-readable summary of configuration.
void	validate () const
	Validate configuration parameters.
CharVocabularyConfig &	withByteLevel (bool byte_level)
	Set whether to use byte-level encoding.
CharVocabularyConfig &	withCaseSensitive (bool sensitive)
	Set whether tokenization is case-sensitive.
CharVocabularyConfig &	withNormalizeUnicode (bool normalize)
	Set whether to normalize Unicode characters.
CharVocabularyConfig &	withSpecialTokens (const SpecialTokens &tokens)
	Configure special tokens.

Detailed Description

Configuration for Character-level tokenizer training.

Character tokenizers split text into individual characters (or bytes). This is the simplest tokenization approach with minimal configuration needs.

Fluent interface allows chaining:

auto config = CharTokenizerConfig()
    .withSpecialTokens(my_tokens)
    .withCaseSensitive(false)
    .withNormalizeUnicode(true);

Member Function Documentation

fromMetadata()

void Mila::Data::CharVocabularyConfig::fromMetadata ( const SerializationMetadata & meta )

inline

Populate configuration from metadata.

Missing keys are ignored leaving defaults intact.

Parameters

meta	Metadata to read configuration from.

toMetadata()

SerializationMetadata Mila::Data::CharVocabularyConfig::toMetadata ( ) const

inline

Convert configuration to metadata for serialization.

Returns

SerializationMetadata containing all configuration parameters.

toString()

std::string Mila::Data::CharVocabularyConfig::toString ( ) const

inline

Produce human-readable summary of configuration.

Suitable for logging and debugging.

Returns

std::string Configuration summary.

validate()

void Mila::Data::CharVocabularyConfig::validate ( ) const

inline

Validate configuration parameters.

Exceptions

std::invalid_argument

if configuration is invalid.

withByteLevel()

CharVocabularyConfig & Mila::Data::CharVocabularyConfig::withByteLevel ( bool byte_level )

inline

Set whether to use byte-level encoding.

When true, operates on raw UTF-8 bytes instead of Unicode characters. This guarantees any text can be represented but increases sequence length.

Parameters

byte_level

True for byte-level, false for character-level (default).

Returns

Reference to this config for method chaining.

withCaseSensitive()

CharVocabularyConfig & Mila::Data::CharVocabularyConfig::withCaseSensitive ( bool sensitive )

inline

Set whether tokenization is case-sensitive.

When false, text is converted to lowercase before tokenization. This reduces vocabulary size but loses case information.

Parameters

sensitive

True for case-sensitive (default), false for case-insensitive.

Returns

Reference to this config for method chaining.

withNormalizeUnicode()

CharVocabularyConfig & Mila::Data::CharVocabularyConfig::withNormalizeUnicode ( bool normalize )

inline

Set whether to normalize Unicode characters.

When true, applies Unicode normalization (e.g., NFC) to ensure consistent representation of characters with multiple encodings.

Parameters

normalize

True to normalize Unicode, false otherwise (default).

Returns

Reference to this config for method chaining.

withSpecialTokens()

CharVocabularyConfig & Mila::Data::CharVocabularyConfig::withSpecialTokens ( const SpecialTokens & tokens )

inline

Configure special tokens.

Parameters

tokens

SpecialTokens configuration.

Returns

Reference to this config for method chaining.

---

The documentation for this class was generated from the following file:

• Mila/Src/Data/Tokenizers/Char/CharVocabularyConfig.ixx