Deployment configuration for Gpt language models. More...

Inheritance diagram for Mila::Dnn::GptModelConfig:

Public Member Functions
	GptModelConfig (dim_t context_length)
	Construct with a required context length.
std::string	toString () const
Public Member Functions inherited from Mila::Dnn::LanguageModelConfig< GptModelConfig >
std::string	baseToString () const
	Produce the base fields portion of a toString() summary.
dim_t	getContextLength () const noexcept
KvCacheCompression	getKvCacheCompression () const noexcept
WeightQuantization	getWeightQuantization () const noexcept
GptModelConfig &	withContextLength (dim_t context_length)
	Set the maximum sequence length.
GptModelConfig &	withFP4Quantization ()
	FP4 quantization – FP4 weights, FP8 KV cache.
GptModelConfig &	withFP8Quantization ()
	FP8 quantization – FP8 weights, FP8 KV cache.
GptModelConfig &	withFullPrecision ()
	Full precision – BF16 weights, BF16 KV cache.
GptModelConfig &	withKvCacheCompression (KvCacheCompression kv)
	Set the KV cache compression mode independently.
GptModelConfig &	withWeightQuantization (WeightQuantization wq)
	Set the weight quantization mode independently.

Additional Inherited Members
Protected Attributes inherited from Mila::Dnn::LanguageModelConfig< GptModelConfig >
dim_t	context_length_
KvCacheCompression	kv_cache_compression_
WeightQuantization	weight_quantization_

Detailed Description

Deployment configuration for Gpt language models.

Inherits all fluent setters and accessors from LanguageModelConfig<GptModelConfig>.

Constructor & Destructor Documentation

Mila::Dnn::GptModelConfig::GptModelConfig ( dim_t context_length )

inlineexplicit

Construct with a required context length.

Parameters

context_length Maximum sequence length in tokens. Must be > 0.

Exceptions

std::invalid_argument if context_length is zero.

The documentation for this struct was generated from the following file: