LanguageModelConfig.ixx File Reference

CRTP base configuration for all deployable Mila language models. More...

#include <stdexcept>
#include <string>
import Dnn.TensorTypes;

Classes
struct	Mila::Dnn::LanguageModelConfig< TDerived >
	CRTP base configuration for all deployable Mila language models. More...

Namespaces
namespace	Mila
	Mila main API namespace.

Enumerations
enum class	Mila::Dnn::KvCacheCompression { None , FP8 }
	KV cache storage and compression strategy for GroupedQueryAttention. More...
enum class	Mila::Dnn::WeightQuantization { None , FP8 , FP4 }
	Weight storage and matmul strategy for Linear components. More...

Functions
std::string	Mila::Dnn::weightQuantizationName (WeightQuantization quantization)
	The scheme name recorded in an artifact and in its manifest.

Detailed Description

CRTP base configuration for all deployable Mila language models.

LanguageModelConfig<TDerived> owns the deployment concerns that are universal across all language model architectures:

1. context_length – maximum sequence length the model is built for. RoPE embeddings and KV cache buffers are sized to this.
2. WeightQuantization – weight storage and matmul strategy for Linear components. Defaults to WeightQuantization::None (BF16 weights).
3. KvCacheCompression – KV cache storage and compression strategy for GroupedQueryAttention components. Defaults to KvCacheCompression::None (no compression).

CRTP Pattern

All fluent setters return TDerived& so that chains work correctly across both base and derived methods without casting at the call site:

QwenModelConfig config = QwenModelConfig( context_length )
    .withFP8Quantization()   // returns QwenModelConfig&
    .withThinkingMode();     // returns QwenModelConfig&

Relationship to ModelConfig

ModelConfig<TDevice, TPrecision> is the structural base for all Mila models. LanguageModelConfig is the deployment configuration counterpart for the language model branch of that hierarchy. Vision model configurations would derive from a sibling VisionModelConfig<TDerived>, not from this class.

Relationship to BuildContext

LanguageModelConfig is the public API surface for deployment configuration. BuildContext is the internal carrier through the component tree. fromPretrained() projects LanguageModelConfig into BuildContext once – they are never the same object.

Quantization Presets vs Fine-Grained Control

Convenience preset methods express common deployment decisions in user vocabulary. Fine-grained setters are available for atypical configurations:

// Preset -- FP8 weights + FP8 KV cache
LlamaModelConfig config = LlamaModelConfig( context_length )
    .withFP8Quantization();
 
// Fine-grained -- FP4 weights, no KV compression
LlamaModelConfig config = LlamaModelConfig( context_length )
    .withWeightQuantization( WeightQuantization::FP4 )
    .withKvCacheCompression( KvCacheCompression::None );

Enumeration Type Documentation

KvCacheCompression

enum class Mila::Dnn::KvCacheCompression

exportstrong

KV cache storage and compression strategy for GroupedQueryAttention.

Maps to the TKvPolicy template parameter on GroupedQueryAttention and CudaGqaOp via the fromPretrained() runtime->compile-time bridge. The mapping is:

None -> NoKvCompression (BF16 cache, no compression overhead) FP8 -> PerChannelKvFp8<> (FP8_E4M3 cache, per-head per-token float32 scales)

New compression algorithms (SlidingWindow, LowRank, TurboQuant) add a value here and a corresponding policy struct in KvCache.QuantPolicy – no other changes are required at this level.

Enumerator
None	No compression – default; BF16 KV cache.
FP8	FP8_E4M3 per-head per-token KV cache compression – Alpha.6 target.

WeightQuantization

enum class Mila::Dnn::WeightQuantization

exportstrong

Weight storage and matmul strategy for Linear components.

Maps to the TWeightQuant template parameter on Linear and CudaLinearOp via the fromPretrained() runtime->compile-time bridge. The mapping is:

None -> NoWeightQuant (BF16 weights, standard cuBLASLt plan) FP8 -> PerChannelFp8<> (FP8_E4M3 weights, per-channel float32 scales) FP4 -> PerGroupFp4<> (future)

This enum is Mila API vocabulary. Callers set it via fluent methods on the concrete model config – they do not interact with the policy structs directly.

Enumerator
None	BF16 weights – default; no quantization overhead.
FP8	FP8_E4M3 per-channel weight quantization – Alpha.5 target.
FP4	Per-group FP4 weight quantization – future target.

Function Documentation

weightQuantizationName()

std::string Mila::Dnn::weightQuantizationName ( WeightQuantization quantization )

inlineexport

The scheme name recorded in an artifact and in its manifest.

It lives beside the enum because it is written by the model that saves the artifact and read by the tool that packages it, and those two must agree exactly: the load side refuses an artifact whose scheme disagrees with the build's compile-time policy, since the bytes are packed differently per scheme and reinterpreting them produces a model that runs and is wrong. It was previously spelled out in both places.