QuantizationDispatch.ixx File Reference

The one place a runtime quantization setting becomes a compile-time policy. More...

#include <format>
#include <stdexcept>
#include <string_view>
import Dnn.Quantization.KvCache.Policy;
import Dnn.Quantization.Weight.Policies;
import Dnn.TensorDataType;
import Dnn.LanguageModelConfig;

Namespaces
namespace	Mila
	Mila main API namespace.

Functions
template<TensorDataType TPrecision, KvCachePolicy TKvCachePolicy, typename TResult, typename TAction>
TResult	Mila::Dnn::dispatchWeightQuantization (WeightQuantization weight_quantization, KvCacheCompression kv_cache_compression, std::string_view caller, TAction &&action)
	Resolve a runtime quantization setting to a policy type and invoke an action.

Detailed Description

The one place a runtime quantization setting becomes a compile-time policy.

Every model entry point that has to reach a template instantiation from a ModelConfig routes through here, so a newly supported mode is added once rather than per model per entry point. See Specifications/MemoryFootprint.md.

Function Documentation

dispatchWeightQuantization()

template<TensorDataType TPrecision, KvCachePolicy TKvCachePolicy, typename TResult, typename TAction>

TResult Mila::Dnn::dispatchWeightQuantization ( WeightQuantization weight_quantization, KvCacheCompression kv_cache_compression, std::string_view caller, TAction && action )

export

Resolve a runtime quantization setting to a policy type and invoke an action.

This bridge existed in four copies – load and footprint, for each of Gemma and Llama – which differed only in the KV policy and the name in their error messages. Four copies of a runtime-to-compile-time mapping is four chances for a newly supported mode to reach the load path and not the footprint path, which would make a model report a figure it does not allocate. That is the exact defect class the footprint work exists to prevent, so the mapping lives once.

Template Parameters

TPrecision	Compute precision; quantized weights require BF16.
TKvCachePolicy	KV policy the caller's chassis uses – Gemma's sliding-window ring, or NoKvCompression. Not derived from the config: it is an architecture property, not a deployment choice.
TResult	What the action returns – a model, or a MemoryStats.

Parameters

weight_quantization	Runtime weight-quantization setting to resolve to a policy type.
kv_cache_compression	Runtime KV-cache setting accompanying it.
caller	Prefix for error messages, e.g. "GemmaModel::fromPretrained".
action	Invoked as action.template operator()<TWeightQuantization, TKvCachePolicy>().

Exceptions

std::runtime_error

if the requested combination is unsupported.