- Generated by
1.17.0
|
Mila
Deep Neural Network Library
|
CRTP base configuration for all deployable Mila language models. More...
Public Member Functions | |
| LanguageModelConfig (dim_t context_length) | |
| Construct with a required context length. | |
| std::string | baseToString () const |
| Produce the base fields portion of a toString() summary. | |
| dim_t | getContextLength () const noexcept |
| KvCacheCompression | getKvCacheCompression () const noexcept |
| dim_t | getLanguageModelHeadPositions () const noexcept |
| WeightQuantization | getWeightQuantization () const noexcept |
| TDerived & | withContextLength (dim_t context_length) |
| Set the maximum sequence length. | |
| TDerived & | withFP4Quantization () |
| FP4 quantization – FP4 weights, FP8 KV cache. | |
| TDerived & | withFP8Quantization () |
| FP8 quantization – FP8 weights, FP8 KV cache. | |
| TDerived & | withFullPrecision () |
| Full precision – BF16 weights, BF16 KV cache. | |
| TDerived & | withKvCacheCompression (KvCacheCompression kv) |
| Set the KV cache compression mode independently. | |
| TDerived & | withLanguageModelHeadPositions (dim_t positions) |
| Positions the language-model head evaluates per pass. | |
| TDerived & | withPrecisionPlan () |
| The family's designed per-role allocation, from a pre-quantized artifact. | |
| TDerived & | withWeightQuantization (WeightQuantization wq) |
| Set the weight quantization mode independently. | |
Protected Attributes | |
| dim_t | context_length_ { 0 } |
| KvCacheCompression | kv_cache_compression_ { KvCacheCompression::None } |
| dim_t | language_model_head_positions_ { 1 } |
| WeightQuantization | weight_quantization_ { WeightQuantization::None } |
CRTP base configuration for all deployable Mila language models.
| TDerived | Concrete config type (e.g. LlamaModelConfig). All fluent setters return TDerived& to support unbroken chain syntax across base and derived methods. |
|
inlineexplicit |
Construct with a required context length.
| context_length | Maximum sequence length in tokens. Must be > 0. |
| std::invalid_argument | if context_length is zero. |
|
inline |
Produce the base fields portion of a toString() summary.
Concrete model configs call this from their own toString() implementation and append architecture-specific fields.
|
inline |
Set the maximum sequence length.
Required before passing the config to load(). RoPE embeddings and KV cache buffers are sized to this value at build time.
| context_length | Maximum sequence length in tokens. Must be > 0. |
| std::invalid_argument | if context_length is zero. |
|
inline |
FP4 quantization – FP4 weights, FP8 KV cache.
Maps to PerGroupFp4<> on Linear (future) and PerChannelKvFp8<> on GroupedQueryAttention. Aggressive compression; some quality loss relative to FP8. FP4 KV cache is not a Mila target.
|
inline |
FP8 quantization – FP8 weights, FP8 KV cache.
Maps to PerChannelFp8<> on Linear and PerChannelKvFp8<> on GroupedQueryAttention. Good quality/compression tradeoff for standard inference on Ada Lovelace and later.
|
inline |
Full precision – BF16 weights, BF16 KV cache.
Resets both quantization axes to their defaults. Useful for explicitly documenting intent or overriding a previously set preset.
|
inline |
Set the KV cache compression mode independently.
Use when the desired KV cache compression does not pair with the default weight quantization of a preset, or when a preset does not exist for the desired combination.
| kv | KV cache compression mode to apply. |
|
inline |
Positions the language-model head evaluates per pass.
Default 1.
Generation reads a logit only at the final position, which is what the default pays for. Teacher-forced scoring needs one at every position, and a whole prefill chunk of logit rows does not fit – at a 248,320 vocabulary a BF16 row is 0.474 MiB – so a scoring deployment raises this to the number of rows it can afford and the head is evaluated in windows of that width.
Sizes buffers at build time exactly as withContextLength does, and like it, describes the deployment rather than the checkpoint. Families that have not implemented scoring ignore it.
| positions | Head width in positions. Must be > 0. |
| std::invalid_argument | if positions is zero. |
|
inline |
The family's designed per-role allocation, from a pre-quantized artifact.
Unlike the two above, this sets no KV compression: a plan allocates WEIGHT bits, and Qwen 3.8's baseline pairs its 2.90-bit body with a BF16 KV cache, which fits at 16K without compression (Qwen3.8.md section 5). A deployment that wants FP8 KV on top asks for it separately.
There is no quantize-on-load path here and there cannot be one: a codebook is fitted offline against calibration data, so the artifact must already carry the codes. A load refuses an artifact whose scheme is not the compiled one.
|
inline |
Set the weight quantization mode independently.
Use when the desired weight quantization does not pair with the default KV cache compression of a preset, or when a preset does not exist for the desired combination.
| wq | Weight quantization mode to apply. |
1.17.0