Mila
Deep Neural Network Library
Loading...
Searching...
No Matches
Mila::Dnn::QwenPrecisionPlan Struct Referenceexport
module Dnn.Components.QwenPrecisionPlan

The section 5 allocation: 25.62 B resident weights at 2.90 bits average, 8.65 GiB. More...

Public Types

using DeltaNetGating = NoWeightQuant
using DeltaNetQueryKey = PerGroupCodebook3<64>
using DeltaNetValueGateOutput = PerGroupCodebook2<32>
using EmbeddingTable = NoWeightQuant
using FeedForwardDown = PerGroupCodebook3<64>
using FeedForwardGateUp = PerGroupCodebook2<32>
using LanguageModelHead = PerGroupFp4<128>
using OutputProjection = PerGroupFp4<128>
using QkvProjection = PerGroupFp4<128>

Detailed Description

The section 5 allocation: 25.62 B resident weights at 2.90 bits average, 8.65 GiB.

Role Params Policy Bits/w GiB
EmbeddingTable 1.271 B BF16, host-resident - 0.000
FeedForwardGateUp 11.408 B PerGroupCodebook2<32> 2.5 3.320
FeedForwardDown 5.704 B PerGroupCodebook3<64> 3.25 2.158
DeltaNetValueGateOutput 4.531 B PerGroupCodebook2<32> 2.5 1.319
DeltaNetQueryKey 1.007 B PerGroupCodebook3<64> 3.25 0.381
DeltaNetGating 0.024 B BF16 – never quantized 16 0.045
QkvProjection 1.678 B PerGroupFp4<128> 4.125 0.806
OutputProjection (above) PerGroupFp4<128> 4.125 (above)
LanguageModelHead 1.271 B PerGroupFp4<128> 4.125 0.610

Three assignments are not "biggest group gets fewest bits" and are load-bearing:

  • DeltaNetGating is NoWeightQuant. beta and decay drive the forget gate, where error compounds EXPONENTIALLY over the sequence rather than linearly. They are 0.1% of parameters; protecting them costs 0.045 GiB.
  • FeedForwardDown sits a half-step above the gate/up pair because down_proj reads post-SwiGLU activations, which carry the heavy outliers.
  • EmbeddingTable is NoWeightQuant because the table is host-resident: it is a gather, never a multiply, so it costs 10 KiB per token over PCIe and no VRAM. The policy states the STORAGE FORMAT, and BF16 is what the host holds; residency is the transformer's decision (item 6), not the policy's.

The full-attention roles collapse to one policy on purpose. Section 5 puts q (with its output-gate half), k, v and o all at FP4, so the fused qkv projection needs no split; the per-projection divergence in this model is between BLOCK KINDS and between the feed-forward pair, not inside the attention block.

The documentation for this struct was generated from the following file: