Qwen.ixx File Reference

Qwen 3.8 decoder-only transformer network (inference: prefill + decode). More...

#include <string>
#include <vector>
#include <memory>
#include <sstream>
#include <stdexcept>
#include <cstdint>
#include <format>
#include <algorithm>
#include <cmath>
#include <type_traits>
import Serialization.WeightsReader;
import Serialization.Mode;
import Compute.ExecutionContextFactory;
import Dnn.TensorDataType;
import Compute.ExecutionContext;
import Dnn.Components.QwenAttentionBlock;
import Dnn.Quantization.Weight.PrecisionPlan;
import Dnn.TensorOps;
import Dnn.ITensor;
import Dnn.Components.QwenDeltaNetBlock;
import Dnn.Tensor;
import Dnn.Components.ITransformerBlock;
import Dnn.RuntimeMode;
import Compute.Device;
import Dnn.TensorTypes;
import Serialization.Tensor;
import Dnn.Components.QwenPrecisionPlan;
import Dnn.Components.QwenConfig;
import Serialization.Metadata;
import Compute.DeviceAllocation;
import Serialization.ModelArchive;
import Logging.Logger;
import Dnn.TensorDataTypeTraits;
import Dnn.Components.TokenEmbedding;
import Dnn.ComponentType;
import Dnn.ModelType;
import Dnn.Components.Linear;
import Dnn.Quantization.KvCache.Policy;
import Compute.DeviceType;
import Dnn.LanguageModelNetwork;
import Dnn.Component;
import Dnn.Components.RmsNorm;
import Compute.CpuMemoryResource;
import Dnn.Quantization.Weight.Policies;
import Compute.DeviceId;
import Compute.GqaWorkspace;
import Compute.DeviceTypeTraits;
import Compute.GqaState;

Classes
class	Mila::Dnn::QwenTransformer< TDeviceType, TPrecision, TWeightPlan, TKvCachePolicy >
	Qwen 3.8 transformer (decoder-only) for autoregressive inference. More...

Namespaces
namespace	Mila
	Mila main API namespace.

Variables
constexpr dim_t	Mila::Dnn::kQwenPrefillChunkFloor = 64
constexpr dim_t	Mila::Dnn::kQwenPrefillChunkRungs [] = { 1024, 512, 256, 128, 64 }

Detailed Description

Qwen 3.8 decoder-only transformer network (inference: prefill + decode).

The heterogeneous layer list is copied from GemmaTransformer, which is the one structure in the tree that already solves this problem: build two block types, store both as ITransformerBlock*, and drive them polymorphically (one virtual call per layer per token step, negligible against the per-layer GEMMs). What is NOT copied is Gemma's bool kGlobal flag: that is right for one mixer at two geometries, and Qwen's two kinds are different mixers (Specifications/Qwen3.8.md section 8, Phase 2 revision).

Two block kinds, two memory models

The published 27B geometry (full_attention_interval: 4) builds three QwenDeltaNetBlocks per QwenAttentionBlock. The two kinds differ in more than their mixer:

• The attention layers hold a KV cache that GROWS with context, and share the pooled workspace and the GQA transient this network owns.
• The DeltaNet layers hold a fixed-size recurrent state plus a short convolution window, neither of which depends on context length. Their activation slots share no width with the attention workspace's, so they pool through a SECOND workspace struct rather than a wider one. Both are owned here and both are counted once, by this network.

One consequence reaches the product rather than the code: rewindKvCache asks every layer, and a DeltaNet layer always refuses, because a recurrent state is a lossy summary that cannot be rolled back to an earlier position. PROMPT-PREFIX REUSE IS THEREFORE UNAVAILABLE for any configuration containing these layers.

What this network cannot do yet

Prefill runs the DeltaNet recurrence sequentially over the chunk (there is no chunked UT-transform kernel yet), so long-prompt prefill on those layers is O(T) in steps.

Untied tables

Qwen 3.8 sets tie_word_embeddings: false – two full-size tables, 1.271 B parameters each. So unlike Gemma there is no weight-tying path here, and the two tables take DIFFERENT policies from the plan: the embedding stays BF16 (section 5 moves it to host memory, where it is a gather and never a multiply) while the head is FP4 because it drives sampling and is re-read in full every decode step.

The embedding table is HOST-RESIDENT on CUDA (section 8 item 6): 2.37 GiB of BF16 in pinned memory, gathered across PCIe at 10 KiB per token, costing the card nothing.