Qwen.DeltaNetBlock.ixx File Reference

Qwen 3.8 Gated DeltaNet decoder block (inference: prefill + decode). More...

#include <memory>
#include <vector>
#include <string>
#include <format>
#include <sstream>
#include <stdexcept>
#include <optional>
#include <type_traits>
import Dnn.Components.QwenPrecisionPlan;
import Dnn.Quantization.Weight.PrecisionPlan;
import Dnn.Quantization.Weight.Policies;
import Serialization.ModelArchive;
import Dnn.Components.CausalConv1d;
import Compute.Device;
import Serialization.Mode;
import Dnn.CompositeComponent;
import Dnn.ComponentType;
import Dnn.Component;
import Dnn.TensorDataTypeTraits;
import Compute.DeviceTypeTraits;
import Dnn.Components.Swiglu;
import Dnn.ActivationType;
import Dnn.TensorTypes;
import Dnn.ITensor;
import Dnn.Components.ITransformerBlock;
import Compute.GqaState;
import Compute.DeviceAllocation;
import Dnn.Components.GatedDeltaRule;
import Compute.DeviceId;
import Dnn.Components.AttentionOutputGate;
import Compute.DeviceType;
import Compute.ExecutionContext;
import Dnn.Tensor;
import Compute.Observation;
import Compute.IExecutionContext;
import Dnn.Components.Linear;
import Compute.ExecutionContextFactory;
import Dnn.Components.Activation;
import Dnn.TensorOps;
import Compute.CpuMemoryResource;
import Dnn.Components.QwenConfig;
import Dnn.Components.RmsNorm;
import Dnn.TensorDataType;
import Dnn.Components.Residual;

Classes
struct	Mila::Dnn::QwenDeltaNetBlock< TDeviceType, TPrecision, TWeightPlan >::BlockContexts
class	Mila::Dnn::QwenDeltaNetBlock< TDeviceType, TPrecision, TWeightPlan >
	One Gated DeltaNet decoder block. More...
struct	Mila::Dnn::QwenDeltaNetBlockWorkspace< TDeviceType, TPrecision >
	Transformer-owned shared activation workspace for QwenDeltaNetBlock (pooling). More...
struct	Mila::Dnn::QwenDeltaNetBlock< TDeviceType, TPrecision, TWeightPlan >::StateSnapshot
	Everything this block carries between chunks – all of it, in one value. More...

Namespaces
namespace	Mila
	Mila main API namespace.

Functions
template<DeviceType TDeviceType, TensorDataType TPrecision> requires PrecisionSupportedOnDevice<TPrecision, TDeviceType>
QwenDeltaNetBlockWorkspace< TDeviceType, TPrecision >	Mila::Dnn::makeQwenDeltaNetBlockWorkspace (const QwenConfig &config, DeviceId device, dim_t B, dim_t prefill_chunk, const std::string &name_prefix)
	Allocate the shared DeltaNet-block workspace.
std::vector< dim_t >	Mila::Dnn::qwenDeltaNetWorkspaceSlotWidths (const QwenConfig &config)
	The width of each workspace slot, one entry per [B, chunk, width] allocation.

Detailed Description

Qwen 3.8 Gated DeltaNet decoder block (inference: prefill + decode).

The other of the two block kinds in the 4-layer interleave – three of these per QwenAttentionBlock. A separate class rather than a flag, because the two kinds differ in the MIXER itself and share no projection shape (Specifications/Qwen3.8.md section 8).

Same pre-norm skeleton as the attention block; only the mixer differs:

x <- x + out_proj( gated_norm( delta_rule( conv(qk), conv(v), a, b ), z ) ) x <- x + down( swiglu( gate_up( post_norm(x) ) ) )

THREE THINGS HAVE NO PRECEDENT IN THE TREE, and each is why a piece was built first:

• The mixer's memory is a fixed-size RECURRENT STATE, not a growing KV cache. Nothing here scales with context length, which is the property that lets 48 of 64 layers cost nothing extra at long context (section 3).
• A short depthwise CAUSAL CONVOLUTION sits between the projections and the mixer, carrying its own rolling window of the last kernel_width - 1 positions.
• The output norm is GATED and, unlike every other norm in this family, uses the RAW weight rather than the unit-offset form. Both conventions live in one model; see the note at createGraph().

PER-ROLE PRECISION. The plan splits this block's projections three ways, and the split is not cosmetic – section 5's parameter counts only add up this way:

DeltaNetQueryKey q, k 1.007 B DeltaNetValueGateOutput v, z, out_proj 4.531 B DeltaNetGating a, b 0.024 B, NEVER quantized

DeltaNetGating is NoWeightQuant on purpose: a and b drive the forget gate, where error compounds EXPONENTIALLY over the sequence rather than linearly.

WEIGHT LAYOUT CONTRACT. The checkpoint fuses q, k and v into one in_proj_qkv [10240, 5120]; this block takes them as TWO projections, [q|k] at 4096 and [v] at 6144, because one tensor cannot carry two storage policies and the plan puts q/k a half-step above v. The converter owns that split, and splits conv1d.weight [10240, 4] the same way. Splitting the convolution is exact rather than approximate: it is DEPTHWISE, so no channel ever reads another, and two convolutions over a partition of the channels compute what one convolution over all of them computes.

Inference-only, like the other blocks: implements ITransformerBlock, not forward/backward.

Function Documentation

makeQwenDeltaNetBlockWorkspace()

template<DeviceType TDeviceType, TensorDataType TPrecision>
requires PrecisionSupportedOnDevice<TPrecision, TDeviceType>

QwenDeltaNetBlockWorkspace< TDeviceType, TPrecision > Mila::Dnn::makeQwenDeltaNetBlockWorkspace ( const QwenConfig & config, DeviceId device, dim_t B, dim_t prefill_chunk, const std::string & name_prefix )

export

Allocate the shared DeltaNet-block workspace.

Lives beside the struct rather than inside QwenTransformer for the reason its attention counterpart does: a layer-streamed harness drives one block at a time and needs the identical slot geometry, and a second copy of these widths would agree until the first time one of them changed.

qwenDeltaNetWorkspaceSlotWidths()

std::vector< dim_t > Mila::Dnn::qwenDeltaNetWorkspaceSlotWidths ( const QwenConfig & config )

inlineexport

The width of each workspace slot, one entry per [B, chunk, width] allocation.

Shared by the transformer's footprint and read against the allocation below, so the two cannot drift – the same reason the attention block's widths live in one place.