GatedDeltaRule.ixx File Reference

Gated delta rule – the linear-attention mixer of Qwen 3.8's DeltaNet layers. More...

#include <memory>
#include <string>
#include <vector>
#include <sstream>
#include <format>
#include <stdexcept>
#include <optional>
#include <cstdint>
import Serialization.SafeTensors;
import Serialization.Tensor;
import Serialization.Metadata;
import Compute.CpuMemoryResource;
import Compute.MemoryResource;
import Compute.ExecutionContextFactory;
import Serialization.Mode;
import Compute.ExecutionContext;
import Compute.DeviceTypeTraits;
import Compute.Observation;
import Compute.OperationTraits;
import Dnn.Components.GatedDeltaRuleConfig;
import Serialization.ModelArchive;
import Dnn.ComponentType;
import Dnn.TensorDataType;
import Dnn.Component;
import Dnn.Tensor;
import Dnn.ITensor;
import Compute.IExecutionContext;
import Dnn.TensorTypes;
import Compute.DeviceId;
import Dnn.TensorDataTypeTraits;
import Compute.DeviceAllocation;
import Compute.Device;
import Dnn.TensorOps;
import Compute.DeviceType;

Classes
class	Mila::Dnn::GatedDeltaRule< TDeviceType, TPrecision >

Namespaces
namespace	Mila
	Mila main API namespace.

Detailed Description

Gated delta rule – the linear-attention mixer of Qwen 3.8's DeltaNet layers.

The counterpart to GroupedQueryAttention in the other layer kind, and the first mixer in the tree whose memory is a fixed-size RECURRENT STATE rather than a growing cache. Per value head the state is [head_key_dim, head_value_dim] and each step is

S <- S * exp(g_t) delta <- (v_t - k_t^T S) * beta_t S <- S + k_t (x) delta out_t <- q_t^T S

with q and k L2-normalized and q scaled by 1/sqrt(head_key_dim).

THE STATE DOES NOT GROW WITH CONTEXT. That is the whole point of the layer kind: 48 of Qwen 3.8 27B's 64 layers hold one of these instead of a KV cache, so their cost is flat in sequence length. It inverts the usual long-context budget – see Qwen3.8.md section 3.

g AND beta ARE DERIVED HERE, from the raw a/b projections and this component's own A_log and dt_bias parameters (beta = sigmoid(b), g = -exp(A_log) * softplus(a + dt_bias)). The alternative – computing them in the block – would put a softplus on the public activation enum for one caller and add two launches over [B, T, heads] tensors.

TWO FORMS, ONE DEFINITION. Decode runs the sequential recurrence above; prefill runs the chunked (UT-transform) form, which regroups the identical arithmetic into one triangular solve per 32 steps and so exposes far more of it at once. The backend chooses on the step count and the component does not know which ran. The recurrent form remains the oracle: the two agree to float reassociation, not by definition.

Inference-only: no backward.