CausalConv1d.ixx File Reference

Depthwise causal 1-D convolution over the sequence axis. More...

#include <memory>
#include <string>
#include <vector>
#include <sstream>
#include <format>
#include <stdexcept>
#include <optional>
#include <cstdint>
import Serialization.Tensor;
import Serialization.Metadata;
import Compute.CpuMemoryResource;
import Compute.ExecutionContextFactory;
import Compute.OperationTraits;
import Compute.ExecutionContext;
import Compute.DeviceTypeTraits;
import Dnn.Component;
import Serialization.SafeTensors;
import Serialization.Mode;
import Compute.Observation;
import Dnn.Components.CausalConv1dConfig;
import Compute.DeviceAllocation;
import Compute.DeviceType;
import Serialization.ModelArchive;
import Compute.MemoryResource;
import Dnn.ComponentType;
import Compute.IExecutionContext;
import Dnn.Tensor;
import Dnn.TensorTypes;
import Dnn.ITensor;
import Dnn.TensorDataTypeTraits;
import Dnn.TensorOps;
import Dnn.TensorDataType;
import Compute.Device;
import Compute.DeviceId;

Classes
class	Mila::Dnn::CausalConv1d< TDeviceType, TPrecision >

Namespaces
namespace	Mila
	Mila main API namespace.

Detailed Description

Depthwise causal 1-D convolution over the sequence axis.

One length-K filter per channel, no cross-channel mixing:

out[b, t, c] = bias[c] + sum(i = 0 .. K-1) weight[c, i] * x[b, t - (K-1) + i, c]

The first component in the tree that carries a convolution, and the first whose memory is a rolling window rather than a growing cache. Qwen 3.8's Gated DeltaNet layers run it over the 10240-wide fused q/k/v stream with K = 4 (linear_conv_kernel_dim).

PREFILL AND DECODE, like the attention components – and for the same reason. A causal convolution needs the K-1 positions to its left, which during decode are in the past rather than in the input. prefill convolves a chunk; decode convolves one token against the retained rows. Both refresh the state afterwards, so chunked prefill composes: chunk N+1 sees chunk N's tail exactly as if the sequence had arrived whole.

NO ACTIVATION. The reference fuses SiLU into its convolution kernel; this component does not, because the tree already has an elementwise activation and fusing them would need a second operation type for a saving that has not been measured. The block composes the two, matching how AttentionOutputGate composes rather than fuses.

Inference-only: no backward. Training the DeltaNet layers is not in scope, and a backward that exists but has never been exercised is worse than one that does not.