AttentionOutputGate.ixx File Reference

Gate on the attention output: out = TGate(gate) * value, two separate inputs. More...

#include <memory>
#include <vector>
#include <string>
#include <sstream>
#include <stdexcept>
#include <format>
#include <optional>
#include "../../Activations/Activation/Kernels/ElementwiseActivation.h"
import Compute.CpuMemoryResource;
import Compute.OperationTraits;
import Serialization.ModelArchive;
import Compute.IExecutionContext;
import Serialization.Mode;
import Compute.DeviceTypeTraits;
import Compute.DeviceType;
import Compute.Observation;
import Compute.DeviceId;
import Compute.DeviceAllocation;
import Dnn.Component;
import Dnn.ComponentType;
import Compute.ExecutionContextFactory;
import Dnn.ActivationType;
import Dnn.ITensor;
import Serialization.Metadata;
import Dnn.Components.ActivationConfig;
import Dnn.TensorOps;
import Dnn.Components.Activation;
import Dnn.TensorDataType;
import Dnn.Tensor;
import Dnn.TensorDataTypeTraits;
import Compute.Device;
import Dnn.Components.AttentionOutputGateConfig;
import Dnn.TensorTypes;

Classes
class	Mila::Dnn::AttentionOutputGate< TDeviceType, TPrecision, TGate >
	Elementwise gate over two same-shaped inputs: out = TGate(gate) * value. More...

Namespaces
namespace	Mila
	Mila main API namespace.

Detailed Description

Gate on the attention output: out = TGate(gate) * value, two separate inputs.

Qwen 3.8 sets attn_output_gate: true, which doubles the query projection to emit [query | gate] and scales the attention output elementwise by the gated half before the output projection (Specifications/Qwen3.8.md section 2).

This is NOT Swiglu with a different name. Swiglu splits ONE buffer into gate|value halves, and both halves are the same projection's output; here the gate arrives from the query projection while the value is the attention output, several operations later. That makes it a binary component – two independently-produced inputs of the same shape.