MixtureOfExperts.ixx File Reference

Mixture-of-experts bank: the stacked experts and the weighted combine, routed from outside. More...

#include <cstddef>
#include <cstdint>
#include <format>
#include <memory>
#include <optional>
#include <sstream>
#include <stdexcept>
#include <string>
#include <vector>
#include "../Activations/Activation/Kernels/ElementwiseActivation.h"
import Serialization.SafeTensors;
import Serialization.Tensor;
import Serialization.Mode;
import Serialization.Metadata;
import Compute.ExecutionContextFactory;
import Compute.OperationTraits;
import Compute.ExecutionContext;
import Compute.IExecutionContext;
import Compute.DeviceId;
import Dnn.Components.MixtureOfExpertsConfig;
import Compute.DeviceTypeTraits;
import Compute.DeviceType;
import Dnn.ITensor;
import Dnn.TensorOps;
import Dnn.ComponentType;
import Dnn.Quantization.Weight.Policies;
import Dnn.TensorDataType;
import Dnn.Component;
import Dnn.Tensor;
import Dnn.TensorTypes;
import Serialization.ModelArchive;
import Dnn.TensorDataTypeTraits;
import Dnn.ActivationType;
import Compute.Observation;
import Compute.DeviceAllocation;
import Dnn.Components.Activation;

Classes
class	Mila::Dnn::MixtureOfExperts< TDeviceType, TPrecision, TGate, TWeightQuantization >
	Stacked experts: gate_up_proj [E, 2I, H] and down_proj [E, H, I], checkpoint-named. More...

Namespaces
namespace	Mila
	Mila main API namespace.

Detailed Description

Mixture-of-experts bank: the stacked experts and the weighted combine, routed from outside.

Routing is an input, not a child. Gemma 4 routes on the raw residual while its experts read pre_feedforward_layernorm_2 of it (Specifications/Gemma4MoE.md Phase 1), so a bank that owned its router would need a family-specific two-input signature. The block wires Router beside it.