MixtureOfExperts.Config.ixx File Reference

Configuration for a mixture-of-experts bank: expert geometry, count, and experts per token. More...

#include <stdexcept>
#include <string>
#include <sstream>
#include <format>
#include <cstdint>
import Serialization.Metadata;
import Dnn.ComponentConfig;
import Dnn.TensorTypes;

Classes
class	Mila::Dnn::MixtureOfExpertsConfig

Namespaces
namespace	Mila
	Mila main API namespace.

Detailed Description

Configuration for a mixture-of-experts bank: expert geometry, count, and experts per token.

Gemma 4 26B-A4B: 128 experts of intermediate width 704 over a 2816-wide stream, 8 per token. See Specifications/Gemma4MoE.md Phase 6.