GatedMLP.Config.ixx File Reference

Configuration for the gated feed-forward (GatedMLP) block. More...

#include <stdexcept>
#include <cstdint>
#include <string>
#include <utility>
#include <sstream>
import Serialization.Metadata;
import Dnn.ActivationType;
import Dnn.ComponentConfig;
import Dnn.TensorTypes;

Classes
class	Mila::Dnn::GatedMLPConfig
	Configuration class for the gated feed-forward (GatedMLP) block. More...

Namespaces
namespace	Mila
	Mila main API namespace.

Detailed Description

Configuration for the gated feed-forward (GatedMLP) block.

The gated FFN (Llama, Mistral, Qwen, ... and MoE experts) projects to a fused 2H gate+up, applies a gated activation, then projects back down. See Specifications/FfnAndMoE.md section 7.