Gemma.Config.ixx File Reference

Network-level configuration for Gemma 4 transformer networks. More...

#include <stdexcept>
#include <string>
#include <sstream>
#include <utility>
#include <cmath>
import Serialization.Metadata;
import Dnn.ComponentConfig;
import Dnn.TensorTypes;

Classes
class	Mila::Dnn::GemmaConfig
	Network-level configuration for Gemma 4 transformer networks. More...

Namespaces
namespace	Mila
	Mila main API namespace.

Detailed Description

Network-level configuration for Gemma 4 transformer networks.

Gemma 4 decouples per-head width from the residual stream: head_dim is NOT embedding_dim / num_heads (Gemma 4 12B: 3840 / 16 = 240, but head_dim is 256). This config therefore carries head_dim as an explicit, first-class field separate from embedding_dim, where LlamaConfig derives it from the residual dimension. See Specifications/Gemma.md sections 2-3.

Step 0 of the Gemma foundation sequence (Gemma.md section 9) is exactly this decoupling. Gemma-specific axes that arrive in later steps are intentionally absent here and noted at their fields:

• global-layer geometry (global_head_dim, num_global_kv_heads, k_eq_v) - Step 1
• dual-RoPE bases / partial-rotary - Step 3
• per-layer sliding/global type, window, logit softcap - Steps 2/5