Rope.Config.ixx File Reference

Configuration for Rotary Position Embedding (RoPE) component. More...

#include <stdexcept>
#include <string>
#include <sstream>
#include <utility>
import Serialization.Metadata;
import Dnn.TensorTypes;
import Dnn.ComponentConfig;
import Dnn.Component;

Classes
class	Mila::Dnn::RopeConfig

Namespaces
namespace	Mila
	Mila main API namespace.

Enumerations
enum class	Mila::Dnn::RotaryLayout { WholeHead , RotaryPrefix }
	Which dimension pairs a partial rotary rotates. More...

Detailed Description

Configuration for Rotary Position Embedding (RoPE) component.

Provides construction, validation and serialization for RoPE configuration.

Design principle (Mila-wide):

• Constructor parameters are structurally required – no sensible default exists.
• Fluent setters are reserved for optional behavioural parameters that have well-known defaults. There are no fluent overrides for constructor parameters.

Required (constructor): channels, n_heads, n_kv_heads, max_seq_len. Optional (fluent): base (default 10000.0f), rotary_dim (default 0 = full head_dim).

Typical usage:

auto cfg = RopeConfig( model_dim, n_heads, n_kv_heads, max_seq_len )
    .withBase( 500000.0f );  // Llama 3 theta

Enumeration Type Documentation

RotaryLayout

enum class Mila::Dnn::RotaryLayout

exportstrong

Which dimension pairs a partial rotary rotates.

Families genuinely disagree.

Only meaningful when rotary_dim is below head_dim; the two coincide at full rotary. Getting it wrong is silent: both layouts rotate the same NUMBER of dimensions and produce a plausible result, and at a short prompt the angles are small enough that generation still looks correct. It diverges with context.

Enumerator
WholeHead	Rotation spans the whole head, pairing channel i with i + head_dim/2, and the partial width is expressed by zero frequencies in the cos/sin cache. Gemma: (x * cos) + (rotate_half(x) * sin) over the unsliced head.
RotaryPrefix	Rotation is confined to the leading rotary_dim channels, pairing i with i + rotary_dim/2; channels at or above rotary_dim pass through untouched. Qwen: q[..., :rotary_dim] then rotate_half inside that slice.