Mila::Dnn::MemoryStats Struct Reference

Memory allocation breakdown for a single component. More...

Public Member Functions
std::size_t	activeDeviceParameterBytes () const noexcept
	Device parameter bytes a single token reads.
MemoryStats &	operator+= (const MemoryStats &rhs) noexcept
	Accumulate another MemoryStats into this one.
std::string	toString () const
	Produce a human-readable summary of this stats instance.
std::size_t	totalBytes () const noexcept
	Total memory allocated across all categories and locations.
std::size_t	totalDeviceBytes () const noexcept
	Total device memory allocated by this component.
std::size_t	totalHostBytes () const noexcept
	Total host memory allocated by this component.

Public Attributes
std::size_t	device_gradient_bytes { 0 }
	Input and parameter gradient buffers.
std::size_t	device_inactive_parameter_bytes { 0 }
	Of device_parameter_bytes, what a single token does not read: the unselected experts of a sparse layer.
std::size_t	device_parameter_bytes { 0 }
	Learnable parameter buffers (weights, biases).
std::size_t	device_scratch_bytes { 0 }
	The execution context's forward scratch: one buffer shared by every operation on the context, sized by the largest single request.
std::size_t	device_state_bytes { 0 }
	Forward and decode output buffers, KV cache.
std::size_t	host_gradient_bytes { 0 }
	Gradient buffers on host (if any).
std::size_t	host_parameter_bytes { 0 }
	Learnable parameter buffers pinned on host (if any).
std::size_t	host_state_bytes { 0 }
	Forward and decode output buffers on host (if any).

Detailed Description

Memory allocation breakdown for a single component.

Reflects the current allocation state at the moment of the call. Categories map directly onto the component build lifecycle:

After construction nothing – construction allocates no device memory After build() parameters + state After setTrainingMode() parameters + state + gradients

Parameters are allocated in onBuilding(), not in the constructor. That is what lets Component::getRequiredMemory() report a footprint from a constructed but unbuilt graph. See Specifications/MemoryFootprint.md section 3.1.

All sizes are in bytes. Device and host allocations are tracked separately as they represent distinct, independently constrained resources.

Member Function Documentation

activeDeviceParameterBytes()

std::size_t Mila::Dnn::MemoryStats::activeDeviceParameterBytes ( ) const

inlinenodiscardnoexcept

Device parameter bytes a single token reads.

Equals device_parameter_bytes when dense.

operator+=()

MemoryStats & Mila::Dnn::MemoryStats::operator+= ( const MemoryStats & rhs )

inlinenoexcept

Accumulate another MemoryStats into this one.

Used by CompositeComponent and Network to aggregate child stats. Scratch is taken by maximum: every operation of a tree shares the one buffer.

toString()

std::string Mila::Dnn::MemoryStats::toString ( ) const

inlinenodiscard

Produce a human-readable summary of this stats instance.

Returns

Formatted multi-line string.

Member Data Documentation

device_gradient_bytes

std::size_t Mila::Dnn::MemoryStats::device_gradient_bytes { 0 }

Input and parameter gradient buffers.

Allocated lazily on first setTraining(true). Retained thereafter.

device_inactive_parameter_bytes

std::size_t Mila::Dnn::MemoryStats::device_inactive_parameter_bytes { 0 }

Of device_parameter_bytes, what a single token does not read: the unselected experts of a sparse layer.

Resident, never a separate allocation; zero when dense.

device_parameter_bytes

std::size_t Mila::Dnn::MemoryStats::device_parameter_bytes { 0 }

Learnable parameter buffers (weights, biases).

Allocated at construction. Static for the component lifetime.

device_scratch_bytes

std::size_t Mila::Dnn::MemoryStats::device_scratch_bytes { 0 }

The execution context's forward scratch: one buffer shared by every operation on the context, sized by the largest single request.

Combined by maximum, never summed. Reserved when a network is built.

device_state_bytes

std::size_t Mila::Dnn::MemoryStats::device_state_bytes { 0 }

Forward and decode output buffers, KV cache.

Allocated at build(). Static after build().

---

The documentation for this struct was generated from the following file:

• Mila/Src/Dnn/Core/Component.MemoryStats.ixx