Mila::Dnn::PrefillChunking Struct Reference

How a transformer would chunk a prefill pass at a given context length. More...

Public Member Functions
bool	isMemoryConstrained () const noexcept
	True when the free memory forced a smaller chunk than the context permits.

Public Attributes
dim_t	chunk_rows { 0 }
	Rows per prefill chunk this context length would use.
bool	fits_available_memory { true }
	False when even the floor rung does not fit the free memory, so chunk_rows is the floor used in spite of it rather than one that fits.
dim_t	unconstrained_chunk_rows { 0 }
	The largest rung this context length permits, memory aside.

Detailed Description

How a transformer would chunk a prefill pass at a given context length.

Deliberately not a field of MemoryStats: that type is summed across a component tree, and a chunk size is a single decision the transformer makes rather than a quantity children contribute to.

Every family takes the largest rung of its table whose whole predicted footprint fits the device's free memory. A longer context, or less free memory, can therefore buy a smaller chunk, and that is the fact this type exists to make askable. See Specifications/MemoryFootprint.md section 11.

Member Data Documentation

unconstrained_chunk_rows

dim_t Mila::Dnn::PrefillChunking::unconstrained_chunk_rows { 0 }

The largest rung this context length permits, memory aside.

Equal to chunk_rows when the free memory did not reduce the chunk.

---

The documentation for this struct was generated from the following file:

• Mila/Src/Dnn/Core/Component.MemoryStats.ixx