Mila::Dnn::Compute::IKvCacheLifecycle Struct Reference

Capability interface for KV-cache state management. More...

Inheritance diagram for Mila::Dnn::Compute::IKvCacheLifecycle:

Public Member Functions
virtual void	initializeKvCache (dim_t batch_size, dim_t max_sequence_length)=0
	Allocate the KV cache for a given batch size and maximum sequence length.
virtual void	resetKvCache ()=0
	Reset the KV cache to an empty state, preserving the allocation.
virtual bool	rewindKvCache (dim_t position)=0
	Rewind the logical cache fill position without touching device K/V buffer contents, so positions [0, position) can be reused by a subsequent partial prefill (PromptCaching.md).

Detailed Description

Capability interface for KV-cache state management.

Implemented by attention operations (GQA, MHA) that allocate and maintain key/value caches across autoregressive decode steps. This concern is orthogonal to positional dispatch – an operation may implement both IPositionalUnaryOp and IKVCacheLifecycle.

Member Function Documentation

initializeKvCache()

virtual void Mila::Dnn::Compute::IKvCacheLifecycle::initializeKvCache ( dim_t batch_size, dim_t max_sequence_length )

pure virtual

Allocate the KV cache for a given batch size and maximum sequence length.

Parameters

batch_size	Number of sequences in the batch.
max_sequence_length	Maximum number of tokens the cache must hold.

rewindKvCache()

virtual bool Mila::Dnn::Compute::IKvCacheLifecycle::rewindKvCache ( dim_t position )

pure virtual

Rewind the logical cache fill position without touching device K/V buffer contents, so positions [0, position) can be reused by a subsequent partial prefill (PromptCaching.md).

Returns

true when the rewind is valid and was applied. Implementations must refuse (return false, cache state unchanged) when reuse would be incorrect – e.g. position exceeds the current fill, or a bounded sliding-window ring has already overwritten the window a continuation from position would attend to. On false the caller falls back to a full prefill, which positionally overwrites regardless of cache state.

---

The documentation for this struct was generated from the following file:

• Mila/Src/Dnn/Compute/Operations/IKVCacheLifecycle.ixx