What a deployment would cost, and how it would prefill. More...

Public Attributes
MemoryStats	memory
PrefillChunking	prefill

Detailed Description

What a deployment would cost, and how it would prefill.

Fitting in memory and running well are different questions, and a caller choosing a context length needs both: the largest context that fits can be one where the prefill chunk has walked down to its floor. The two answers come from one graph construction because they are two readings of the same arithmetic.

The documentation for this struct was generated from the following file:

Mila/Src/Dnn/Core/Component.MemoryStats.ixx

Public Attributes

Detailed Description