Per-call inputs to LanguageModel::generate. More...

Public Attributes
std::optional< int >	max_new_tokens
SamplingParams	sampling {}
std::vector< TokenId >	stop_tokens {}

Detailed Description

Per-call inputs to LanguageModel::generate.

max_new_tokens is an optional ceiling – nullopt runs to EOS or the deployment context bound (no magic default, no silent truncation). sampling is the only slice forwarded to the sampler; loop control never reaches it. stop_tokens overrides the model's default stop set for this call (empty => model defaults), kept for advanced structured generation – EOS is otherwise a model/tokenizer property established at construction, not a per-call parameter.

The documentation for this struct was generated from the following file:

Mila/Src/Dnn/Components/Transformers/GenerateParams.ixx

Public Attributes

Detailed Description