Token sequence loader for autoregressive language models. More...

Inheritance diagram for Mila::Data::TokenSequenceLoader< TMemoryResource >:

Public Types
using	BaseLoader = DataLoader<TensorDataType::INT32, TensorDataType::INT32, TMemoryResource>
using	HostType = typename TensorHostTypeMap<TensorDataType::INT32>::host_type
using	TensorType = Tensor<TensorDataType::INT32, TMemoryResource>
Public Types inherited from Mila::Data::DataLoader< TensorDataType::INT32, TensorDataType::INT32, TMemoryResource >
using	InputDataType
	Input tensor abstract data type.
using	InputTensor
	Input tensor type alias.
using	MemoryResource
	Memory resource type for tensor allocation.
using	TargetDataType
	Target tensor abstract data type.
using	TargetTensor
	Target tensor type alias.

Public Member Functions
	TokenSequenceLoader (const std::filesystem::path &tokens_file, int64_t batch_size, int64_t seq_length, bool is_training, DeviceId device, const TokenSequenceLoaderConfig &config=TokenSequenceLoaderConfig())
	Constructs streaming autoregressive sequence loader.
	TokenSequenceLoader (const TokenSequenceLoader &)=delete
	TokenSequenceLoader (TokenSequenceLoader &&)=delete
const TensorType &	inputs () const override
	Provides immutable access to input tensor for current batch.
TensorType &	inputs () override
	Provides mutable access to input tensor for current batch.
void	nextBatch () override
	Loads the next batch of data from the dataset.
int64_t	numBatches () const override
	Returns the total number of batches in the dataset.
size_t	numTokens () const
size_t	numWindows () const
TokenSequenceLoader &	operator= (const TokenSequenceLoader &)=delete
TokenSequenceLoader &	operator= (TokenSequenceLoader &&)=delete
void	reset () override
	Resets the loader to the beginning of the dataset.
int64_t	sequenceLength () const
const TensorType &	targets () const override
	Provides immutable access to target tensor for current batch.
TensorType &	targets () override
	Provides mutable access to target tensor for current batch.
size_t	windowSizeTokens () const
Public Member Functions inherited from Mila::Data::DataLoader< TensorDataType::INT32, TensorDataType::INT32, TMemoryResource >
	DataLoader (const DataLoader &)=delete
	Copy operations explicitly deleted for performance safety.
	DataLoader (DataLoader &&)=default
	Move operations for efficient ownership transfer.
	DataLoader (int64_t batch_size)
	Constructs data loader with specified batch configuration.
virtual	~DataLoader ()=default
	Virtual destructor ensuring proper cleanup in derived classes.
int64_t	batchSize () const noexcept
	Returns the configured batch size.
int64_t	currentBatch () const noexcept
	Returns the current batch index.
virtual std::string	getDatasetInfo () const
	Returns dataset statistics for optimization and analysis.
virtual bool	hasNext () const
	Checks if more batches are available.
DataLoader &	operator= (const DataLoader &)=delete
DataLoader &	operator= (DataLoader &&)=default
virtual bool	validateCurrentBatch () const
	Validates current batch data integrity.

Additional Inherited Members
Static Public Member Functions inherited from Mila::Data::DataLoader< TensorDataType::INT32, TensorDataType::INT32, TMemoryResource >
static constexpr bool	supportsMixedPrecision () noexcept
	Checks if data loader supports mixed-precision workflows.
static constexpr bool	usesPinnedMemory () noexcept
	Checks if data loader uses pinned memory for GPU optimization.
Static Public Attributes inherited from Mila::Data::DataLoader< TensorDataType::INT32, TensorDataType::INT32, TMemoryResource >
static constexpr TensorDataType	input_data_type
	Compile-time input data type constant.
static constexpr bool	is_mixed_precision
	Mixed-precision workflow detection.
static constexpr TensorDataType	target_data_type
	Compile-time target data type constant.
static constexpr bool	uses_pinned_memory
	Pinned memory optimization (CUDA-only; false on CPU-only builds).
Protected Member Functions inherited from Mila::Data::DataLoader< TensorDataType::INT32, TensorDataType::INT32, TMemoryResource >
void	incrementBatch () noexcept
	Increments current batch counter.
void	setCurrentBatch (int64_t batch_index) noexcept
	Updates current batch counter.

Detailed Description

template<typename TMemoryResource>
requires (std::is_same_v<TMemoryResource, CpuMemoryResource>)
class Mila::Data::TokenSequenceLoader< TMemoryResource >

Token sequence loader for autoregressive language models.

Loads tokenized text data for causal language modeling tasks such as GPT, LLaMA, and other transformer-based models. Reads from pre-tokenized binary .tokens files and produces batches of (input, target) sequence pairs where target[i] = input[i+1] (next-token prediction).

Implementation uses efficient disk streaming with double-buffered producer-consumer pattern for high-throughput training on large corpora.

Template Parameters

TMemoryResource CpuMemoryResource or CudaPinnedMemoryResource

Constructor & Destructor Documentation

◆ TokenSequenceLoader()

template<typename TMemoryResource>

Mila::Data::TokenSequenceLoader< TMemoryResource >::TokenSequenceLoader	(	const std::filesystem::path &	tokens_file,
		int64_t	batch_size,
		int64_t	seq_length,
		bool	is_training,
		DeviceId	device,
		const TokenSequenceLoaderConfig &	config = TokenSequenceLoaderConfig() )

inline

Constructs streaming autoregressive sequence loader.

Parameters

tokens_file	Path to binary .tokens file (uint32_t format)
batch_size	Number of sequences per batch
seq_length	Context window length (tokens per sequence)
is_training	Enable shuffling and continuous epochs
device	Compute device for tensor allocation
config	Performance and streaming configuration

Exceptions

std::invalid_argument	If batch_size or seq_length is zero
std::runtime_error	If file operations or initialization fails

Member Function Documentation

◆ inputs() [1/2]

template<typename TMemoryResource>

const TensorType & Mila::Data::TokenSequenceLoader< TMemoryResource >::inputs ( ) const

inlineoverridevirtual

Provides immutable access to input tensor for current batch.

Derived classes must implement this method to provide read-only access to the tensor containing input data for the currently loaded batch.

Returns: Const reference to input tensor containing current batch data

Note: Enables safe access for analysis and debugging without modification risk; Should return same data as mutable version

Implements Mila::Data::DataLoader< TensorDataType::INT32, TensorDataType::INT32, TMemoryResource >.

◆ inputs() [2/2]

template<typename TMemoryResource>

TensorType & Mila::Data::TokenSequenceLoader< TMemoryResource >::inputs ( )

inlineoverridevirtual

Provides mutable access to input tensor for current batch.

Derived classes must implement this method to provide access to the tensor containing input data for the currently loaded batch. The tensor should be properly shaped and contain valid data after nextBatch() call.

Returns: Mutable reference to input tensor containing current batch data

Note: Tensor shape should match expected input dimensions for the model; Data should be preprocessed and ready for model consumption; Memory layout should be optimized for target compute device

Implements Mila::Data::DataLoader< TensorDataType::INT32, TensorDataType::INT32, TMemoryResource >.

◆ nextBatch()

template<typename TMemoryResource>

void Mila::Data::TokenSequenceLoader< TMemoryResource >::nextBatch ( )

inlineoverridevirtual

Loads the next batch of data from the dataset.

Derived classes must implement this method to load the next batch of data into the input and target tensors. Implementation should handle data preprocessing, memory allocation, and batch composition according to the specific dataset requirements.

Exceptions

std::runtime_error	If no more batches are available
std::runtime_error	If data loading fails

Note: Implementation must update current_batch_ counter after successful load; Should handle end-of-dataset conditions appropriately; May involve complex preprocessing pipelines and data augmentation

Implements Mila::Data::DataLoader< TensorDataType::INT32, TensorDataType::INT32, TMemoryResource >.

◆ numBatches()

template<typename TMemoryResource>

int64_t Mila::Data::TokenSequenceLoader< TMemoryResource >::numBatches ( ) const

inlineoverridevirtual

Returns the total number of batches in the dataset.

Derived classes must implement this method to report the total number of batches available in their specific dataset. This information is essential for training loop progress tracking and epoch management.

Returns: Total number of batches available in the dataset

Note: Implementation should account for partial batches at dataset end; Value may change if dataset is modified or resampled; Used for training progress reporting and epoch boundary detection

Implements Mila::Data::DataLoader< TensorDataType::INT32, TensorDataType::INT32, TMemoryResource >.

◆ reset()

template<typename TMemoryResource>

void Mila::Data::TokenSequenceLoader< TMemoryResource >::reset ( )

inlineoverridevirtual

Resets the loader to the beginning of the dataset.

Resets the internal state to start iteration from the first batch. Derived classes may override this method to implement additional reset functionality such as dataset reshuffling or preprocessing pipeline reinitialization.

Note: Base implementation resets batch counter to zero; Called automatically at epoch boundaries in training loops; Override to implement custom reset behavior (shuffling, etc.)

Reimplemented from Mila::Data::DataLoader< TensorDataType::INT32, TensorDataType::INT32, TMemoryResource >.

◆ targets() [1/2]

template<typename TMemoryResource>

const TensorType & Mila::Data::TokenSequenceLoader< TMemoryResource >::targets ( ) const

inlineoverridevirtual

Provides immutable access to target tensor for current batch.

Derived classes must implement this method to provide read-only access to the tensor containing target/label data for the currently loaded batch.

Returns: Const reference to target tensor containing current batch labels

Note: Enables safe access for analysis and debugging without modification risk; Should return same data as mutable version

Implements Mila::Data::DataLoader< TensorDataType::INT32, TensorDataType::INT32, TMemoryResource >.

◆ targets() [2/2]

template<typename TMemoryResource>

TensorType & Mila::Data::TokenSequenceLoader< TMemoryResource >::targets ( )

inlineoverridevirtual

Provides mutable access to target tensor for current batch.

Derived classes must implement this method to provide access to the tensor containing target/label data for the currently loaded batch. The tensor should contain ground truth data corresponding to the inputs.

Returns: Mutable reference to target tensor containing current batch labels

Note: Target data should align with input batch ordering; Data format should match model's expected output structure; For mixed-precision workflows, may use different data type than inputs

Implements Mila::Data::DataLoader< TensorDataType::INT32, TensorDataType::INT32, TMemoryResource >.

The documentation for this class was generated from the following file:

Mila/Src/Data/Loaders/TokenSequenceLoader.ixx

Public Types

Public Member Functions

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ TokenSequenceLoader()

Member Function Documentation

◆ inputs() [1/2]

◆ inputs() [2/2]

◆ nextBatch()

◆ numBatches()

◆ reset()

◆ targets() [1/2]

◆ targets() [2/2]