Gaussian Error Linear Unit (GELU) activation component. More...

Inheritance diagram for Mila::Dnn::Gelu< TDeviceType, TPrecision >:

Public Types
using	ComponentBase = Component<TDeviceType, TPrecision>
using	MR = typename DeviceTypeTraits<TDeviceType>::memory_resource
using	TensorType = Tensor<TPrecision, MR>

Public Member Functions
	Gelu (const std::string &name, const GeluConfig &config, std::optional< DeviceId > device_id=std::nullopt)
	Construct GELU activation component with optional ExecutionContext ownership.
TensorType &	backward (const TensorType &input, const TensorType &output_grad)
	Compute gradients with respect to the component input.
TensorType &	forward (const TensorType &input)
	Run the forward computation for this GELU component.
ApproximationMethod	getApproximationMethod () const
	Return the configured GELU approximation method.
DeviceId	getDeviceId () const override
	Get the device identifier for this component.
std::vector< ITensor * >	getGradients () const override
	Get parameter gradient tensors.
MemoryStats	getMemoryStats () const override
	Return memory allocation breakdown.
std::vector< ITensor * >	getParameters () const override
	Get trainable parameter tensors.
const ComponentType	getType () const override
	Get the component type identifier.
dim_t	parameterCount () const override
	Number of trainable parameters.
void	save_ (ModelArchive &archive, SerializationMode mode) const override
	Persist component state to archive.
void	synchronize () override
	Wait for all asynchronous work submitted by this component to complete.
std::string	toString () const override
	Generate human-readable description of the component.
Public Member Functions inherited from Mila::Dnn::Component< TDeviceType, TPrecision >
	Component (const std::string &name)
	Construct component with required name identifier.
virtual void	build (const BuildContext &context) final
	Build the component with the provided BuildContext (canonical overload).
const std::string	getName () const
	Get the component's name identifier.
virtual std::vector< std::string >	getParameterNames () const
	List all available parameter names for this component.
virtual MemoryStats	getRequiredMemory (const BuildContext &context) const
	Report what build( context ) would allocate, without allocating it.
TrainingMode	getTrainingMode () const noexcept
	The current runtime behavioral mode of this Component.
virtual bool	isBuilt () const final
	Returns true if build() has completed successfully.
virtual void	load_ (ModelArchive &archive, SerializationMode mode)
	Restore this component's parameters from its archive scope.
virtual void	loadParameter (const std::string &, const Serialization::ITensorBlob &)
	Load a parameter from serialized tensor data.
virtual void	requireSerializableParameters () const
	Verify this component can serialize whatever parameters it owns.
virtual void	saveFlatTensors (Serialization::SafeTensorsWriter &writer, const std::string &prefix, Serialization::TensorSavePass pass) const
	Drive this component's tensors through one pass of a flat safetensors save.
void	setTrainingMode (TrainingMode mode)
	Set the runtime behavioral mode for this Component.
virtual void	zeroGradients ()
	Clear all model-owned gradients for this component.

Static Public Member Functions
static std::unique_ptr< Gelu >	fromArchive_ (ModelArchive &archive, const std::string &component_name, IExecutionContext *exec_context)
Static Public Member Functions inherited from Mila::Dnn::Component< TDeviceType, TPrecision >
static constexpr DeviceType	getDeviceType ()
	Compile-time device type for this component instance.
static constexpr TensorDataType	getPrecision () noexcept
	Compile-time tensor precision for this component instance.

Protected Member Functions
void	onBuilding (const BuildContext &build_context) override
	Hook invoked during build() to initialize backend operation.
void	onExecutionContextSet () override
	Hook invoked after ExecutionContext is set.
void	onTrainingModeChanging (TrainingMode training_mode) override
	Hook invoked when training mode changes.
Protected Member Functions inherited from Mila::Dnn::Component< TDeviceType, TPrecision >
IExecutionContext *	getExecutionContext () const
	Get the shared execution context.
bool	hasExecutionContext () const noexcept
	Check if execution context has been set.
template<TensorDataType TParameterPrecision, typename TMemoryResource>
void	loadParameterFromBlob (const std::string &param_name, const Serialization::ITensorBlob &blob, Tensor< TParameterPrecision, TMemoryResource > &target, const shape_t &expected_shape)
	Load a tensor blob into a parameter tensor with validation.
template<TensorDataType TParameterPrecision, typename TMemoryResource>
void	saveParameterToArchive (ModelArchive &archive, const std::string &parameter_name, const Tensor< TParameterPrecision, TMemoryResource > &parameter) const
	Write one parameter tensor into the archive under "tensors/<name>".
template<TensorDataType TParameterPrecision, typename TMemoryResource>
void	saveParameterToWriter (Serialization::SafeTensorsWriter &writer, const std::string &flat_name, const Tensor< TParameterPrecision, TMemoryResource > &parameter, Serialization::TensorSavePass pass) const
	Drive one parameter through one pass of a flat safetensors save.
void	setExecutionContext (IExecutionContext *context)
	Set the execution context for this component.

Additional Inherited Members
Protected Types inherited from Mila::Dnn::Component< TDeviceType, TPrecision >
using	HostStagingMemoryResource
	Host memory a device-resident parameter stages through.
Protected Attributes inherited from Mila::Dnn::Component< TDeviceType, TPrecision >
BuildContext	build_context_ { shape_t{ 1 }, RuntimeMode::Training }
	The BuildContext stored at build time.

Detailed Description

template<DeviceType TDeviceType, TensorDataType TPrecision>
requires PrecisionSupportedOnDevice<TPrecision, TDeviceType>
class Mila::Dnn::Gelu< TDeviceType, TPrecision >

Gaussian Error Linear Unit (GELU) activation component.

Device-templated GELU component that performs forward (and optionally backward) computation by delegating to a registered device-specific UnaryOperation implementation found via the OperationRegistry.

Template Parameters

TDeviceType	Compile-time device identifier (DeviceType::Cpu or DeviceType::Cuda).
TPrecision	Tensor data precision used by this component.

Construction Modes:

Standalone mode: Construct with DeviceId to create and own an ExecutionContext. The component manages the context lifetime and uses it for operation execution.
Shared mode: Construct without DeviceId; parent (Network/CompositeComponent) provides ExecutionContext via setExecutionContext() after construction.

Ownership:

Standalone mode: Component owns its ExecutionContext (stored in owned_exec_context_).
Shared mode: Component borrows ExecutionContext from parent; lifecycle managed externally.

Preconditions:

The component's build(const shape_t&) must be called before forward() to fully initialize the backend operation.
Shared mode requires parent to call setExecutionContext() before build().

Behavior:

Stateless: no trainable parameters; parameterCount() returns 0 and save()/load() are minimal but include template metadata so a loader can validate instantiation parameters.
forward() delegates to the backend UnaryOperation. The caller is responsible for providing device-compatible ITensor objects.
backward() computes input gradients for GELU activation (no parameter gradients).

Threading / Synchronization:

Component does not guarantee thread-safety; call synchronize() to wait for outstanding device work to complete when needed.

Constructor & Destructor Documentation

◆ Gelu()

template<DeviceType TDeviceType, TensorDataType TPrecision>

Mila::Dnn::Gelu< TDeviceType, TPrecision >::Gelu	(	const std::string &	name,
		const GeluConfig &	config,
		std::optional< DeviceId >	device_id = std::nullopt )

inlineexplicit

Construct GELU activation component with optional ExecutionContext ownership.

Supports two construction modes:

Standalone mode (device_id provided):

Creates and owns an ExecutionContext for the specified device.
Registers the owned context with the base Component class via setExecutionContext().
Backend operation is created immediately in onExecutionContextSet() hook.
Use case: Unit tests, standalone component usage.

Shared mode (device_id not provided):

Does not create ExecutionContext; expects parent to provide one.
Parent (Network/CompositeComponent) calls setExecutionContext() after construction.
Backend operation created when parent sets context.
Use case: Components added to Network via addComponent<Gelu>(...).

Parameters

name	Component name identifier (mandatory).
config	GELU configuration (approximation method).
device_id	Optional device identifier. If provided, creates owned ExecutionContext for standalone mode. If nullopt, expects shared context from parent.

Exceptions

std::invalid_argument	if config is invalid (via config.validate()).
std::invalid_argument	if device_id.type does not match TDeviceType.
std::runtime_error	if ExecutionContext creation fails (standalone mode).
std::runtime_error	if backend operation creation fails in onExecutionContextSet().

Note: In standalone mode, setExecutionContext() is called to register the owned context with the base class, enabling getExecutionContext() and triggering the onExecutionContextSet() hook for operation creation.

// Standalone mode (owns context)
GeluConfig config;
Gelu<DeviceType::Cpu, TensorDataType::FP32> gelu("gelu", config, Device::Cpu());

// Shared mode (borrows parent's context)
Network<DeviceType::Cpu, TensorDataType::FP32> net(Device::Cpu(), "my_net");
net.addComponent<Gelu>("gelu", GeluConfig());

Member Function Documentation

◆ backward()

template<DeviceType TDeviceType, TensorDataType TPrecision>

TensorType & Mila::Dnn::Gelu< TDeviceType, TPrecision >::backward	(	const TensorType &	input,
		const TensorType &	output_grad )

inline

Compute gradients with respect to the component input.

Delegates to the backend UnaryOperation::backward implementation to compute the gradient of GELU with respect to the input. The component owns the input gradient buffer which is allocated during onBuilding().

The gradient computation follows the chain rule: dL/dinput = dL/doutput * dGELU(input)/dinput

Parameters

input	Const reference to the original forward input.
output_grad	Const reference to the gradient w.r.t. component output (?L/?output).

Returns: Pointer to an ITensor containing the computed input gradient.

Exceptions

std::runtime_error	if component has not been built via build().
std::runtime_error	if component is not in training mode.
std::runtime_error	if operation backend is not initialized.

Note: GELU has no parameters, so no parameter gradients are computed.; The implementation may accumulate into the returned tensor (backend-dependent).; Requires isTrainingMode == true for gradient computation in some backends.

◆ forward()

template<DeviceType TDeviceType, TensorDataType TPrecision>

TensorType & Mila::Dnn::Gelu< TDeviceType, TPrecision >::forward ( const TensorType & input )

inline

Run the forward computation for this GELU component.

Dispatches to the device-specific UnaryOperation backend via a cached output view. The view is pre-initialized at build time and rebuilt only when the input shape changes (e.g. prefill vs decode). Zero heap allocation in steady-state.

Parameters

input Const reference to the input tensor.

Returns: Reference to the cached output view.

Exceptions

std::runtime_error if component has not been built via build().

◆ getApproximationMethod()

template<DeviceType TDeviceType, TensorDataType TPrecision>

ApproximationMethod Mila::Dnn::Gelu< TDeviceType, TPrecision >::getApproximationMethod ( ) const

inline

Return the configured GELU approximation method.

Returns: Configured GeluConfig::ApproximationMethod value (Exact or Tanh).

◆ getDeviceId()

template<DeviceType TDeviceType, TensorDataType TPrecision>

DeviceId Mila::Dnn::Gelu< TDeviceType, TPrecision >::getDeviceId ( ) const

inlineoverridevirtual

Get the device identifier for this component.

Returns the DeviceId from the ExecutionContext. In standalone mode, this is the device specified at construction. In shared mode, this is the parent's device.

Returns: DeviceId indicating device type and index.

Exceptions

std::runtime_error if ExecutionContext has not been set.

Implements Mila::Dnn::Component< TDeviceType, TPrecision >.

◆ getGradients()

template<DeviceType TDeviceType, TensorDataType TPrecision>

std::vector< ITensor * > Mila::Dnn::Gelu< TDeviceType, TPrecision >::getGradients ( ) const

inlineoverridevirtual

Get parameter gradient tensors.

GELU has no trainable parameters, therefore no gradients.

Returns: Empty vector.

Implements Mila::Dnn::Component< TDeviceType, TPrecision >.

◆ getMemoryStats()

template<DeviceType TDeviceType, TensorDataType TPrecision>

MemoryStats Mila::Dnn::Gelu< TDeviceType, TPrecision >::getMemoryStats ( ) const

inlineoverridevirtual

Return memory allocation breakdown.

Implements Mila::Dnn::Component< TDeviceType, TPrecision >.

◆ getParameters()

template<DeviceType TDeviceType, TensorDataType TPrecision>

std::vector< ITensor * > Mila::Dnn::Gelu< TDeviceType, TPrecision >::getParameters ( ) const

inlineoverridevirtual

Get trainable parameter tensors.

GELU has no trainable parameters.

Returns: Empty vector.

Implements Mila::Dnn::Component< TDeviceType, TPrecision >.

◆ getType()

template<DeviceType TDeviceType, TensorDataType TPrecision>

const ComponentType Mila::Dnn::Gelu< TDeviceType, TPrecision >::getType ( ) const

inlineoverridevirtual

Get the component type identifier.

Used for serialization and runtime type identification.

Returns: Component type enum value.

Implements Mila::Dnn::Component< TDeviceType, TPrecision >.

◆ onBuilding()

template<DeviceType TDeviceType, TensorDataType TPrecision>

void Mila::Dnn::Gelu< TDeviceType, TPrecision >::onBuilding ( const BuildContext & build_context )

inlineoverrideprotectedvirtual

Hook invoked during build() to initialize backend operation.

Delegates shape-dependent initialization to the backend UnaryOperation. Must be called before forward() or backward().

State guards:

Expects ExecutionContext to be set (required for operation creation)
Expects operation to be initialized (created in onExecutionContextSet)
Expects component to be unbuilt (guaranteed by Component::build)

Parameters

build_context Build-time context carrying the expected input shape.

Exceptions

std::invalid_argument	if the input shape is incompatible with the component configuration.
std::runtime_error	if backend allocation or build fails.
std::runtime_error	if operation is not initialized.

Reimplemented from Mila::Dnn::Component< TDeviceType, TPrecision >.

◆ onExecutionContextSet()

template<DeviceType TDeviceType, TensorDataType TPrecision>

void Mila::Dnn::Gelu< TDeviceType, TPrecision >::onExecutionContextSet ( )

inlineoverrideprotectedvirtual

Hook invoked after ExecutionContext is set.

Called by Component::setExecutionContext() after the context is registered. Creates the backend UnaryOperation using the OperationRegistry.

This hook is triggered in two scenarios:

Standalone mode: Immediately in constructor after owned context creation
Shared mode: When parent calls setExecutionContext() after construction

State guards:

Expects ExecutionContext to be set (guaranteed by Component::setExecutionContext)
Can be called before build() (operation creation is independent of shape)

Exceptions

std::runtime_error	if operation creation fails.
std::runtime_error	if "GeluOp" is not registered for this device/precision.

Reimplemented from Mila::Dnn::Component< TDeviceType, TPrecision >.

◆ onTrainingModeChanging()

template<DeviceType TDeviceType, TensorDataType TPrecision>

void Mila::Dnn::Gelu< TDeviceType, TPrecision >::onTrainingModeChanging ( TrainingMode training_mode )

inlineoverrideprotectedvirtual

Hook invoked when training mode changes.

Propagates training mode to the backend operation. Called by Component::setTrainingMode() with the training mutex held.

State guards:

Expects operation to be initialized (should be created in onExecutionContextSet)
Can be called before or after build()

Parameters

training_mode New training mode state.

Note: Do not call setTrainingMode() from this hook (reentrancy prohibited).; If operation is not initialized, silently returns (may occur during construction).

Reimplemented from Mila::Dnn::Component< TDeviceType, TPrecision >.

◆ parameterCount()

template<DeviceType TDeviceType, TensorDataType TPrecision>

dim_t Mila::Dnn::Gelu< TDeviceType, TPrecision >::parameterCount ( ) const

inlineoverridevirtual

Number of trainable parameters.

GELU is stateless and exposes no trainable parameters.

Returns: 0

Implements Mila::Dnn::Component< TDeviceType, TPrecision >.

◆ save_()

template<DeviceType TDeviceType, TensorDataType TPrecision>

void Mila::Dnn::Gelu< TDeviceType, TPrecision >::save_	(	ModelArchive &	archive,
		SerializationMode	mode ) const

inlineoverridevirtual

Persist component state to archive.

GELU is stateless (no trainable tensors) but persists:

Component type ("Gelu") and version (1)
Component name from config
Template parameters (device type and precision) for loader validation
Serialized GeluConfig (approximation method)

Files written:

meta.json: Component metadata and template parameters
config.json: GeluConfig serialization

Parameters

archive	Archive to write to.
mode	Serialization mode (currently unused, all state is always saved).

Implements Mila::Dnn::Component< TDeviceType, TPrecision >.

◆ synchronize()

template<DeviceType TDeviceType, TensorDataType TPrecision>

void Mila::Dnn::Gelu< TDeviceType, TPrecision >::synchronize ( )

inlineoverridevirtual

Wait for all asynchronous work submitted by this component to complete.

Synchronizes the underlying ExecutionContext. On CPU implementations this may be a no-op. Use to ensure results are visible on the host or to measure synchronous timings.

Exceptions

std::runtime_error if ExecutionContext has not been set.

Implements Mila::Dnn::Component< TDeviceType, TPrecision >.

◆ toString()

template<DeviceType TDeviceType, TensorDataType TPrecision>

std::string Mila::Dnn::Gelu< TDeviceType, TPrecision >::toString ( ) const

inlineoverridevirtual

Generate human-readable description of the component.

Produces a multi-line string showing:

Component name
Device type
Approximation method

Returns: Formatted string representation.

Implements Mila::Dnn::Component< TDeviceType, TPrecision >.

The documentation for this class was generated from the following file:

Mila/Src/Dnn/Components/Activations/Gelu/Gelu.ixx

Public Types

Public Member Functions

Static Public Member Functions

Protected Member Functions

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ Gelu()

Member Function Documentation

◆ backward()

◆ forward()

◆ getApproximationMethod()

◆ getDeviceId()

◆ getGradients()

◆ getMemoryStats()

◆ getParameters()

◆ getType()

◆ onBuilding()

◆ onExecutionContextSet()

◆ onTrainingModeChanging()

◆ parameterCount()

◆ save_()

◆ synchronize()

◆ toString()