Weight Initialization

Draws weights from a distribution scaled by the number of input and output units in a layer. Designed to maintain consistent variance of activations and gradients across layers when using sigmoid or tanh activation functions.

• Variance scales as 2 / (n_in + n_out) for uniform distribution
• Prevents signal from becoming too small or too large during forward and backward passes
• Default initialization in many frameworks for shallow to medium-depth networks
• Critical for RVTDNN predistorter architectures using tanh nonlinearities

Scales initial weights based solely on the number of input connections, optimized for layers followed by ReLU and its variants. Accounts for the fact that ReLU zeros out half the activations, requiring larger initial weights to compensate.

• Variance scales as 2 / n_in for uniform distribution
• Essential for deep residual networks and predistorters with ReLU activations
• Prevents dying ReLU problem in early training stages
• Standard choice for CVNN architectures using complex ReLU variants

Proposed by Yann LeCun, this method scales weights by 1 / n_in, designed for networks using sigmoid activation functions in the early days of deep learning. Produces smaller initial weights than Xavier or He methods.

• Variance scales as 1 / n_in
• Assumes linear regime of sigmoid around zero
• Less common in modern predistorter designs but foundational to the field
• Useful as a baseline when comparing initialization strategies for behavioral modeling

Initializes weight matrices as orthogonal or near-orthogonal matrices, preserving the norm of signals as they propagate through the network. Particularly effective for recurrent neural networks and deep linear networks.

• Uses QR or SVD decomposition to generate orthogonal matrices
• Exactly preserves gradient magnitude in linear networks
• Helps mitigate vanishing/exploding gradients in BPTT for recurrent predistorters
• Can be combined with gain scaling for nonlinear activations

Initializing all weights to zero causes every neuron in a layer to compute the same output and receive identical gradients, preventing symmetry breaking and learning. Random initialization breaks this symmetry.

• Zero initialization: neurons remain identical, network fails to learn diverse features
• Small random values: breaks symmetry but may cause vanishing gradients in deep networks
• Large random values: may cause exploding gradients and saturated activations
• Biases can safely be initialized to zero regardless of weight initialization strategy

Initializes a predistorter network using weights from a model previously trained on a different power amplifier or signal condition. Leverages learned representations to accelerate convergence on new PA devices.

• Reduces required training data by 50-80% for new PA characterization
• Lower layers capture generic nonlinear dynamics; upper layers adapt to device-specific behavior
• Common in model extraction workflows across PA device variants
• Requires careful handling of input/output layer dimensions when architectures differ

A weight initialization scheme designed to maintain consistent variance of activations and gradients across layers. Weights are sampled from a distribution with variance scaled by the fan-in and fan-out of each layer.

• Uniform variant: ( W \sim U\left[-\frac{\sqrt{6}}{\sqrt{n_{in} + n_{out}}}, \frac{\sqrt{6}}{\sqrt{n_{in} + n_{out}}}\right] )
• Normal variant: ( W \sim \mathcal{N}\left(0, \frac{2}{n_{in} + n_{out}}\right) )
• Best suited for layers with symmetric activation functions like tanh and sigmoid
• Prevents the exponential growth or decay of forward-propagated signals in deep RVTDNN architectures

An initialization strategy optimized for ReLU and its variants (Leaky ReLU, PReLU). It scales weights by ( \sqrt{2 / n_{in}} ) to compensate for the fact that ReLU zeros out half of the activations.

• Formula: ( W \sim \mathcal{N}\left(0, \sqrt{\frac{2}{n_{in}}}\right) )
• Preserves variance when activations are asymmetric and non-negative
• Critical for deep cascade forward neural networks used in PA linearization
• Empirically outperforms Xavier when hidden layers use ReLU activation in predistorter models

A failure mode in deep networks where the gradient signal diminishes exponentially as it backpropagates from the output layer to earlier layers. This prevents weight updates in the initial layers.

• Caused by activation functions that squash inputs into small output ranges (sigmoid, tanh)
• Early layers learn extremely slowly or not at all
• Poor weight initialization amplifies this effect in deep predistorter networks
• Mitigated by ReLU activations, batch normalization, and residual connections

The opposite of vanishing gradients: error gradients accumulate multiplicatively during backpropagation, resulting in exponentially large weight updates.

• Manifests as NaN losses or wildly oscillating training curves
• Common in recurrent neural network predistorters trained with BPTT
• Improper initialization with variance too high is a primary trigger
• Solutions include gradient clipping and careful scaling of initial weight magnitudes

A technique where weight matrices are initialized as orthogonal or near-orthogonal matrices, preserving the norm of signals during forward and backward propagation.

• Achieved via QR decomposition or SVD of random matrices
• Exactly preserves norm stability in linear layers
• Particularly effective for very deep networks and recurrent architectures
• Used in complex-valued neural networks (CVNNs) to maintain phase relationships in I/Q signal paths

A layer inserted between the linear transformation and activation that normalizes activations to zero mean and unit variance across each mini-batch.

• Reduces sensitivity to initial weight scales
• Allows higher learning rates and faster convergence
• Acts as a regularizer, reducing the need for dropout
• Standard practice in deep predistorter networks to stabilize training of memory polynomial neural network models