Long Short-Term Memory (LSTM)

The core innovation that solves the vanishing gradient problem. The CEC is a linear self-connected unit that maintains a constant error flow across time steps without decay.

• Mechanism: A recurrent connection with a fixed weight of 1.0, allowing gradient information to flow unchanged
• Purpose: Provides short-term memory storage that persists across hundreds of time steps
• Analogy: A conveyor belt moving information through time without degradation
• Without CEC: Standard RNNs exponentially lose gradient signal after ~10 time steps
• With CEC: LSTMs can learn dependencies spanning 1,000+ time steps in network traffic sequences

A sigmoid-activated neural layer that decides what information to discard from the cell state. It examines the previous hidden state and current input, outputting a value between 0 and 1 for each number in the cell state.

• Output of 1: Completely retain this information
• Output of 0: Completely forget this information
• Formula: f_t = σ(W_f · [h_{t-1}, x_t] + b_f)
• Network telemetry example: Forgetting outdated signal strength measurements when a user moves to a new cell
• Practical impact: Prevents saturation of memory with irrelevant historical data, critical for detecting concept drift in KPI streams

A two-part mechanism controlling what new information is stored in the cell state. This gate works in tandem with a candidate generation layer to selectively update memory.

• Input Gate (sigmoid): Decides which values to update: i_t = σ(W_i · [h_{t-1}, x_t] + b_i)
• Candidate Layer (tanh): Creates new candidate values: C̃_t = tanh(W_C · [h_{t-1}, x_t] + b_C)
• Combined operation: Cell state update = (forget gate × old state) + (input gate × candidate values)
• Anomaly detection relevance: Allows the model to store new baseline patterns when network behavior legitimately shifts, preventing false positives from data drift

The final gating mechanism that determines what part of the cell state is exposed as the hidden output for the current time step. It filters the cell state through a tanh activation and multiplies by a sigmoid gate.

• Formula: o_t = σ(W_o · [h_{t-1}, x_t] + b_o)
• Hidden state: h_t = o_t × tanh(C_t)
• Function: Decouples memory storage from memory retrieval
• Telemetry forecasting: The output gate might suppress cell state information irrelevant to predicting the next KPI value while exposing trend components
• Stacking: In multi-layer LSTMs, this hidden state becomes the input to the next layer, enabling hierarchical feature extraction from raw performance management counters

An architectural enhancement where the gates are given direct access to the cell state. Standard LSTM gates see only the previous hidden state and current input; peephole connections add the cell state as a third input.

• Forget gate with peephole: f_t = σ(W_f · [C_{t-1}, h_{t-1}, x_t] + b_f)
• Input gate with peephole: i_t = σ(W_i · [C_{t-1}, h_{t-1}, x_t] + b_i)
• Output gate with peephole: o_t = σ(W_o · [C_t, h_{t-1}, x_t] + b_o)
• Benefit: Gates can make more informed decisions by inspecting the actual memory content
• Use case: Precise timing tasks like change point detection in seasonal network load patterns, where knowing the exact cell state improves gating precision

A dual-pass configuration that processes sequences in both forward and backward directions, combining two independent LSTM layers to capture context from past and future time steps simultaneously.

• Forward pass: Standard LSTM processing from t=1 to t=T
• Backward pass: Reverse LSTM processing from t=T to t=1
• Concatenation: Final hidden state = [h_forward, h_backward]
• Network telemetry advantage: Anomaly detection benefits from seeing both preceding baselines and subsequent recovery patterns
• Trade-off: Requires the full sequence before inference, adding latency unsuitable for real-time gRPC streaming telemetry but ideal for offline root cause analysis of historical performance management counter data

The mechanism that decides which information to discard from the cell state. It examines the previous hidden state and current input, outputting a value between 0 and 1 for each number in the cell state. A value of 1 means 'completely keep this,' while 0 means 'completely forget this.' In network telemetry, this allows the model to flush irrelevant historical noise—like a brief, resolved latency spike—while retaining patterns indicative of a developing fault.

The long-term memory highway running through the entire chain of LSTM cells. Information flows linearly across time steps with only minor linear interactions, allowing gradients to flow unchanged. This architecture directly solves the vanishing gradient problem that plagues standard RNNs. For anomaly detection, the cell state preserves the signature of normal network behavior over thousands of time steps, enabling comparison against sudden deviations.

A two-part mechanism that controls what new information is stored in the cell state. A sigmoid layer decides which values to update, while a tanh layer creates a vector of new candidate values. Together, they scale how much each piece of new information should be added to memory. In a RAN context, this gate learns to selectively store critical telemetry shifts—such as a rising bit error rate—while ignoring transient fluctuations.

Determines the next hidden state based on a filtered version of the cell state. The gate applies a sigmoid function to decide which parts of the cell state to output, then multiplies it by a tanh of the cell state to push values between -1 and 1. This produces the hidden state used for prediction. For time-series forecasting, this gate generates the actual anomaly score or predicted KPI value at each time step.

A fundamental training obstacle where gradients shrink exponentially as they are backpropagated through time, preventing standard RNNs from learning dependencies longer than ~10 time steps. LSTM's Constant Error Carousel (CEC) in the cell state provides an uninterrupted gradient highway. This is critical for detecting collective anomalies in network traffic—patterns that unfold over minutes or hours, not seconds.

An architecture that runs two independent LSTM layers: one processing the sequence forward and another backward. The outputs are concatenated, giving the model context from both past and future states at each time point. For offline network forensics, this allows anomaly detection to consider both the precursor events leading to a fault and the subsequent recovery pattern, dramatically improving root cause analysis accuracy.