Online Learning

The model processes data one observation at a time in a sequential stream, updating its parameters incrementally after each example. This contrasts with batch learning, which requires the entire training dataset to be available upfront.

• Each data point is used once and then discarded, making it memory-efficient for unbounded streams
• Updates occur via Online Gradient Descent, computing the gradient of the loss for a single sample
• Enables learning from never-ending data streams like click logs, sensor telemetry, or financial tick data

Online learning inherently addresses concept drift by continuously incorporating the most recent data into the model. As the statistical relationship between inputs and targets shifts, the model's parameters adjust automatically.

• Eliminates the staleness window between batch retraining cycles
• Critical for environments where consumer behavior, market conditions, or fraud patterns evolve rapidly
• Requires drift detection mechanisms to distinguish between genuine shifts and transient noise

In online decision-making contexts like recommendation and dynamic pricing, the model must balance exploitation (choosing the known best action) with exploration (trying new actions to gather data).

• Contextual Multi-Armed Bandits formalize this trade-off mathematically
• Pure exploitation leads to feedback-loop bias, where the model only reinforces past choices
• Algorithms like Thompson Sampling and Upper Confidence Bound provide principled exploration strategies

A critical challenge in online learning is that the true label or outcome is often not immediately available after a prediction. For example, a product recommendation's success may only be known days later when a purchase occurs.

• Requires reward attribution windows to associate delayed outcomes with past predictions
• Partial feedback scenarios occur when only outcomes for chosen actions are observed
• Techniques like importance-weighted sampling correct for the bias introduced by delayed or partial feedback

As the model updates on new data, it risks catastrophic forgetting—abruptly losing previously learned patterns. This is especially acute in neural network-based online learners.

• Experience Replay stores past examples in a buffer and interleaves them with new data during updates
• Elastic Weight Consolidation penalizes large changes to parameters important for previous tasks
• Sliding window approaches retain a fixed-size recent history to balance recency with stability

Online learning operates under strict latency budgets since updates must complete before the next sample arrives. This imposes unique architectural requirements.

• Model updates must be O(1) or O(n) per sample, not scaling with dataset size
• Feature freshness is critical—stale features cause training-serving skew
• Often deployed alongside feature stores that serve pre-computed and real-time features with sub-millisecond latency

The statistical properties of the target variable change over time, breaking the model's core assumptions. Online learning is the primary countermeasure.

• Sudden drift: An abrupt shift (e.g., a Black Friday sales spike)
• Incremental drift: Gradual change (e.g., evolving user preferences)
• Recurring drift: Cyclical patterns (e.g., seasonal trends)

Without detection and adaptation, a model suffering from concept drift becomes increasingly inaccurate.

The mathematical engine of online learning. Unlike batch gradient descent, OGD updates model parameters after each individual sample arrives.

• Computes the gradient of the loss for a single data point
• Updates weights immediately: w_{t+1} = w_t - η∇ℓ(w_t, (x_t, y_t))
• Enables sub-linear regret bounds in adversarial settings

This per-sample update loop is what allows models to adapt in real-time without storing the entire dataset.

The Achilles' heel of continuous learning. When a neural network learns new patterns, it can abruptly overwrite previously acquired knowledge.

• Cause: Shared weights shift to optimize for recent data
• Severity: Particularly acute in deep networks trained with SGD
• Mitigations:
  • Experience Replay: Interleave old samples with new data
  • Elastic Weight Consolidation: Penalize changes to important weights
  • Progressive Networks: Freeze old subnetworks, add new capacity

Solving this is critical for any online learning system that must retain long-term knowledge.

The fundamental dilemma in online decision-making. A model must balance exploiting known high-reward actions against exploring uncertain alternatives.

• Exploitation: Serve the best-known recommendation to maximize immediate reward
• Exploration: Try a new action to gather data that may improve future decisions
• Contextual Bandits: Algorithms like Thompson Sampling and Upper Confidence Bound formalize this trade-off

Pure exploitation leads to stagnation; pure exploration sacrifices short-term performance. Online learning systems must dynamically navigate this balance.

A measure of the temporal gap between when a feature value is computed and when it is used for inference. Stale features are a primary cause of training-serving skew.

• Real-time features: Computed at request time (e.g., clicks in current session)
• Near-real-time features: Updated within seconds (e.g., trending product velocity)
• Batch features: Refreshed hourly or daily (e.g., user lifetime value)

Online learning demands that feature pipelines deliver values with latency measured in milliseconds, not hours.

A critical challenge where the true label for a prediction is not known until significantly later. The model must update without immediate ground truth.

• Example: A product recommendation's conversion may occur days after the click
• Impact: Naive online updates may use incorrect labels, degrading performance
• Solutions:
  • Importance weighting: Correct for the delay distribution
  • Surrogate models: Predict the eventual outcome using intermediate signals
  • Dual-window approaches: Separate models for immediate and delayed feedback

Handling delayed feedback is essential for online learning in e-commerce and advertising.