Offline/Online Consistency

The non-negotiable requirement that every data transformation applied during training is replicated exactly in the serving pipeline.

• Identical code: Use shared libraries or feature definitions, not rewritten logic
• Same aggregation windows: If training uses 7-day rolling averages, serving must compute the same window
• Consistent handling of nulls: Missing value imputation must match between offline and online paths

A single discrepancy—such as a different scaling factor or timestamp truncation—introduces silent prediction errors that no monitoring dashboard will catch until business metrics degrade.

Training datasets must be constructed with temporal fidelity to prevent future information from leaking into historical feature values.

• Time travel queries: Feature values are retrieved as they existed at each historical timestamp, not current values
• No lookahead bias: A user's label at time T must only use features computed before time T
• Consistent timestamp semantics: The same event-time processing logic applies in both training data generation and online serving

Without point-in-time joins, a model trained on 'future' feature values will exhibit inflated offline metrics that collapse in production.

The preprocessing graph—tokenization, normalization, one-hot encoding, embedding lookups—must be serialized with the model artifact and executed identically at inference.

• TensorFlow Transform / scikit-learn Pipelines: Bundle preprocessing with the model graph
• Vocabulary consistency: Token-to-index mappings generated during training must be frozen and deployed
• Numerical scaling: Min-max ranges, z-score means, and standard deviations are training-set statistics that become serving constants

Embedding preprocessing directly into the model artifact ensures that no manual translation step can introduce skew between environments.

Even with architectural safeguards, runtime validation is essential to catch drift in feature distributions between training and serving.

• Training-serving skew metrics: Compare summary statistics (mean, variance, min/max) of features at training time vs. inference time
• Population Stability Index (PSI): Quantifies distribution shift for categorical and binned continuous features
• Feature freshness alerts: Monitor the lag between feature computation time and inference time to catch stale values

These checks act as a circuit breaker, alerting teams before degraded predictions impact downstream business outcomes.

A trained model must be packaged as an immutable, self-contained artifact that includes all dependencies required for inference.

• Containerized serving: Docker images with pinned library versions eliminate environment drift
• Serialized preprocessing: All transformations embedded in the model file (e.g., SavedModel with serving signatures)
• Versioned feature definitions: The exact feature set used at training is recorded in the model registry

This immutability guarantees that a model promoted to production will execute the same mathematical function that was validated during offline evaluation.

The primary failure mode that offline/online consistency prevents. Training-serving skew occurs when the code path or data transformation logic used during model training diverges from the logic used during inference. This creates a silent performance degradation where the model appears accurate in evaluation but fails in production because it processes features differently than it was trained to expect. Common causes include hardcoded preprocessing in training notebooks that isn't replicated in the serving layer, or timestamp-dependent features computed at different cadences.

A critical metric measuring the temporal gap between when a feature value is computed and when it is consumed for inference. Stale features are a direct cause of inconsistency—a model trained on daily-aggregated features will underperform if the serving stack computes them hourly with different aggregation windows. Maintaining consistency requires aligning feature refresh cadences across training and serving pipelines, and implementing time-to-live (TTL) policies that invalidate features exceeding their freshness threshold.

A discipline that treats feature transformations as version-controlled, testable software artifacts rather than ad-hoc notebook cells. By defining features in a shared library or DSL that compiles to both batch processing (Spark, SQL) and real-time serving (REST, gRPC) implementations, teams eliminate the dual-implementation problem. Tools like dbt for data transformation and Feast for feature serving enforce this pattern, ensuring the exact same logic executes in both contexts.

Continuous observation of production model behavior that detects when offline/online consistency has broken. Model monitoring systems track:

• Prediction drift: Changes in the distribution of model outputs
• Feature attribution shifts: Which features drive predictions differently than during training
• Input-output correlation decay: Weakening relationships between features and predictions When monitoring detects these signals, it triggers investigation into whether a consistency gap—not genuine concept drift—is the root cause.