Smoke Test

A smoke test is intentionally non-exhaustive. It executes a minimal set of high-priority checks to verify the service is operational and its critical paths are functional. The goal is not to find deep bugs but to catch blocking failures that would cause a system-wide outage.

• Examples: Can the endpoint be reached? Does it return a valid HTTP 200 status for a simple request? Does the response contain the expected JSON structure? Is basic authentication functioning?

This test is executed immediately after a deployment, before the new version is integrated into the production traffic flow. It acts as a final, automated sanity check in the deployment pipeline, often triggered as part of a Continuous Delivery (CD) stage. A failed smoke test should automatically halt the rollout and trigger a rollback or alert, preventing a broken build from affecting users.

The test validates the minimum viable functionality required for the service to be considered 'alive.' For a machine learning inference endpoint, this typically means:

• Service Health: The container/pod is running and the web server is listening.
• Model Loading: The correct model artifact (weights, vocabulary) is loaded into memory.
• Basic Inference: The model can perform a forward pass on a canonical, sanitized input and produce a non-error output within a reasonable latency bound (e.g., < 500ms).

Speed is critical. Smoke tests must execute in seconds, not minutes, to provide rapid feedback and not delay the deployment process. They are designed to be fully automated and idempotent, integrating seamlessly into CI/CD pipelines (e.g., GitHub Actions, GitLab CI, Jenkins). Their results are binary: pass (proceed to canary/gradual rollout) or fail (initiate rollback).

It's crucial to differentiate a smoke test from other validation stages:

• vs. Unit/Integration Tests: These run before deployment during the CI phase, testing code logic in isolation or with dependencies.
• vs. Load/Stress Tests: These assess performance under high traffic, which is a deeper validation done separately.
• vs. A/B Testing/Canary Analysis: These occur after the smoke test passes and the service is receiving live traffic, comparing business metrics between versions.
• vs. Health Checks: These are continuous, lightweight pings used by load balancers, whereas a smoke test is a one-time event per deployment.

In machine learning systems, smoke tests are often implemented as a suite of API calls to the newly deployed inference endpoint. A robust pattern involves:

1. Synthetic Request: Sending a small batch of valid, representative data from a golden dataset.
2. Output Validation: Checking for a successful HTTP status code, expected schema, and that predictions are within plausible ranges (e.g., not NaN).
3. Dependency Check: Verifying connections to required downstream services (e.g., feature stores, vector databases) are alive. Tools like Postman, pytest with requests, or custom scripts within the deployment orchestration (e.g., Kubernetes Job, Argo Workflows) are commonly used.

Ensures the model correctly adheres to its defined API contract, preventing integration failures.

• Valid Input: Send a request that matches the exact expected schema (data types, field names, tensor shapes). Verify the response schema is also correct.
• Invalid Input Handling: Send a malformed request (e.g., missing a required field, wrong data type like a string where a number is expected). The service should return a descriptive 4xx client error (e.g., 422 Unprocessable Entity) and not a 5xx server error or crash.
• Example: For an object detection model expecting a base64-encoded image in a field called image_data, test with a valid image and with a plain text string. The latter should trigger a clear validation error.

A basic correctness check using a small set of known, static inputs with expected outputs.

• Golden Set Inference: Run inference on 3-5 hand-picked, representative examples where the expected output is known and stable. Compare the model's predictions against these golden outputs. Use a relaxed tolerance for numerical models.
• Consistency Check: Send the same input multiple times in rapid succession. The outputs should be deterministic (or within a very small variance for stochastic models).
• Example: For a sentiment analysis model, input "I love this product!" should return a strongly positive sentiment score. For a regression model predicting house prices, an input with known features should yield a value within 10% of the previously recorded prediction.

Confirms the deployed instance is correctly provisioned and can handle initial load without immediate failure.

• Memory Footprint: After a few inference calls, check the container's memory usage is within expected limits and not leaking.
• Concurrent Requests: Send 5-10 parallel, valid requests to the endpoint. The service should handle them without crashing, and all should return successful responses. This tests thread/worker pool configuration.
• GPU Utilization (if applicable): For GPU-backed models, verify the kernel loads correctly and the GPU is being used for inference, not just sitting idle.
• Example: Use a tool like curl in parallel or a simple Python script with concurrent.futures to simulate minimal concurrent load.

Validates that all external dependencies and runtime configurations are correctly set up in the new environment.

• External Service Calls: If the model calls external services (e.g., a vector database for RAG, a feature store), verify connectivity with a simple query. For a RAG model, this might be a test retrieval call.
• Configuration Loading: Ensure environment variables (API keys, model paths, threshold values) are correctly read. A common test is to have the service log its configuration version or a specific flag on startup.
• Example: For a model using a specific MODEL_VERSION env var, the smoke test can call a /config endpoint (if exposed) or infer from the response metadata that the correct version is active.

Confirms that the deployment is properly instrumented, so its behavior can be monitored once live traffic starts.

• Log Generation: Verify that inference requests generate structured logs in the expected location (e.g., stdout, a logging aggregator). Check for the presence of key fields like request_id, model_version, and latency.
• Metric Emission: Confirm that core metrics are being recorded. After a few test calls, query the metrics backend (e.g., Prometheus) for counters like inference_requests_total or histograms like inference_duration_seconds and ensure they have increased.
• Tracing: If using distributed tracing (e.g., OpenTelemetry), verify a trace is generated for a test request and contains the expected spans for model inference.
• Example: Query Prometheus for rate(inference_requests_total{model_version="v1.2.3"}[1m]) after the smoke test runs to see the activity.

A deployment strategy where a new model version is initially rolled out to a small, specific subset of users or infrastructure to validate its performance and stability before a full rollout.

• Controlled Exposure: Traffic is routed to the new 'canary' version for a limited group (e.g., 5% of users).
• Real-World Validation: Performance metrics (latency, error rate, business KPIs) are monitored in a live environment.
• Progressive Rollout: If metrics are acceptable, traffic is gradually increased; if issues are detected, an immediate rollback is triggered.

A deployment technique where a new model processes live production traffic in parallel with the current model, but its predictions are logged and not used to affect user-facing decisions.

• Zero-Risk Comparison: The 'shadow' model receives a copy of every inference request sent to the primary model.
• Performance Logging: Its outputs, latency, and resource usage are logged for offline analysis against the primary model's results.
• Validation Phase: Used to gather statistical evidence of model behavior under real load and data distribution before any user is exposed to it.

A periodic probe, often an HTTP request, sent to a service (like a model server) to verify its operational status and readiness to handle traffic.

• Liveness Probe: Determines if the service container or process is running.
• Readiness Probe: Verifies the service is fully initialized and can accept requests (e.g., model loaded, dependencies connected).
• Integration Point: Used by orchestration systems (Kubernetes, ECS) and load balancers to automatically route traffic away from unhealthy instances.

A resilience pattern in distributed systems that temporarily stops sending requests to a failing service (like a model endpoint) to prevent cascading failures and allow time for recovery.

• Failure Detection: Monitors for timeouts, connection errors, or a high rate of 5xx HTTP status codes.
• State Machine: Operates in Closed (normal), Open (requests fail-fast), and Half-Open (trial requests) states.
• System Protection: Prevents a single failing model server from exhausting connection pools or threads in upstream services, preserving overall system stability.

A predefined plan and set of automated or manual procedures for reverting a system, such as a model deployment, to a previous known-good state in response to detected failures or regressions.

• Automated Triggers: Can be initiated by alerts from smoke test failures, metric breaches (SLOs), or canary analysis.
• Immutable Artifacts: Relies on versioned model artifacts and infrastructure-as-code to ensure the previous state is perfectly reproducible.
• Key Components: Includes updating load balancer rules, feature flags, or model registry pointers to direct traffic back to the previous version.

The practice of routing a percentage of incoming inference requests to different model versions or endpoints, typically controlled by a load balancer or service mesh.

• Control Mechanism: Enables A/B testing, canary releases, and gradual rollouts by precisely directing user traffic.
• Implementation Tools: Often managed via service meshes (Istio, Linkerd), API gateways, or specialized MLOps platforms.
• Metric-Driven: The split ratios are adjusted based on real-time performance and business metric comparisons between model variants.