Smoke Test

A smoke test is designed for breadth over depth. It executes a minimal set of tests that verify the system's most critical pathways are operational. The goal is not to find all bugs but to answer the binary question: "Is the build fundamentally broken?"

• Examples: Verifying an API server starts and accepts connections, ensuring a core user login flow completes, or confirming a data pipeline's initial ingestion step succeeds.
• Contrast with Unit/Integration Tests: While unit tests validate individual functions in isolation, and integration tests check module interactions, a smoke test validates the integrated system's most basic user-facing functionality.

Speed is a defining characteristic. Smoke tests must execute quickly—often in minutes—to provide immediate feedback to developers after a new build or deployment. This rapid feedback loop is essential in Continuous Integration/Continuous Deployment (CI/CD) pipelines.

• Pipeline Gate: A failed smoke test typically blocks progression to more expensive, time-consuming test suites (like regression or performance tests).
• Resource Efficiency: By catching catastrophic failures early, smoke tests prevent the waste of computational resources and engineering time on deeper testing of a broken build.

Smoke tests are fully automated and produce deterministic pass/fail results. They avoid flakiness by relying on stable, core functionalities and predefined assertions. Manual execution defeats the purpose of providing immediate, reliable feedback in automated pipelines.

• Automation Framework: These tests are typically scripted using the same frameworks as other automated tests (e.g., Pytest, JUnit, Selenium for UI).
• Clear Exit Criteria: Success is not subjective; it is defined by the test suite passing all its assertions without errors or timeouts.

A primary use case is validating a deployment or new build in a production-like environment (e.g., staging, pre-prod). After code is deployed, a smoke test suite runs against the live environment to ensure the deployment was successful and the application is responsive.

• Health Check Plus: It goes beyond a simple endpoint health check by validating a sequence of key business logic steps.
• Foundation for Canary Releases: In canary deployment strategies, smoke tests are run against the canary instance before traffic is routed to it, serving as a final sanity check.

While often used interchangeably, smoke testing and sanity testing have nuanced differences in scope and intent.

• Smoke Testing: "Build verification testing." Answers: Did the build work? It's performed on a new build to reject a broken application.
• Sanity Testing: "Narrow regression testing." Answers: Did our specific change work as expected? It's performed on a stable build after a minor change or bug fix to ensure no new issues were introduced in that specific area.

In practice, smoke tests are a subset of the broader regression suite, focused on the most critical paths.

In the context of autonomous agents and recursive error correction, smoke tests act as a first-layer guardrail in an agent's self-evaluation loop. Before an agent proceeds with complex, multi-step reasoning or tool calls, it can run an internal smoke test on its proposed plan or initial output.

• Pre-execution Validation: An agent might verify that required APIs are reachable or that generated code compiles before attempting full execution.
• Circuit Breaker: A failed smoke test can trigger a rollback strategy or corrective action planning, preventing the agent from cascading into a faulty state. This aligns with fault-tolerant agent design principles.

This foundational smoke test verifies an autonomous agent can successfully boot, load its core models, and connect to essential dependencies. It's the first gate before any complex task execution.

Key checks include:

• Model server connectivity and API latency < 1 second.
• Vector database or knowledge graph connection status.
• Availability of critical external tools and APIs.
• Basic memory (context window) allocation.

Failure here indicates a fundamental infrastructure or configuration issue, halting further deployment.

Validates that an agent's primary tool-calling mechanisms—such as those defined by the Model Context Protocol (MCP)—are functional. This ensures the agent can interact with the external world.

Example test flow:

1. Instruct the agent to perform a simple, non-destructive API call (e.g., GET current weather for a known city).
2. Verify the tool is correctly selected from the agent's registry.
3. Confirm the API request is properly formatted and executed.
4. Validate that the agent can parse the successful response.

A pass confirms the agent's action-taking apparatus is online.

A smoke test for the agent's cognitive architecture. It confirms the agent can complete a simple, multi-step plan that requires basic reasoning, without assessing the quality of the output.

Sample test prompt: "Count the number of words in the following sentence and then provide that number squared: 'The quick brown fox.'"

Pass/Fail Criteria:

• PASS: Agent outputs a final numerical answer (e.g., 16). It demonstrates decomposition, tool use (counting), and arithmetic.
• FAIL: Agent crashes, gets stuck in a loop, outputs irrelevant text, or fails to execute a step.

This test probes the stability of the agent's plan-act-observe cycle.

Checks that the agent adheres to basic output schemas and does not violate primary safety guardrails on a trivial task. This is about structure and boundaries, not content sophistication.

Tests include:

• Instructing the agent to "Return a JSON object with keys name and id" and validating the schema.
• Providing a prompt that gently edges towards a restricted topic and verifying the agent uses its refusal mechanism correctly.
• Ensuring the agent's response stays within a specified token limit for a simple query.

Failure indicates a breakdown in prompt adherence or basic safety containment.

In a multi-agent system, the smoke test verifies basic inter-agent communication. It confirms that Agent A can send a message and Agent B can receive and acknowledge it.

Simple orchestration test:

1. A Planner agent is given a trivial task.
2. It must delegate a single subtask to a Worker agent.
3. The test validates that a delegation message was sent via the correct channel (e.g., pub/sub, direct API).
4. It confirms the Worker agent received the task and emitted a 'received' signal.

This does not test the quality of the work done, only the integrity of the communication pathway—a critical circuit breaker for complex orchestrations.

Verifies the agent's basic short-term memory functionality. It checks if the agent can retain and reference information from earlier in the same session.

Procedure:

1. Provide the agent with a simple fact: "My favorite color is azure."
2. In the next user turn, ask: "What is my favorite color?"
3. The test passes if the agent correctly recalls "azure."

Why it's a smoke test: It tests the fundamental ability to maintain state across turns. If this fails, more advanced context management or retrieval-augmented generation (RAG) tests are pointless. It often catches issues with context window trimming or session management.

A deployment technique where a new model or system processes live production traffic in parallel with the incumbent system, but its outputs are not used to affect user decisions. It's a validation step beyond synthetic testing.

• Key Benefit: Provides a zero-risk environment to gather performance data (latency, accuracy) and detect potential failures against real-world inputs.
• Comparison to Smoke Test: A smoke test is a quick, shallow check; shadow mode is a deep, ongoing validation in production.
• Use Case: Running a new LLM-based classifier alongside the current rule-based system to compare classification results and confidence scores on every real user query.