Unit Testing

A unit test must verify a single unit of code—typically a function, method, or class—in complete isolation from its dependencies. This is achieved using test doubles like mocks, stubs, and fakes to replace external systems (e.g., databases, APIs, file systems).

• Purpose: Ensures test failures point directly to a bug in the unit under test, not in a connected service.
• Key Technique: Dependency Injection to allow substituting real implementations with controlled doubles.
• Example: Testing a processOrder function by mocking the paymentGateway and inventoryService dependencies.

A unit test must be deterministic, producing the same pass/fail result every time it runs, given the same code and environment. Non-deterministic tests ("flaky tests") erode trust in the test suite.

• Causes of Flakiness: Unmanaged external state, reliance on system time (DateTime.Now), random number generation, or concurrent execution.
• Solution: Use fixed, predictable test data and control all inputs. For time, inject a time provider interface. For randomness, use a seeded random generator.
• Impact: Essential for Continuous Integration (CI) pipelines, where a false failure can block deployment.

The unit test suite must execute extremely quickly to provide immediate feedback to developers. Slow tests discourage frequent execution, breaking the Test-Driven Development (TDD) rhythm.

• Benchmark: A suite of thousands of unit tests should run in seconds, not minutes.
• Speed Enablers:
  • Isolation: Avoiding real network calls (APIs) or disk I/O.
  • Parallelization: Running independent tests concurrently.
• Consequence: Fast tests enable them to be run on every code change, a cornerstone of shift-left testing.

FIRST is a mnemonic summarizing the core attributes of effective unit tests.

• Fast: Tests run quickly.
• Isolated: A test does not depend on or affect the outcome of another test. It uses fresh fixtures and cleans up state.
• Repeatable: Tests are deterministic, as described above.
• Self-Validating: The test automatically passes or fails; no manual interpretation of logs or outputs is required.
• Timely: Ideally written before the production code (following TDD), ensuring the code is testable by design.

Arrange-Act-Assert (AAA) is the universal structural pattern for organizing unit test code, enhancing clarity and maintainability.

• Arrange: Set up all necessary preconditions and inputs. Instantiate the object under test and configure its dependencies (mocks).
• Act: Invoke the single method or function you are testing.
• Assert: Verify that the expected outcome occurred. Check return values, state changes, or that specific dependencies were called correctly.

Example:

pythonCopy
# Arrange
mock_api = Mock(ExternalAPI)
mock_api.fetch_data.return_value = {'status': 'ok'}
processor = DataProcessor(mock_api)

# Act
result = processor.run()

# Assert
assert result is True
mock_api.fetch_data.assert_called_once()

Unit tests should validate the public contract and observable behavior of a unit, not its private implementation details. Tests tied to implementation become brittle and require frequent updates for harmless refactoring.

• Test Public API: Verify outputs and state changes resulting from inputs.
• Avoid Over-Specification: Don't assert that a private helper method was called in a specific sequence. This locks in the how instead of the what.
• Refactoring Safety: A good test suite allows you to confidently change how a feature works internally (refactor) as long as the external behavior remains correct.

A test double is a generic term for any object or component used in place of a real system dependency during testing. It isolates the unit under test from external systems like databases, APIs, or services, allowing for fast, deterministic, and focused validation of logic.

Common types include:

• Stubs: Provide canned answers to calls made during the test.
• Mocks: Pre-programmed with expectations about calls they will receive; they fail the test if these expectations are not met.
• Fakes: Have working implementations but are simplified (e.g., an in-memory database).
• Spies: Record information about how they were called for later verification.

In API testing, a mock HTTP client is a critical test double that simulates API responses without making network calls.

Integration testing is a software testing level where individual software modules are combined and tested as a group to verify they interact correctly according to their defined interfaces. It sits above unit testing in the test pyramid.

For API-driven systems, this involves:

• Testing the interaction between an AI agent's tool-calling logic and a real or simulated API endpoint.
• Validating that data flows correctly between the orchestration layer, the external system connector, and the backend service.
• Ensuring authentication tokens are passed correctly and response schemas are properly parsed.

Unlike unit tests that use test doubles, integration tests often rely on API mocking or lightweight versions of real services.

Test-Driven Development (TDD) is a software development methodology where developers write automated tests for a desired feature before writing the minimal amount of code necessary to pass those tests. The cycle is Red-Green-Refactor: write a failing test (Red), write code to pass it (Green), then improve the code structure (Refactor).

Applied to tool-calling agents, TDD would involve:

1. Writing a unit test that defines the expected structured output (e.g., a correctly formatted API call object) for a given natural language instruction.
2. Implementing the agent's prompt architecture or function-calling logic to produce that output.
3. Refactoring the prompt or code for clarity and efficiency.

This ensures the agent's core capability to generate correct API calls is verifiable from the start.

Contract testing is a methodology that validates the interactions between two separate services (e.g., a consumer and a provider) by verifying that requests and responses conform to a shared, agreed-upon specification or 'contract'. It is crucial for microservices and API integrations.

In an AI agent context:

• The consumer is the AI agent generating API calls.
• The provider is the external service (REST API, database, etc.).
• The contract is often an OpenAPI Schema or JSON Schema.

Tools like Pact or Spring Cloud Contract allow you to test that the agent's generated requests match the provider's expected format, and that the agent can handle the provider's documented responses, without needing the live provider during testing.

Test coverage is a quantitative metric that measures the amount of a software system's code, branches, paths, or requirements that are exercised by a test suite. It indicates the comprehensiveness of the testing effort.

Key coverage types relevant to unit testing AI agents:

• Line Coverage: Percentage of code lines executed by tests.
• Branch Coverage: Percentage of decision branches (e.g., if/else statements) taken.
• Path Coverage: Percentage of unique paths through a function executed.

For a tool-calling function, high branch coverage would ensure tests validate both successful API call generation and error-handling logic for malformed user inputs. While high coverage doesn't guarantee bug-free code, low coverage almost certainly indicates untested and risky code paths.

Mutation testing is a fault-based testing technique that evaluates the quality of a test suite by introducing small, deliberate faults (mutants) into the source code and checking if the existing tests can detect them. A mutant is 'killed' if a test fails; it 'survives' if all tests still pass, indicating a test gap.

Common mutation operators include changing arithmetic operators, altering logical conditions, or removing statements.

For unit tests of API request builders or response parsers, mutation testing is powerful. If a mutant that changes a query parameter key survives, it reveals that the test suite does not adequately validate the correctness of the generated request structure, potentially allowing broken calls to go undetected.