Prompt Unit Test

The test input is the complete prompt template, including any static instructions and the specific test case variables that are injected for execution. This defines the exact query sent to the model.

• Static Prompt: The unchanging instruction text and context.
• Test Variables: The specific values (e.g., {"user_query": "Summarize this contract"}) that populate placeholders in the template.
• Example: For a summarization prompt, the test input combines the instruction "Summarize the following text:" with the variable {"text": "[Actual contract text]"}.

The expected output is the definitive, correct response the model should generate for the given test input. It serves as the golden standard against which the actual model output is compared.

• Can be an exact string match, a JSON object, or a set of semantic criteria (e.g., "must contain the terms X and Y").
• For non-deterministic models, assertions may check for key entities, logical consistency, or adherence to a JSON schema rather than a literal match.
• This component is critical for the Golden Set Evaluation methodology.

The evaluation metric is the algorithm or rule that compares the model's actual output to the expected output, producing a pass/fail result. The assertion is the specific condition that must be met.

• Exact Match: Simple string equality; used for Deterministic Output Tests (temperature=0).
• Schema Validation: Checks if structured output (e.g., JSON) conforms to a defined schema.
• Semantic Similarity: Uses embeddings (e.g., cosine similarity) to assess meaning, enabling Semantic Invariance Tests.
• Rule-Based Check: Verifies the presence/absence of keywords or performs regex matching.

This component defines the execution environment for the test, specifying which model and inference parameters are used. Consistency here is essential for reproducible tests.

• Model Identifier: The specific model version (e.g., gpt-4-turbo-2024-04-09).
• Inference Parameters: Fixed settings like temperature=0, top_p=1, and a controlled seed for Stochastic Seed Control.
• System Prompt: Any high-level, session-defining instructions provided separately from the test input.
• Changes to this context require re-baselining expected outputs.

The integration hook is the code or service call that executes the test within a Prompt CI/CD Pipeline. It connects the test definition to the live model API or inference endpoint.

• Handles authentication, API request formatting, and response parsing.
• Manages concurrency and error handling for batch test execution.
• Outputs structured results (pass/fail, latency, tokens used) to a Prompt Monitoring Dashboard.
• This component enables the test to be automated as part of a Regression Test Suite.

Metadata tags and classifies the test for organization, reporting, and integration into larger test suites. It provides context for result analysis.

• Test ID & Description: Unique identifier and human-readable purpose.
• Tags: Labels like "deterministic", "security", "summary", or "high-cost.
• Related Requirements: Links to functional specs or user stories.
• Performance Baselines: Records of historical latency and token usage for Latency Under Load and cost analysis.
• This data is crucial for filtering tests in an Adversarial Test Suite or A/B Testing campaign.

Verifies a language model produces identical outputs for identical inputs when configured with deterministic sampling parameters (e.g., temperature=0, seed=42). This is foundational for reproducible testing and debugging.

• Purpose: Ensures prompt changes do not introduce non-deterministic noise.
• Example: A prompt asking for a JSON object with a specific structure must return the exact same string on every run.

Automatically verifies that a model's structured output conforms to a predefined JSON Schema, ensuring correct data types, required fields, and value constraints.

• Purpose: Guarantees downstream systems can parse and use the output without errors.
• Implementation: Uses libraries like jsonschema to validate the model's response against the schema, failing the test on any mismatch.

Evaluates whether a model's output remains semantically unchanged when the input prompt is rephrased while preserving its core meaning. This tests for robustness against natural language variation.

• Purpose: Ensures the prompt logic is tied to intent, not specific phrasing.
• Method: Uses embedding similarity (e.g., cosine similarity) or entailment models to compare outputs from varied prompts like "Summarize this text" and "Provide a brief overview of this document."

A quantitative metric that measures how well a model's output follows the specific directives and constraints outlined in its system or user prompt.

• Purpose: Validates that the model follows complex instructions (e.g., "respond in French," "use bullet points," "do not mention X").
• Measurement: Can be automated using a secondary LLM-as-a-judge with a scoring rubric or via rule-based checks for explicit constraints.

Measures and investigates how often a language model declines to answer a query, typically to understand the behavior of safety or content filters.

• Purpose: Monitors the balance between helpfulness and safety guardrails.
• Test Suite: Includes a set of benign queries that should be answered and policy-violating queries that should be refused. Drifts in refusal rates can indicate model or prompt degradation.

Compares a model's outputs against a curated, high-quality dataset of expected or ideal responses for a given set of test inputs.

• Purpose: Provides a ground-truth benchmark for accuracy and quality.
• Process: For each input in the golden set, the model's output is compared to the expected output using string matching, semantic similarity, or key fact extraction. This is a core test for regression detection.

A collection of deliberately crafted or perturbed inputs designed to evaluate a language model's robustness against malicious or unexpected prompts. This suite is critical for security and safety testing.

• Purpose: To discover vulnerabilities like jailbreak attempts or prompt injections.
• Content: Includes edge cases, contradictory instructions, and role-playing scenarios designed to bypass safety filters.
• Outcome: Measures a model's refusal rate and adherence to safety guidelines under pressure.

An evaluation method that compares a model's outputs against a curated, high-quality dataset of expected or ideal responses for a given set of test inputs.

• Core Function: Serves as the ground truth for measuring factual accuracy and instruction adherence.
• Creation: Requires significant domain expertise to assemble correct, comprehensive, and unambiguous reference answers.
• Use Case: The benchmark for calculating automated evaluation metrics in a regression test suite.

An automated software development workflow for continuously integrating, testing, and deploying prompt changes to production environments. It brings engineering rigor to prompt management.

• Components: Includes version control, automated prompt unit tests, canary deployments, and rollback mechanisms.
• Goal: To enable rapid, safe iteration on prompts while maintaining system stability and monitoring for latency under load or performance regression.
• Benefit: Ensures prompt changes are as reliable as code changes.

A test that evaluates whether a model's output remains semantically unchanged when the input prompt is rephrased while preserving its core meaning. This is a key measure of prompt robustness.

• Method: Presents the model with multiple phrasings of the same instruction (e.g., "Summarize this article," "Provide a summary of this text").
• Metric: Assesses output consistency across variations, ensuring the prompt is not brittle to minor wording changes.
• Importance: Directly impacts user experience, as different users will phrase requests differently.

A collection of tests run after changes to a prompt or system to ensure that existing functionality has not been broken or degraded. It is the safety net for prompt evolution.

• Content: Includes all critical prompt unit tests, deterministic output tests, and performance benchmarks.
• Trigger: Runs automatically after any prompt modification or model version update.
• Purpose: To prevent the introduction of bugs, hallucinations, or performance drops, ensuring system reliability over time.

A quantitative, algorithmically computed score used to assess the quality, relevance, or correctness of a language model's output without requiring human judgment. It enables testing at scale.

• Examples: BLEU, ROUGE for text similarity; JSON schema validation for structured output; custom classifiers for toxicity or factual accuracy.
• Role: Provides immediate, objective feedback in a prompt CI/CD pipeline, though often supplemented by human evaluation scores for nuanced tasks.
• Challenge: Designing metrics that truly correlate with human judgment of quality.