Contract Test Generation from OpenAPI Spec Prompt

The complete OpenAPI 3.x specification document as a JSON or YAML string. Provides the source of truth for all endpoints, schemas, parameters, and responses.

{"openapi": "3.0.3", "info": {...}, "paths": {...}}

Parse as valid JSON or YAML. Must contain at least one path item. Reject if spec version is not 3.x or if the document fails structural validation against the OpenAPI schema.

The target test framework and language for generated test code. Determines assertion style, test structure, and import statements.

pytest (Python 3.11+) with requests and jsonschema

Must be a recognized framework. Validate against a supported list: pytest, jest, JUnit, Go testing, etc. Reject unsupported frameworks to prevent un-runnable output.

A list of specific HTTP methods, status codes, or schema components to prioritize for test generation. Controls scope when full spec coverage is impractical.

["POST /users", "GET /users/{id}", "400 responses"]

Each entry must match an operationId, path+method pattern, or response code present in [OPENAPI_SPEC]. Warn if a target matches nothing. Empty list means generate for all operations.

Authentication requirements for the API under test. Needed to generate correct auth headers, token acquisition steps, or mock configurations in test setup.

{"type": "Bearer", "token_env_var": "API_TEST_TOKEN"}

Must be a valid JSON object with a type field. Supported types: Bearer, Basic, APIKey, OAuth2, None. Reject if required credentials are missing for the specified type.

Sample test code from the target repository showing existing conventions for fixtures, mocking, assertions, and naming. Ensures generated tests blend into the codebase.

def test_create_user_valid_payload(db_session): ...

Optional. If provided, parse as code and extract fixture names, import patterns, and assertion style. Warn if the sample uses a different framework than [TEST_FRAMEWORK].

Custom rules for edge case generation beyond spec defaults. Defines boundary values, invalid types, and fuzzing strategies specific to the API domain.

{"string_fields": ["empty", "max_length+1", "SQL_injection"], "integer_fields": ["-1", "0", "max+1"]}

Must be a valid JSON object. Each key must reference a JSON Schema type. Reject rules that target types not present in [OPENAPI_SPEC] schemas to avoid irrelevant test cases.

The exact structure for each generated test case. Defines the fields the model must populate for every test.

{"test_name": "string", "method": "string", "path": "string", "request_body": "object|null", "expected_status": "integer", "response_schema_checks": ["string"]}

Must be a valid JSON Schema object. Validate that required fields include test_name, method, path, and expected_status at minimum. Reject schemas missing these core fields.

string

Must match pattern ^[a-z0-9_]+$ and reference the source [OPENAPI_SPEC_PATH]

array of objects

Array length must be >= 1; reject empty array

string

Must be unique within the array; format: [ENDPOINT_METHOD][ENDPOINT_PATH][SCENARIO]

object

Must contain method and path matching an operation in [OPENAPI_SPEC_PATH]

string

Must be one of: schema_validation, status_code, response_shape, edge_case, required_field, enum_boundary

object

Must conform to the requestBody schema from the referenced endpoint; null allowed for GET/DELETE

integer

Must be a valid HTTP status code defined in the endpoint responses map

array of objects

Each assertion must include field_path, operator, and expected_value; operator must be in [equals, contains, matches_schema, is_present, is_absent, type_check]

Generated tests cover all schema properties, required fields, and nested objects from [OPENAPI_SPEC]. Score 3 if coverage >= 95%, 2 if >= 85%, 1 if >= 70%, 0 if below 70%.

Missing test cases for documented properties, especially nested or optional fields. Test suite skips entire response schemas.

Parse generated test code and extract all property paths asserted. Compare against flattened property list from parsed OpenAPI spec. Compute coverage percentage.

Tests cover all documented response status codes per endpoint, including 2xx, 4xx, and 5xx where defined. Score 3 if all endpoints have full status coverage, 2 if missing one code per endpoint, 1 if missing multiple, 0 if only happy-path tested.

Generated tests only assert 200 responses. Error codes documented in [OPENAPI_SPEC] have no corresponding test cases.

Extract status codes from generated test assertions. Compare against status codes listed in each operation's responses object in the spec. Flag missing codes.

Tests include boundary values, empty arrays, null fields, maximum/minimum constraints, and enum validation. Score 3 if >= 3 edge case categories present per endpoint, 2 if 2 categories, 1 if 1 category, 0 if none.

Tests only use valid example values from the spec. No boundary, null, empty, or out-of-range inputs are tested.

Scan generated test inputs for null, empty string, empty array, min/max boundary values, and invalid enum values. Count distinct edge case categories per endpoint.

Generated code compiles and runs without modification in the target framework specified by [TEST_FRAMEWORK]. Score 3 if all tests pass syntax check, 2 if minor import fixes needed, 1 if major refactor required, 0 if unusable.

Import errors, missing dependencies, framework API misuse, or syntax errors that prevent test execution.

Execute a syntax check or dry-run of generated test files in the target framework environment. Count errors and categorize by severity.

Each test case asserts specific expected values or types, not just status codes. Score 3 if all tests have property-level assertions, 2 if most do, 1 if only type checks, 0 if status-code-only assertions.

Tests only assert response.status == 200 with no body validation. Assertions are too generic to catch schema violations.

Parse assertion statements in generated tests. Classify each as status-only, type-check, or property-value assertion. Compute ratio of property-value assertions to total assertions.

Generated output matches [OUTPUT_SCHEMA] exactly, including file structure, naming conventions, and test organization. Score 3 if fully compliant, 2 if minor deviations, 1 if major restructuring needed, 0 if unrecognizable format.

Test files are generated in wrong directory structure, use incorrect naming patterns, or omit required setup/teardown files specified in the output contract.

Validate generated file tree and file contents against the expected structure defined in [OUTPUT_SCHEMA]. Check file names, directory layout, and required boilerplate presence.

All [PLACEHOLDER] values from the prompt are correctly substituted with spec-derived values. Score 3 if no unresolved placeholders, 2 if cosmetic placeholders remain, 1 if functional placeholders missing, 0 if placeholder leakage.

Generated tests contain literal '[BASE_URL]', '[API_KEY]', or other prompt template tokens instead of concrete values or framework-native configuration references.

Scan generated output for square-bracket tokens matching the prompt template pattern. Flag any unresolved placeholders. Verify substitution completeness.

Running the same [OPENAPI_SPEC] through the prompt twice produces structurally equivalent test suites. Score 3 if identical structure, 2 if minor ordering differences, 1 if different test selection, 0 if radically different output.

Two runs produce different endpoint coverage, different assertion strategies, or different file organization for the same input spec.

Run prompt twice with identical inputs. Compare generated test file trees, test case names, and assertion counts. Measure structural similarity score.