SDK Code Example Validation and Testing Prompt

This prompt is designed for documentation engineers, SDK maintainers, and CI/CD pipeline owners who need to programmatically verify that code examples embedded in documentation are functional against a specific SDK version. The primary job-to-be-done is automating a pre-release or periodic documentation health check, replacing the manual, error-prone process of copying, pasting, and running each code block by hand. The ideal user has access to a containerized execution environment where SDK dependencies can be installed and code can be run safely, along with a documentation source (Markdown, MDX, or HTML) that contains fenced code blocks with language identifiers.

You should use this prompt when you have a batch of documentation files to validate and you can provide the exact SDK version and package name to test against. The prompt assumes that the extracted code blocks are intended to be runnable as scripts or within a specific context (e.g., an async function). It is most effective when paired with a harness that can execute the generated test scripts in isolated containers, capture stdout/stderr, and compare exit codes. Do not use this prompt for code that requires complex, stateful external resources (like a live database with specific pre-existing data) unless you can provide those fixtures in the execution environment. It is also unsuitable for validating purely illustrative pseudocode or code blocks that are intentionally incomplete to demonstrate a concept.

Before using this prompt, ensure you have a clear [OUTPUT_SCHEMA] defined for your test report, such as a JSON object containing file path, block index, status (pass/fail/error), and diagnostic messages. The prompt is not a replacement for a full integration test suite; it validates that examples compile and run without runtime errors, not that they produce a specific correct business logic outcome. For high-risk SDKs where a broken example could lead to security vulnerabilities (e.g., auth token leakage, insecure defaults), always include a human review step after automated validation to inspect any code blocks that were modified or flagged. Start by running the prompt against a small, known-good documentation set to calibrate its false positive rate in code block extraction before scaling to your entire docs corpus.

This template is designed to be dropped directly into your orchestration layer—whether that's a Python script, a LangChain node, or a custom evaluation harness. It accepts raw documentation content containing code blocks, identifies the target SDK and language runtime, and produces a structured validation report. The prompt instructs the model to generate an executable test script, run it in a controlled environment, and return pass/fail results with specific diagnostic information. Before using this template, ensure you have a sandboxed execution environment available and that your model has access to a code execution tool or that you plan to execute the generated script in a separate CI step.

textCopy
You are an SDK validation engineer. Your task is to extract code examples from the provided documentation, generate a test harness, execute it against the current SDK version, and report the results.

## INPUT
Documentation content with embedded code blocks:

[DOCUMENTATION_CONTENT]

codeCopy

## TARGET ENVIRONMENT
- Language: [LANGUAGE] (e.g., python, javascript, java)
- SDK Package: [PACKAGE_NAME]
- SDK Version: [SDK_VERSION] (e.g., latest, 2.1.0)
- Execution Command: [EXECUTION_COMMAND] (e.g., python test_script.py)

## CONSTRAINTS
- Extract ALL code blocks from the input, including inline snippets and fenced blocks.
- Generate a single, self-contained test script that imports the SDK, executes each example, and asserts basic success (no exceptions, non-null responses where expected).
- Wrap each example in a try-catch block. Log failures with the example index, error type, and error message.
- If an example requires API keys or external resources, generate a mock or skip it with a clear "SKIPPED: requires live credentials" note.
- Flag any usage of deprecated methods, classes, or parameters based on the specified SDK version.
- Do NOT modify the original example logic. Only add execution wrappers and assertions.

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "total_examples_found": <int>,
  "examples_executed": <int>,
  "examples_passed": <int>,
  "examples_failed": <int>,
  "examples_skipped": <int>,
  "deprecation_warnings": [
    {
      "example_index": <int>,
      "deprecated_symbol": "<string>",
      "suggestion": "<string>"
    }
  ],
  "failures": [
    {
      "example_index": <int>,
      "error_type": "<string>",
      "error_message": "<string>",
      "code_snippet": "<string>"
    }
  ],
  "skipped": [
    {
      "example_index": <int>,
      "reason": "<string>"
    }
  ]
}

## INSTRUCTIONS
1. Parse the input and extract all code blocks. Identify the language of each block.
2. Generate a test script that imports [PACKAGE_NAME] and executes each example sequentially.
3. Execute the script using [EXECUTION_COMMAND].
4. Capture stdout, stderr, and exit code.
5. Parse the execution output and populate the OUTPUT SCHEMA.
6. Return ONLY the JSON object. No markdown fences, no commentary.

To adapt this template for your specific documentation pipeline, replace the square-bracket placeholders with concrete values. For a Python SDK, you might set [LANGUAGE] to python, [PACKAGE_NAME] to acme-client, and [EXECUTION_COMMAND] to python3 generated_test.py. The [DOCUMENTATION_CONTENT] placeholder should receive the raw markdown or HTML of your documentation page. If your SDK requires authentication even for basic operations, extend the constraints section to include environment variable setup instructions. For high-risk production documentation where broken examples directly impact developer trust, add a human review step after this prompt runs: route any output where examples_failed > 0 or deprecation_warnings is non-empty to a documentation engineer for manual inspection before publishing.

This prompt is designed to operate inside a deterministic harness, not as a one-off chat interaction. The typical integration point is a CI job that runs on documentation pull requests or a scheduled nightly job that validates all published examples against the latest SDK release. The harness must extract code blocks from Markdown or MDX files, execute each block in an isolated environment with the target SDK version installed, and collect pass/fail results. The LLM prompt itself handles the generation of the test runner script and the analysis of execution output—it does not perform the extraction or execution directly. Your application code owns extraction, sandboxing, and timeout enforcement.

Build the harness in three stages. First, extract fenced code blocks tagged with the target language (e.g., ```python) from documentation files. Use a parser that handles nested fences, file includes, and frontmatter correctly—regex-based extraction will miss edge cases. Second, for each extracted block, construct the prompt by populating [SDK_LANGUAGE], [SDK_VERSION], [CODE_BLOCK], and [EXPECTED_BEHAVIOR] placeholders. The model returns a test script and a valid boolean. Third, execute the generated test script in a sandboxed container with only the declared SDK version and dependencies available. Capture stdout, stderr, exit codes, and execution time. If the script exits non-zero or the model's valid field is false, flag the example for review. Do not execute arbitrary model-generated code without sandboxing—the test scripts are generated by an LLM and must be treated as untrusted input.

Add a validation layer before execution to catch obvious extraction failures. Check that the extracted code block is non-empty, exceeds a minimum line count (single-line snippets are often incomplete), and does not contain placeholder tokens like YOUR_API_KEY or .... If the model's valid field is false but the script executed successfully, log this as a false positive in extraction and tune the prompt's [CONSTRAINTS] to reduce sensitivity. Conversely, if valid is true but execution fails, log a false negative and inspect whether the test harness environment matches the documented prerequisites. Store all results—extraction metadata, prompt version, model version, execution logs, and pass/fail status—in a structured report. For high-stakes SDKs where broken examples directly cause customer churn, route all failures to a human review queue before blocking the documentation publish pipeline.