Agent Tool Discovery Prompt with Confidence Scoring

This prompt is for agent framework developers and platform engineers who need to build uncertainty-aware tool selection into their systems. The core job-to-be-done is generating a structured capability manifest from a set of tool descriptions where the model must explicitly distinguish between properties it can confirm from the provided schema, properties it can reasonably infer, and properties that remain uncertain. You need this when your agent operates in a dynamic tool ecosystem—such as an MCP server fleet with heterogeneous, third-party, or poorly documented tools—and silent capability hallucination is an unacceptable failure mode. The ideal user is an AI engineer wiring this prompt into a tool registration pipeline, an agent initialization sequence, or a runtime capability refresh loop.

Do not use this prompt when you have a static, fully documented tool set with hand-curated capability descriptions. If every tool has a complete, validated JSON Schema and human-authored constraints, a simple schema pass-through is cheaper and more reliable. This prompt adds value when tool metadata is sparse, when descriptions come from natural-language documentation or code annotations, or when you need to surface uncertainty to a downstream planner or a human reviewer. It is also inappropriate for real-time, latency-critical tool selection on every turn; the confidence scoring and structured output generation add overhead better suited for registration-time or refresh-interval workflows rather than per-call decisioning.

Before using this prompt, ensure you have a well-defined input contract: a list of tool names, their available descriptions, and any schemas or parameter lists you already possess. The output is a structured JSON array where each tool entry includes a capabilities list, and each capability carries a confidence score of confirmed, inferred, or uncertain, along with a rationale string. Wire this into a validation step that checks for missing confidence fields, capabilities that contradict the input schema, and confidence inflation on sparse inputs. For high-stakes production systems, route uncertain capabilities to a human review queue before registering them in the agent's active tool registry. The next step after reading this section is to copy the prompt template, adapt the placeholders to your tool metadata format, and build the validation harness described in the implementation section.

This prompt template instructs an agent to introspect its available tool set and produce a structured capability manifest where every claim is accompanied by a confidence score. The scoring distinguishes between confirmed capabilities (directly observed from tool schemas), inferred capabilities (reasoned from parameter patterns or naming conventions), and uncertain capabilities (speculative or based on incomplete metadata). This prevents the agent from silently hallucinating tool features that don't exist, which is the most common failure mode in dynamic tool ecosystems.

textCopy
You are an agent performing tool capability discovery. Your task is to examine the available tools listed below and produce a structured capability manifest. For every capability claim you make, you must assign an explicit confidence score.

## AVAILABLE TOOLS
[TOOLS]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "discovery_timestamp": "ISO-8601 timestamp of discovery",
  "registry_source": "[REGISTRY_SOURCE]",
  "capabilities": [
    {
      "tool_name": "string",
      "capability": "string describing what the tool can do",
      "confidence": "confirmed | inferred | uncertain",
      "evidence": {
        "source": "schema_field | parameter_pattern | naming_convention | documentation | inference",
        "detail": "specific field, parameter, or pattern that supports this claim"
      },
      "input_parameters": [
        {
          "name": "string",
          "type": "string",
          "required": true,
          "description": "string",
          "confidence": "confirmed | inferred | uncertain"
        }
      ],
      "output_schema": {
        "description": "string describing expected output",
        "confidence": "confirmed | inferred | uncertain"
      },
      "constraints": ["list of known limitations or constraints"],
      "requires_approval": false,
      "is_destructive": false
    }
  ],
  "gaps": [
    {
      "description": "capability the agent might need but no tool provides",
      "severity": "blocking | major | minor"
    }
  ],
  "uncertain_items": [
    {
      "tool_name": "string",
      "claim": "the uncertain claim",
      "reason": "why confidence is not confirmed",
      "recommended_action": "human_review | test_invocation | schema_request"
    }
  ]
}

## CONFIDENCE DEFINITIONS
- **confirmed**: The capability is explicitly declared in the tool's schema, parameter definitions, or official documentation. No inference required.
- **inferred**: The capability is reasonably deduced from parameter names, return types, tool naming patterns, or ecosystem conventions, but is not explicitly stated in the schema.
- **uncertain**: The capability is speculative. Evidence is weak, contradictory, or based solely on ambiguous naming. Must be flagged for human review.

## RULES
1. Do not claim any capability that cannot be traced to evidence in the provided tool definitions.
2. If a tool's schema is incomplete or ambiguous, mark affected parameters and capabilities as "inferred" or "uncertain" rather than guessing.
3. List all uncertain items in the `uncertain_items` array with recommended actions.
4. If you detect capability gaps where no tool satisfies a likely agent need, list them in the `gaps` array.
5. For destructive operations (delete, write, modify, send), set `is_destructive` to true and `requires_approval` to true.
6. If the tool registry is empty or unparseable, return a valid JSON object with empty `capabilities` and a `gaps` entry describing the failure.

## EXAMPLES
[EXAMPLES]

## CONSTRAINTS
[CONSTRAINTS]

Adaptation notes: Replace [TOOLS] with the serialized tool definitions from your registry, MCP server response, or API spec. Use [REGISTRY_SOURCE] to tag the origin (e.g., mcp-server:filesystem-v2, openapi:payment-service). The [EXAMPLES] block should contain one confirmed, one inferred, and one uncertain capability example to calibrate the model's scoring behavior. The [CONSTRAINTS] block can inject domain-specific rules such as "never claim write access unless the schema contains a write or mutate operation." Test this prompt with tools that have deliberately sparse or ambiguous schemas to verify that the model defaults to inferred or uncertain rather than confirmed.

Before integrating this prompt into a production agent harness, validate the output against the JSON Schema defined above. Reject any response where a confirmed capability lacks a specific evidence.source value. For high-risk tool ecosystems, route all uncertain items to a human review queue before registering them in the agent's active tool set. The next section covers how to wire this prompt into an application with retry logic, schema validation, and audit logging.

This prompt is designed to sit inside a tool discovery step that runs at session start, after a registry refresh, or when an agent encounters an unfamiliar tool endpoint. The implementation harness must treat the prompt output as a structured artifact that downstream planning and tool-selection logic depends on. Because confidence scores directly influence whether the agent proceeds, falls back, or escalates, the harness must validate the output schema before any tool call is authorized. A common pattern is to wrap this prompt in a dedicated ToolDiscoveryService that caches results, compares them against the current registry, and emits structured diffs for the agent's planner.

Validation and schema enforcement are the first integration concern. The harness must confirm that every capability claim includes a confidence field with one of the allowed enum values (confirmed, inferred, uncertain), a source field that traces back to a specific tool description or schema field, and a capability_id that matches the registry. Use a JSON Schema validator or a lightweight Pydantic model in the service layer. If validation fails, retry the prompt once with the validation error injected into the [CONSTRAINTS] block. If it fails again, log the malformed output and fall back to the last known valid discovery result rather than passing unvalidated capabilities to the agent planner. For high-risk domains where tool misuse could cause data loss or external side effects, insert a human review gate for any capability scored as uncertain before it enters the agent's available tool list.

Model choice and latency budgeting matter here. This prompt benefits from models with strong instruction-following and structured output support, such as GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro. Use response_format with a strict JSON schema where the provider supports it. Set a timeout appropriate to your tool count: 5-10 seconds for registries under 50 tools, scaling up for larger manifests. If the discovery step runs on every session start, cache the result with a TTL tied to your registry refresh interval. Observability requires logging the full prompt input, the raw output, the validation result, and the final capability list that the agent receives. Attach trace IDs that link discovery outputs to subsequent tool calls so you can audit whether an agent used a capability it should not have had. The most common production failure mode is a stale registry causing the prompt to infer capabilities that no longer exist; your harness should compare discovery timestamps against registry last_modified fields and force a refresh when drift is detected.

Retry and fallback logic must handle the case where the model refuses to score or produces an empty capability list. Implement a circuit breaker: if three consecutive discovery attempts fail validation or return zero capabilities, stop retrying, log a critical alert, and either load a static fallback manifest or escalate to the on-call channel. Never let the agent proceed with an empty or unvalidated tool list unless your system explicitly supports a zero-tool degraded mode. Finally, wire the confidence scores into your agent's planning prompt. The planner should treat confirmed capabilities as safe to use, inferred capabilities as usable with caution and explicit error handling, and uncertain capabilities as requiring user confirmation or alternative tool search before invocation. This closes the loop from discovery to safe execution.