Tool Access Policy Dry-Run Evaluation Prompt

This prompt is designed for safety engineers and platform operators who need to quantify the impact of a proposed tool access policy change before enforcing it in production. The core job-to-be-done is a structured dry-run: evaluating a new policy draft against a representative sample of historical tool invocation logs to produce a decision record of what would have been denied, what would have been allowed, and a diff analysis against the current policy. This workflow is essential when a policy change could break legitimate user workflows, create false positive denial storms, or fail to close newly discovered attack patterns. The ideal user has access to a policy document, a log sample of real traffic, and a need for a quantified estimate of denial rate, false positive risk, and policy gap coverage before committing the change.

Use this prompt when you have three concrete inputs: a proposed policy document in natural language or structured format, a sample of historical tool invocation logs with user context and tool arguments, and the current policy for diff comparison. The prompt is not a replacement for a real-time enforcement engine; it is a pre-deployment evaluation tool. Do not use this prompt when you lack historical traffic data, when the policy is still in early ideation without a stable draft, or when you need real-time blocking decisions. The output is a structured analysis record, not an enforcement action. For high-risk production environments, always pair the dry-run results with a human review step before approving the policy for enforcement, and ensure the evaluation sample is representative of production traffic volume and diversity.

After running this evaluation, the reader should have a clear decision record showing the estimated denial rate, a list of false positive candidates, and a gap analysis of attack patterns the new policy would still miss. The next step is to review the false positives and gaps with the policy team, adjust the policy draft if necessary, and re-run the dry-run until the risk profile is acceptable. Avoid the common mistake of treating a low denial rate as a safety signal without inspecting the specific requests that would be denied; a single false positive on a critical user workflow can be more damaging than a high denial rate on low-value traffic. Once the dry-run results are approved, the policy can be promoted to a shadow enforcement mode or a staged rollout with real-time monitoring.

This prompt is designed to be pasted directly into your evaluation harness. It instructs the model to act as a policy auditor, comparing each historical tool invocation against a proposed access policy without actually blocking any requests. The model outputs a structured decision for each invocation, including whether the policy would have denied it, the specific rule triggered, and a confidence score. Use this template to run a dry-run analysis before deploying a new policy to production.

textCopy
You are a tool access policy auditor. Your task is to evaluate a proposed tool access policy against a set of historical tool invocation records. You must determine, for each record, whether the proposed policy would have denied the request. You are not executing any tools. You are only performing a dry-run evaluation.

## PROPOSED POLICY
[POLICY_DOCUMENT]

## HISTORICAL TOOL INVOCATIONS
[INVOCATION_RECORDS]

## EVALUATION RULES
1. For each invocation record, compare the tool name, arguments, user role, and session context against every rule in the proposed policy.
2. If any rule would deny the invocation, mark it as DENIED. Cite the specific rule ID and the condition that matched.
3. If no rule applies, mark it as ALLOWED.
4. If the policy is ambiguous for a given invocation, mark it as AMBIGUOUS and explain the ambiguity.
5. For each decision, provide a confidence score between 0.0 and 1.0.

## OUTPUT SCHEMA
Return a JSON object with a single key "evaluations" containing an array of objects. Each object must have the following fields:
- "invocation_id": string (from the input record)
- "decision": "DENIED" | "ALLOWED" | "AMBIGUOUS"
- "rule_id": string | null (the ID of the triggering rule, if DENIED)
- "matched_condition": string | null (description of the matched condition)
- "ambiguity_reason": string | null (required if AMBIGUOUS)
- "confidence": number (0.0 to 1.0)

## CONSTRAINTS
- Do not invent invocation records. Only evaluate the records provided.
- Do not modify the proposed policy. Evaluate it as written.
- If the policy document is malformed or unparseable, return a single error object with key "error" and a description.
- Output only the JSON object. No markdown fences, no commentary.

## EXAMPLES
[FEW_SHOT_EXAMPLES]

To adapt this prompt, replace each square-bracket placeholder with real data before running the dry-run. The [POLICY_DOCUMENT] should contain the full text of your proposed tool access policy, including rule IDs, conditions, and scopes. The [INVOCATION_RECORDS] should be a JSON array of historical tool calls, each with an invocation_id, tool_name, arguments, user_role, and session_context. The [FEW_SHOT_EXAMPLES] placeholder should be replaced with 2-4 worked examples showing correct DENIED, ALLOWED, and AMBIGUOUS decisions to calibrate the model's output format and reasoning. After running the evaluation, always validate the output JSON against the schema before computing denial rates or false positive estimates. For high-stakes policy changes, have a human safety engineer review a sample of AMBIGUOUS decisions and any DENIED records with confidence scores below 0.85 before trusting the aggregate metrics.

This prompt is not a one-off manual test; it is designed to be the core of an automated evaluation harness that runs against a golden dataset of historical tool invocations. The harness replays each request through the proposed policy prompt, compares the predicted denial decision against the known ground truth (or the current policy's decision), and surfaces discrepancies. The primary goal is to estimate the false positive rate—how many previously allowed requests the new policy would incorrectly block—and to identify specific request patterns that cause unexpected denials.

To implement the harness, first assemble a labeled dataset where each record contains the original [USER_CONTEXT], [TOOL_CALL], and the actual production decision (allow/deny) under the current policy. For each record, render the prompt template with the record's fields and send it to the model. Parse the structured [OUTPUT_SCHEMA] from the response, which should include the decision (allow/deny), the policy_clause triggered, and a confidence_score. Log every decision alongside the input record. A post-processing script should then calculate the false positive rate (new policy denies, old policy allowed), false negative rate (new policy allows, old policy denied), and a diff of policy clauses triggered. For high-risk domains, any record where the model's confidence is below a configurable threshold (e.g., 0.85) should be flagged for human review before the policy is approved.

The harness must also validate the model's output against the expected schema. If the model fails to return valid JSON or omits required fields, implement a retry loop with a maximum of two attempts, feeding the raw output and a schema error message back into the model. If retries are exhausted, log the failure and treat the record as a 'flag for review' case. Do not silently default to 'deny' or 'allow' on parse failures, as this corrupts the false positive rate estimation. Finally, version the prompt, the dataset, and the evaluation results together so that any policy change can be traced to its impact on the dry-run metrics. This traceability is critical for audit and for rolling back a policy that introduces unacceptable collateral denial rates.