Canary Token Output Detection Harness Prompt

This prompt is designed for security monitoring teams who have embedded canary tokens in system instructions and need an automated, structured way to scan model outputs for token leakage. The primary job-to-be-done is runtime detection of extraction attacks: when an adversary successfully exfiltrates hidden system prompt markers, this harness provides immediate, machine-readable evidence. The ideal user is a security engineer or MLOps practitioner integrating this as a post-processing step in an AI gateway, where every model response passes through the detector before reaching the user. Required context includes a list of known canary tokens, their associated severity tiers, and the raw model output string to scan.

Use this prompt when you have a defined canary token strategy already in place—tokens are embedded, their formats are known, and you need consistent, low-latency detection at inference time. It is particularly valuable in defense-in-depth architectures where output filtering complements input sanitization, rate limiting, and human review. The harness returns structured JSON with confidence scores, match locations, and false-positive handling notes, making it suitable for automated alerting pipelines and SIEM integration. Do not use this prompt as a standalone security control; it is one detection layer and cannot prevent token leakage, identify novel extraction techniques, or replace upstream defenses like instruction hierarchy hardening and context window management.

Before deploying, ensure your canary token list is complete and up-to-date. Missing tokens will produce false negatives. Configure alert thresholds based on token severity: surface-level tokens might warrant logging, while deep instruction tokens should trigger immediate escalation. The harness includes false-positive handling for legitimate token-like strings that appear in normal conversation, but you should tune the confidence scoring against your own production traffic patterns. For high-severity detections, always route to human review before taking automated action. Next, wire the output into your incident response playbook so that detection events trigger investigation, not just logging.

The following prompt template is designed to be dropped into an automated detection pipeline. It accepts a raw model output string and a list of known canary tokens, then returns a structured JSON result indicating whether any token was detected, along with confidence scores and supporting evidence. The template uses square-bracket placeholders for all dynamic inputs, making it straightforward to parameterize in code before each invocation.

textCopy
You are a security-focused output scanner. Your job is to examine the provided model output for the presence of any known canary tokens. A canary token is a unique, non-functional string deliberately embedded in system instructions to detect instruction leakage or extraction attacks.

[TOKEN_DEFINITIONS]

[OUTPUT_SCHEMA]

[CONSTRAINTS]

Model output to scan:
---
[INPUT]
---

Scan the above output for any canary token defined in [TOKEN_DEFINITIONS]. Return ONLY a valid JSON object conforming to [OUTPUT_SCHEMA]. Do not include any other text, commentary, or markdown formatting.

To adapt this template, replace each placeholder with concrete values. [TOKEN_DEFINITIONS] should contain a structured list of canary tokens, their expected formats, and any variant patterns (e.g., base64-encoded versions, delimiter-wrapped versions). [OUTPUT_SCHEMA] must define the exact JSON shape you expect, including fields for detected (boolean), confidence (0.0 to 1.0), matched_token (string or null), match_location (character offset or null), and evidence (string excerpt or null). [CONSTRAINTS] should specify handling rules such as "treat partial matches as non-detections unless confidence exceeds 0.9" or "flag any token that appears verbatim regardless of surrounding context." [INPUT] receives the raw model output string to be scanned. For high-risk deployments, always pair this prompt with a post-processing validator that confirms the JSON structure matches the expected schema before acting on the result.

When integrating this prompt into a production harness, avoid relying solely on the model's detection capability. Implement a secondary regex-based or exact-match check against the known token list as a deterministic fallback. This dual-layer approach catches cases where the model misses an obvious token or hallucinates a false positive. Log every scan result—including the raw input, the model's JSON response, and the deterministic check outcome—for audit and false-positive analysis. If the detection pipeline triggers an alert, require human review before taking automated action such as session termination or key rotation.

The Canary Token Output Detection Harness Prompt is designed to be the final gate in your model response pipeline. It should be invoked synchronously after every model generation, before the output is returned to the user or logged to an external system. The harness receives the raw model output and a pre-configured list of active canary tokens, then returns a structured detection result. This is not a prompt you expose to end users; it is an internal control plane component that runs inside your AI gateway, middleware, or model-serving layer. The primary integration point is a post-generation hook that calls the LLM with this prompt, parses the structured JSON response, and takes action based on the detection_decision field.

To wire this into production, wrap the prompt in a thin service function that accepts model_output: string and active_canaries: list[CanaryToken] where each CanaryToken object includes a token_value, token_tier (e.g., surface, deep, critical), and alert_threshold. The function should construct the prompt by injecting these values into the [CANARY_TOKEN_LIST] and [MODEL_OUTPUT] placeholders. After receiving the LLM's response, validate the output against a strict JSON schema that requires detection_decision (enum: CLEAN, SUSPICIOUS, CONFIRMED_LEAK), detected_tokens (array of objects with token_id, token_tier, confidence_score, and match_location), and recommended_action (enum: LOG_ONLY, ALERT, BLOCK_AND_ESCALATE). If the JSON fails to parse or schema validation fails, retry once with a repair prompt. If the retry also fails, default to BLOCK_AND_ESCALATE and log the raw output for manual review.

Model choice matters here. Use a fast, cheap model for this detection task—GPT-4o-mini, Claude Haiku, or a fine-tuned small model are appropriate. The detection logic is pattern-matching with semantic context, not complex reasoning. Set temperature=0 to eliminate variance in detection results. For high-throughput systems, consider batching multiple outputs into a single detection request by formatting the [MODEL_OUTPUT] as a numbered list, but be aware that this increases the risk of cross-contamination in the prompt context. Always log every detection result—including CLEAN decisions—to a structured audit store with fields for timestamp, session_id, model_output_hash, detection_decision, detected_tokens, and llm_latency_ms. This audit trail is essential for incident response and for tuning false-positive rates over time.

Configure alerting thresholds in your harness wrapper, not in the prompt itself. The prompt returns structured data; your application code decides when to fire a PagerDuty alert, increment a Prometheus counter, or block a response. A CONFIRMED_LEAK on a critical tier token should trigger immediate blocking and escalation. A SUSPICIOUS detection with a confidence score below 0.7 might only increment a metric and log for later review. False positives are the primary failure mode here—the model may flag legitimate outputs that coincidentally contain canary-like strings. Mitigate this by using high-entropy canary tokens (UUIDs, random base64 strings) that are vanishingly unlikely to appear in normal text, and by tuning the confidence_score threshold in your application logic rather than relying solely on the model's judgment. Run weekly reviews of all SUSPICIOUS detections to identify patterns and adjust thresholds.