Fallback vs Retry Decision Trace Prompt

Raw trace segment containing the original model response, error metadata, and router decision event

{"model": "claude-3-opus", "status": "timeout", "error_code": "UPSTREAM_TIMEOUT", "retry_count": 2, "fallback_available": true}

Must be valid JSON with model, status, error_code, retry_count, and fallback_available fields. Parse check required before prompt assembly. Null status triggers pre-validation rejection.

Maximum retries configured for the primary model before fallback is required

3

Must be a positive integer. Compare against retry_count in trace context. If retry_count exceeds budget, decision is automatically flagged as policy violation regardless of model reasoning.

Available fallback models with capability profiles, cost tiers, and latency characteristics

[{"model_id": "claude-3-haiku", "capabilities": ["text-generation", "tool-use"], "cost_tier": "low", "p50_latency_ms": 400}]

Must be a JSON array with at least one entry. Each entry requires model_id, capabilities array, cost_tier, and p50_latency_ms. Missing capability fields cause false capability-gap flags in audit output.

Declared routing policy rules that govern retry vs fallback decisions

"Retry primary model up to 3 times for timeout errors. Fallback to haiku-tier model on 4th failure. Never retry on content-policy refusals."

Free text or structured policy object. If structured, validate against policy schema before prompt injection. Policy absence produces 'policy_unavailable' flag in decision audit and prevents adherence scoring.

Configured cost optimization objective for routing decisions

"minimize_total_cost"

Must match enum: minimize_total_cost, minimize_latency, balanced, or maximize_quality. Mismatch between declared objective and trace behavior is a primary audit finding. Null allowed only if cost analysis is out of scope.

Maximum acceptable end-to-end latency in milliseconds before fallback is preferred over retry

2000

Must be a positive integer. Compare against cumulative latency in trace. If retry would exceed threshold and fallback was available but not used, flag as suboptimal decision. Null allowed if latency analysis is out of scope.

UTC timestamp of the routing decision event for correlation with deployment and incident timelines

"2025-03-15T14:32:17Z"

Must parse as ISO 8601 UTC. Used for trend correlation and drift detection. Missing timestamp prevents time-window analysis and produces 'timestamp_missing' warning in audit output.

Expected structure for the decision audit output including misclassification counts and per-decision scores

{"decision_id": "string", "appropriateness_score": "appropriate|suboptimal|inappropriate", "misclassification_type": "string|null", "evidence": ["string"]}

Must be a valid JSON Schema object. Validate that prompt output conforms to this schema before accepting audit results. Schema mismatch triggers output repair or retry.

string (UUID)

Must match a valid UUID v4 format. Parse check via regex.

string

Must be non-empty and match the source trace identifier. Cross-reference with input.

enum: retry | fallback | escalate

Must be one of the three allowed values. Schema check.

number (0.0-1.0)

Must be a float between 0.0 and 1.0 inclusive. Confidence threshold: score < 0.7 triggers human review flag.

integer

Must be >= 0. If null or negative, fail validation.

Every trace event is classified as retry, fallback, or escalation with a clear primary reason

Unclassified events, ambiguous labels, or missing reason fields in the output

Run against 20 labeled trace samples with known decisions; require >=90% exact match on classification label

Retry count per model does not exceed the configured [RETRY_BUDGET] and each retry is justified by a transient error signal

Retries exceeding budget without explanation or retries on deterministic failures

Parse retry_count from output; assert retry_count <= [RETRY_BUDGET] for all events; flag any retry on non-transient error codes

Fallback is only recommended when [FALLBACK_MODELS] list is non-empty and at least one alternative is available

Fallback suggested when fallback pool is empty or all alternatives are unavailable

Check output.fallback_recommended flag; assert false when [FALLBACK_MODELS] is empty or all marked unavailable in trace context

Output.misclassification_count matches the number of events where the router's actual decision differs from the scored appropriate decision

Count mismatch between summary and per-event audit list

Sum the misclassified boolean flags from the per-event audit array; assert equals output.misclassification_count

Each scored decision includes a justification field referencing specific trace evidence (error code, latency threshold, capability gap)

Justification fields that are empty, generic ('seems wrong'), or reference data not present in the trace

Parse justification field for each event; assert length > 20 characters and contains at least one trace field reference (e.g., error_code, latency_ms, model_capability)

Confidence scores are provided per decision and correlate with decision ambiguity (borderline cases get lower scores)

All confidence scores are 1.0 or 0.0 with no intermediate values, or scores are missing entirely

Check confidence_score range is 0.0-1.0; assert standard deviation > 0.1 across a batch of 10 mixed-difficulty traces

Output matches the [OUTPUT_SCHEMA] exactly with all required fields present and correctly typed

Missing required fields, wrong types, or extra fields not in schema

Validate output against [OUTPUT_SCHEMA] using JSON Schema validator; assert no validation errors

Cumulative latency attributed to retries and fallbacks sums to within 5% of total trace latency delta

Attributed latency sum deviates >5% from total trace latency or negative latency values appear

Sum retry_latency_ms and fallback_latency_ms fields; assert abs(sum - total_latency_delta_ms) / total_latency_delta_ms <= 0.05