Document Type Classification Prompt for Legal Intake

The full text of the document to classify. Must be plain text with no markup or binary content.

LEASE AGREEMENT

This Lease Agreement is made on...

Check that length is between 50 and 100,000 characters. Reject empty strings. Strip leading/trailing whitespace before insertion.

A controlled list of valid document types the model may output. Each entry must be a unique string label.

["Lease Agreement", "Non-Disclosure Agreement", "Employment Contract", "Service Agreement", "Settlement Agreement", "Court Filing", "Correspondence", "Invoice", "Regulatory Filing", "Other"]

Validate that the taxonomy is a non-empty JSON array of unique strings. Reject if duplicates exist or if the array contains fewer than 2 entries.

Short definitions for each document type in the taxonomy to disambiguate boundary cases.

{"Lease Agreement": "A contract granting use of property for a specified term in exchange for rent.", "Non-Disclosure Agreement": "A contract restricting the sharing of confidential information between parties."}

Check that every key in [DOCUMENT_TYPE_TAXONOMY] has a corresponding description entry. Descriptions must be non-empty strings between 10 and 500 characters.

A float between 0.0 and 1.0. If the model's confidence in the top classification is below this value, the output should default to 'Uncertain' or trigger human review.

0.75

Parse as float. Reject values outside [0.0, 1.0]. If null or missing, default to 0.7 and log a warning.

The maximum number of tokens the model may use for the classification response. Controls cost and latency.

256

Parse as integer. Must be between 50 and 4096. Reject if the value would truncate the required output schema fields.

The JSON Schema that defines the required output structure, including fields for document_type, confidence_score, and reasoning.

{"type": "object", "properties": {"document_type": {"type": "string", "enum": [...]}, "confidence_score": {"type": "number"}, "reasoning": {"type": "string"}}, "required": ["document_type", "confidence_score"]}

Validate that the schema is valid JSON Schema draft-07 or later. Confirm that 'document_type' has an 'enum' constraint matching [DOCUMENT_TYPE_TAXONOMY]. Reject if required fields are missing.

Optional array of input-output pairs demonstrating correct classification behavior, especially for boundary cases and multi-type documents.

[{"input": "This agreement governs the employment of...", "output": {"document_type": "Employment Contract", "confidence_score": 0.92, "reasoning": "Explicitly states employment terms and conditions."}}]

If provided, validate that each example has 'input' and 'output' fields. Check that output.document_type exists in [DOCUMENT_TYPE_TAXONOMY]. Allow null if no examples are needed.

string

Must exactly match one label from [TAXONOMY]. Case-sensitive exact match required. Reject on partial match or synonym.

number

Must be a float between 0.0 and 1.0 inclusive. Reject if null, negative, or >1.0. If below [CONFIDENCE_THRESHOLD], flag for human review.

string

Must equal the value of document_type. Used for downstream routing. Reject if mismatched with document_type.

array of strings

If present, each element must be a valid label from [TAXONOMY]. Reject if any element is not in the taxonomy. Empty array allowed.

boolean

Must be true if confidence_score < [CONFIDENCE_THRESHOLD] or if document_type is 'UNKNOWN'. Reject if false when confidence is below threshold.

string

Must be a non-empty string between 20 and 500 characters. Reject if empty, whitespace-only, or exceeds length limit. Must reference specific document features.

string

If present, must contain a page, section, or paragraph reference from [DOCUMENT_TEXT]. Reject if citation points to non-existent location.

string (ISO 8601)

Must be a valid ISO 8601 datetime string in UTC. Reject on parse failure or non-UTC timezone offset.

Output label exactly matches a value in the [DOCUMENT_TYPE_TAXONOMY] enum. No extra whitespace, punctuation, or variant spellings.

Label not found in taxonomy list; label is a substring or superstring of a valid value; label is a synonym not in the controlled vocabulary.

Parse output label. Assert label ∈ [DOCUMENT_TYPE_TAXONOMY] using exact string match. Run on 100% of test set.

For documents clearly belonging to one type, the predicted label matches the ground truth label with ≥95% accuracy.

Accuracy below 95% on single-type documents; systematic confusion between similar types (e.g., NDA vs. Confidentiality Agreement).

Confusion matrix on single-type test subset. Calculate precision, recall, F1 per label. Flag any label pair with >5% cross-error.

When [ALLOW_MULTI_LABEL] is false, the prompt returns exactly one label. When true, it returns an array of labels, all valid per taxonomy.

Multiple labels returned when [ALLOW_MULTI_LABEL] is false; single label returned for a compound document when multi-label is true; array contains invalid labels.

Parameterized test: run same multi-type document with [ALLOW_MULTI_LABEL] = false and true. Assert output cardinality matches parameter. Validate all labels against taxonomy.

When [ALLOW_UNKNOWN] is true and document type is not in taxonomy, output label is exactly [UNKNOWN_LABEL_VALUE]. When false, model must not output [UNKNOWN_LABEL_VALUE].

Model outputs a best-guess taxonomy label instead of [UNKNOWN_LABEL_VALUE] when [ALLOW_UNKNOWN] is true; model outputs [UNKNOWN_LABEL_VALUE] when [ALLOW_UNKNOWN] is false.

Test with 10 out-of-scope documents (e.g., marketing flyer, restaurant menu). Assert label equals [UNKNOWN_LABEL_VALUE] only when [ALLOW_UNKNOWN] is true.

When [INCLUDE_CONFIDENCE] is true, output includes a confidence field with a float between 0.0 and 1.0. Confidence <0.7 correlates with incorrect classifications.

Confidence field missing or null when [INCLUDE_CONFIDENCE] is true; confidence >0.9 on incorrect classifications; confidence values outside 0.0-1.0 range.

Parse confidence field. Assert 0.0 ≤ confidence ≤ 1.0. For all misclassifications, assert confidence <0.7. Run on full test set.

Output is valid JSON matching the [OUTPUT_SCHEMA]. All required fields present. No extra fields. Field types match schema definitions.

JSON parse error; missing required field (e.g., document_type); extra undeclared field; string where number expected; array where string expected.

Validate output against [OUTPUT_SCHEMA] using a JSON Schema validator. Run on 100% of test set. Any validation failure is a hard fail.

When [DOCUMENT_TEXT] is empty, whitespace-only, or unparseable, output includes an error field or returns [UNKNOWN_LABEL_VALUE] without crashing.

JSON parse error; model hallucinates a document type from empty input; model returns a 200-style response with a made-up classification.

Test with empty string, 500 spaces, base64 garbage, and a 1-character input. Assert valid JSON output. Assert document_type is [UNKNOWN_LABEL_VALUE] or error field is populated.

For documents exceeding the model's context window, classification is based on the first [MAX_TOKENS] tokens and accuracy remains within 10% of full-document accuracy.

Model refuses to classify; model classifies based on tail of document only; accuracy drops >10% compared to full-document baseline.

Select 10 long documents (>100K tokens). Truncate to [MAX_TOKENS] before sending. Compare accuracy against full-document classification (using a long-context model as ground truth). Assert accuracy delta ≤10%.