Handwritten Annotation vs. Printed Text Prompt

This prompt is designed for document automation engineers who need to reliably separate original printed content from added handwriting in forms, contracts, reports, and scanned documents. The core job-to-be-done is per-region classification: for each text region on a page, determine whether it is printed, handwritten, or mixed, and extract the text content for both layers with bounding boxes and confidence scores. This is not a general OCR prompt—it assumes you already have text regions detected and need to classify their origin. The ideal user is building a document processing pipeline where downstream actions depend on knowing what was originally printed versus what was added later, such as extracting filled form values, detecting unauthorized annotations, or preserving the original document layer for archival purposes.

Use this prompt when you have documents with overlapping annotations, margin notes, mixed ink colors, or handwriting that intersects with printed text. It handles cases where a single bounding box contains both printed and handwritten content by producing separate extraction results for each layer. The prompt requires input regions with bounding box coordinates and the document image or extracted text for context. You should not use this prompt for documents where the distinction between print and handwriting is irrelevant to your workflow, or when you need character-level rather than region-level classification. It is also not suitable for real-time video streams or documents where handwriting is the only content type present—those scenarios require different prompt architectures.

Before deploying this prompt, you must define your confidence thresholds for routing to human review. The prompt produces confidence scores per classification, and you should establish a calibration set of documents with known ground truth to determine at what threshold misclassifications become unacceptable. Common failure modes include stylized fonts (especially script and decorative typefaces) being misclassified as handwriting, and light pencil annotations on dark backgrounds being missed entirely. Plan for these by including counterexamples in your few-shot demonstrations and implementing a human review queue for low-confidence regions. If your pipeline processes regulated documents, such as medical forms or legal contracts, human review of all handwriting classifications is strongly recommended until you have validated performance on your specific document population.

This prompt template separates handwritten annotations from printed text in document images. It is designed for document processing pipelines that must distinguish original printed content from added handwriting—such as margin notes, filled form fields, corrections, and signatures. The template expects a document image as input and produces per-region classification with extracted text for both layers, bounding box coordinates, and confidence scores. Use this prompt when you need structured, machine-readable output that can be validated and routed downstream. Do not use this prompt for full OCR of clean digital-born documents; a standard text extraction pipeline is more appropriate for that case.

textCopy
You are a document analysis system that separates handwritten annotations from printed text in document images.

Analyze the provided document image and classify each text region as either `handwritten` or `printed`. For each region, extract the text content and provide bounding box coordinates.

[INPUT]
- Document image: [DOCUMENT_IMAGE]
- Page number: [PAGE_NUMBER]

[OUTPUT_SCHEMA]
Return a JSON object with the following structure:
{
  "page_number": [PAGE_NUMBER],
  "regions": [
    {
      "region_id": "string",
      "classification": "handwritten | printed",
      "content": "string",
      "bounding_box": {
        "x": number,
        "y": number,
        "width": number,
        "height": number,
        "unit": "pixel"
      },
      "confidence": number (0.0 to 1.0),
      "characteristics": ["string"]
    }
  ],
  "overlap_regions": [
    {
      "region_ids": ["string", "string"],
      "overlap_type": "handwritten_over_printed | printed_over_handwritten | adjacent",
      "notes": "string"
    }
  ],
  "uncertain_regions": [
    {
      "region_id": "string",
      "reason": "string",
      "suggested_review": "string"
    }
  ]
}

[CONSTRAINTS]
1. Classify each distinct text region independently. Do not merge regions with different classifications.
2. For overlapping annotations (e.g., handwriting on top of printed text), include both regions and add an entry to `overlap_regions`.
3. Set `confidence` below 0.7 for any region where classification is ambiguous. Add such regions to `uncertain_regions` with a specific reason.
4. In the `characteristics` array, include relevant observations such as `cursive`, `all_caps`, `stylized_font`, `light_ink`, `colored_ink`, `margin_note`, `strikethrough`, `underline`, `circle`, `arrow`.
5. For stylized fonts that resemble handwriting (e.g., script typefaces), classify as `printed` and note `stylized_font` in characteristics.
6. For light handwriting on dark backgrounds, note `light_ink` and `low_contrast` in characteristics. Flag in `uncertain_regions` if contrast is below readable threshold.
7. Include margin notes, interlinear additions, and text outside the main content area.
8. Do not classify stamps, watermarks, or printed decorations as handwritten.

[EXAMPLES]
Example characteristics for handwritten regions:
- ["cursive", "margin_note", "blue_ink"]
- ["all_caps", "pencil", "light_ink"]
- ["mixed_case", "interlinear", "black_ink"]

Example characteristics for printed regions:
- ["serif_font", "bold"]
- ["stylized_font", "italic", "resembles_handwriting"]
- ["monospace", "small_size"]

[RISK_LEVEL]
Medium. Misclassification can cause data loss in downstream extraction pipelines. Route `uncertain_regions` to human review before final ingestion.

Adapt this template by replacing [DOCUMENT_IMAGE] with your image input method—base64-encoded data, a pre-signed URL, or a file path reference depending on your model's multimodal capabilities. Replace [PAGE_NUMBER] with the actual page index if processing multi-page documents. If your use case does not require overlap detection, remove the overlap_regions field from the output schema. For high-volume production pipelines, add a [BATCH_SIZE] parameter and adjust the output schema to handle multiple pages in a single request. Always validate the JSON output against the schema before passing results downstream; malformed bounding boxes or missing region_id values are common failure modes.

This prompt is designed to be called as a single step within a larger document processing pipeline, not as a standalone chat interaction. The primary integration point is after page-level OCR and layout analysis have completed but before structured field extraction. The pipeline should pass the pre-processed page image (or a cropped region) along with the OCR text layer and bounding boxes as [CONTEXT]. The model receives both visual and textual evidence, which allows it to disambiguate cases where OCR has merged handwritten and printed text into a single line. The prompt expects a structured JSON output conforming to [OUTPUT_SCHEMA], which downstream field extraction and validation steps will consume.

Wire the prompt into an application by wrapping it in a thin service function that handles image encoding, schema validation, and retry logic. The function should accept a page image (base64-encoded PNG or JPEG), an OCR result object containing word-level bounding boxes and text, and an optional region-of-interest mask. Before calling the model, validate that the image resolution meets a minimum threshold (e.g., 150 DPI) and that the OCR confidence scores are available for each word. After receiving the model response, validate the JSON against the expected schema: each region must have classification (one of handwritten, printed, mixed), extracted_text, bounding_box (four normalized coordinates), and confidence. Reject responses with malformed coordinates or missing required fields and trigger a retry with the validation error message appended to [CONSTRAINTS]. For high-stakes workflows such as legal document review or medical form processing, route any region with confidence below 0.85 or classification of mixed to a human review queue before downstream extraction proceeds.

Model choice matters here. Use a multimodal model with strong vision-language capabilities (e.g., GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro) because the task requires joint reasoning over pixel-level handwriting cues and OCR text. Avoid text-only models even if provided with OCR output; they cannot detect faint pencil marks, erasures, or overlapping ink colors that are visually obvious. Set temperature to 0 or a very low value (0.1) to maximize output consistency across repeated runs on the same document. Log every request and response, including the raw image hash, OCR input, model output, validation result, and any retry attempts. This audit trail is essential for debugging misclassifications—particularly the known failure mode where stylized printed fonts (e.g., cursive or script typefaces) are misclassified as handwritten. When you observe this failure in production, add the misclassified region to a few-shot example set and include it in [EXAMPLES] for future calls.

Do not use this prompt for real-time user-facing annotation tools that require sub-second latency; the multimodal model call plus validation adds 2–5 seconds per page. Instead, run it as an asynchronous preprocessing step whose results are cached and served to the UI. If your pipeline processes documents with known templates (e.g., standard government forms), consider pre-computing the printed text regions from the template and skipping the model call for those areas entirely—use the prompt only for regions where annotations are expected. Finally, pair this prompt with the Form Field Completeness Audit Prompt downstream: once you have separated handwritten annotations from printed text, the audit prompt can verify that all required handwritten fields (signatures, dates, initials) are present and legible before the document proceeds to the next stage.