Form Field Extraction with Bounding Box Prompt

This prompt is for engineering teams building structured extraction pipelines that require spatial grounding. Use it when you must extract form fields with their content, labels, types, states, and precise bounding box coordinates across multi-page documents. The prompt produces field-level extraction with coordinate normalization, making it suitable for downstream validation, human review routing, and integration with document annotation systems. The ideal user is a developer or ML engineer who already has page images or rendered pages with a coordinate system available and needs to move beyond simple key-value extraction into layout-aware field mapping.

The required context for this prompt includes page images with a known coordinate space (e.g., pixel dimensions at a specific DPI), a defined schema for the fields you expect to extract, and a clear understanding of which field states matter for your workflow (filled, empty, checked, signed). You should also know your coordinate normalization strategy upfront—whether you need coordinates normalized to [0,1] for resolution independence, or absolute pixel values tied to a specific rendering. The prompt assumes you can provide either the image directly to a multimodal model or a structured representation of the page layout with text and spatial information. Without this spatial context, the prompt cannot produce bounding boxes, and you should use a simpler text-only extraction approach.

Do not use this prompt for simple key-value extraction without spatial requirements, or when the document has no visual layout to reference (e.g., plain text transcripts, markdown files, or linearized PDF text streams). It is also the wrong tool when you only need field content without location data—the bounding box computation adds latency and token cost that provides no value if coordinates are unused. If your documents are entirely digital forms with tagged PDF structure and reliable logical hierarchy, consider using a direct structural parser instead, reserving this prompt for cases where visual layout is the ground truth. Before implementing, verify that your evaluation pipeline can measure bounding box accuracy on rotated, skewed, and irregular form layouts, as spatial extraction failures are often silent and propagate into downstream field association errors.

This prompt template is designed for extraction pipelines that need more than raw text—they require spatial grounding. It instructs the model to return each form field's content, label, type, state, bounding box coordinates, and page number. The square-bracket placeholders let you inject your specific document content, output schema, and operational constraints without rewriting the core instruction set. Use this template when you need field-level extraction that downstream systems can map back to pixel locations on the original document image.

textCopy
You are a precise document form field extractor. Your task is to analyze the provided document page and extract every form field present.

For each field you identify, return a structured object containing:
- `field_id`: A unique identifier for the field on this page (e.g., "field_01").
- `label`: The text of the question or prompt associated with the field. If no label is visually present, use null.
- `content`: The user-entered or pre-filled text/value within the field. If the field is empty, use null.
- `type`: The visual type of the field. Must be one of: "text", "checkbox", "radio", "signature", "dropdown", "list", "stamp", "barcode", "other".
- `state`: The completion state. Must be one of: "filled", "empty", "checked", "unchecked", "indeterminate".
- `bounding_box`: The spatial coordinates of the field in the format [x1, y1, x2, y2] where (x1,y1) is the top-left corner and (x2,y2) is the bottom-right corner. All coordinates must be normalized to a range of 0.0 to 1000.0, representing a percentage of the page width and height.
- `page_number`: The page number where the field appears, starting from 1.
- `confidence`: A score from 0.0 to 1.0 indicating your confidence in the extraction.

[OUTPUT_SCHEMA]

Return the extracted fields as a single JSON object with a key "fields" containing an array of field objects.

[CONSTRAINTS]
- Do not extract purely decorative elements, watermarks, or background graphics as fields.
- If a field's content is illegible, set `content` to null and `confidence` to a value below 0.5.
- For checkboxes and radio buttons, the `content` field should be null, and the `state` should reflect its visual status.
- Normalize all bounding box coordinates to a 0-1000 scale relative to the page dimensions.

[INPUT]
The document page image to analyze is provided below.

To adapt this template, replace the placeholders with your specific requirements. The [OUTPUT_SCHEMA] placeholder is critical for integration: replace it with a strict JSON Schema definition that matches your application's data contract. The [CONSTRAINTS] section should be updated to reflect your domain-specific rules, such as ignoring certain form regions or handling custom field types. The [INPUT] placeholder is where your application will inject the base64-encoded image or a reference to the document page. For high-stakes workflows, always pair this prompt with a post-processing validation step that checks bounding box ranges, required field presence, and confidence thresholds before the data enters your system of record.

Integrating the Form Field Extraction with Bounding Box prompt into a production system requires more than a single API call. The prompt expects a pre-processed page image and a JSON schema as input, and it returns structured field data with spatial coordinates. The application layer must handle image preparation, coordinate normalization, output validation, and routing for low-confidence results. This section covers the implementation harness needed to make the prompt reliable at scale, including model selection, validation logic, retry strategies, and human review integration.

Input Preparation and Model Selection: Before calling the prompt, convert each document page into a high-resolution image (at least 200 DPI for forms with small fields). Pass the page image and the target [OUTPUT_SCHEMA] as inputs. For model selection, use a vision-capable model that supports structured output and bounding box prediction. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro are strong defaults. If processing sensitive documents, prefer models available in private deployments or with zero-data-retention policies. For high-volume pipelines, implement a model router that sends straightforward forms to a faster, cheaper model and escalates complex layouts (dense fields, rotated text, irregular grids) to a more capable model.

Output Validation and Coordinate Normalization: The prompt returns field-level bounding boxes in page-relative coordinates. Validate every output against the expected schema before ingestion. Check that all required fields are present, field types match the schema, and bounding box coordinates fall within page dimensions. Normalize coordinates to a consistent origin (top-left, in pixels or normalized 0-1 range) across all pages in a multi-page document. If the document was deskewed or rotated during preprocessing, apply the inverse transformation to map coordinates back to the original page. Implement a validation layer that flags outputs with missing required fields, coordinates outside page bounds, or type mismatches. Log every validation failure with the page image and raw model output for debugging.

Retry and Escalation Logic: When validation fails, implement a retry loop with a maximum of two additional attempts. On retry, include the validation error messages in the prompt context so the model can self-correct. If the model consistently fails on a specific field, escalate to a fallback strategy: try a different model, apply a specialized single-field extraction prompt, or route to human review. For low-confidence fields (confidence below a configurable threshold, typically 0.7), flag them for human review rather than silently ingesting uncertain data. Build a review queue that presents the original page image with the extracted field highlighted, the model's output, and the confidence score. Human reviewers can accept, correct, or reject each flagged field, and their corrections should be logged as training data for future fine-tuning.

Logging, Monitoring, and Continuous Improvement: Log every extraction request with the document ID, page number, model used, latency, token count, validation results, and any human corrections. Monitor extraction accuracy over time by sampling outputs and comparing against ground truth. Track failure modes: which field types produce the most validation errors, which layouts cause bounding box drift, and which documents require the most human review. Use this data to refine the prompt, adjust confidence thresholds, and build few-shot examples for edge cases. When you accumulate enough human-corrected examples, consider fine-tuning a smaller vision model on your specific form types to reduce cost and latency while maintaining accuracy.