Document Chunk Boundary Quality Assessment Prompt

This prompt is designed for search engineers and RAG developers who need to diagnose why a retrieval-augmented generation system produced a poor-quality answer, specifically when the root cause is suspected to be poor text splitting. The core job-to-be-done is to programmatically evaluate the quality of chunk boundaries from a production trace. Instead of manually reviewing hundreds of retrieved text blocks, you use this prompt to automatically flag chunks that start or end mid-sentence, split a numbered list, or sever a core concept like a definition from its term. The ideal user is an engineer inspecting a specific failing trace, not a data scientist running an offline batch evaluation over a static dataset.

To use this prompt effectively, you must provide the raw text of a sequence of retrieved chunks exactly as they appeared in the trace, preserving their order. The required context includes the chunk index and its full text. This prompt is not a replacement for embedding-based similarity checks or relevance scoring; it is a structural quality assessment. You should not use this prompt when the primary failure mode is irrelevant retrieval or when the chunks are already known to be entire documents. It is also inappropriate for real-time guardrails due to the latency of an LLM call; instead, run it as a diagnostic step during incident review or as a periodic audit on sampled production traces. The output is a structured boundary quality score and a list of specific violations, which you can log, trend over time, and use as evidence to adjust your chunking strategy's chunk_size and chunk_overlap parameters.

Before relying on this prompt, establish a baseline by running it on a set of manually labeled chunks to calibrate your tolerance for false positives. A chunk that begins with 'and' might be a stylistic choice rather than a true error. The prompt's value is in surfacing candidates for review, not in making an infallible judgment. After receiving the output, the next step is to correlate the boundary quality score with the final answer's faithfulness score from the same trace. If low boundary quality correlates with hallucination or omission, you have strong evidence to prioritize re-chunking your document store. Avoid using this prompt on chunks from a system where you cannot action the results, as it will only generate noise without a path to remediation.

The following prompt template is designed to be copied directly into your prompt management system, evaluation harness, or trace review tool. It accepts a production trace containing retrieved chunks and returns a structured boundary quality assessment. Every placeholder is enclosed in square brackets and must be replaced with real values before execution. The prompt assumes you have already extracted the retrieved chunks and their surrounding context from your trace data.

textCopy
You are a document structure analyst evaluating chunk boundary quality in a retrieval pipeline. Your task is to assess whether the boundaries of each retrieved chunk split coherent ideas, sentences, or concepts.

## INPUT

You will receive a production trace containing:
- A user query: [USER_QUERY]
- A set of retrieved document chunks with their boundary markers: [RETRIEVED_CHUNKS]
- Optional: surrounding context for each chunk (up to [SURROUNDING_WINDOW_SIZE] tokens before and after the chunk boundary): [SURROUNDING_CONTEXT]

## TASK

For each retrieved chunk, evaluate both the start boundary and end boundary against the following criteria:

1. **Sentence Integrity**: Does the boundary occur at a natural sentence break (period, question mark, exclamation point, or equivalent)?
2. **Concept Coherence**: Does the boundary split a named entity, technical term, multi-word expression, or logical unit?
3. **Syntactic Completeness**: Does the boundary leave a clause, phrase, or dependency unresolved?
4. **Semantic Closure**: Can the chunk's content be understood without reading the adjacent text across the boundary?

## OUTPUT SCHEMA

Return a JSON object with this exact structure:

{
  "trace_id": "[TRACE_ID]",
  "assessment_timestamp": "[TIMESTAMP]",
  "chunk_assessments": [
    {
      "chunk_id": "string",
      "chunk_text_preview": "string (first 100 chars)",
      "start_boundary": {
        "splits_sentence": true/false,
        "splits_concept": true/false,
        "evidence": "string (quote the text spanning the boundary)",
        "severity": "none|minor|moderate|severe"
      },
      "end_boundary": {
        "splits_sentence": true/false,
        "splits_concept": true/false,
        "evidence": "string (quote the text spanning the boundary)",
        "severity": "none|minor|moderate|severe"
      },
      "overall_boundary_quality_score": 0.0-1.0,
      "recommendation": "keep|adjust|split_differently|merge_with_neighbor"
    }
  ],
  "aggregate_metrics": {
    "total_chunks_assessed": 0,
    "chunks_with_severe_boundary_issues": 0,
    "average_boundary_quality_score": 0.0,
    "most_common_failure_mode": "string"
  }
}

## CONSTRAINTS

- Score boundary quality from 0.0 (both boundaries severely split ideas) to 1.0 (both boundaries are clean).
- A chunk that starts mid-sentence AND ends mid-sentence should score below 0.3.
- A chunk that starts at a sentence boundary but ends mid-concept should score between 0.4 and 0.7 depending on severity.
- Flag any boundary that splits a proper noun, date range, measurement with unit, or parenthetical expression as "splits_concept: true".
- If surrounding context is provided, use it to determine whether the boundary disrupts coherence. If not provided, assess based on the chunk text alone and note the limitation.
- Do not evaluate chunk content relevance to the query. Only assess boundary quality.
- If a chunk is the first or last chunk of a document, note this in the evidence field but still assess the available boundary.

## EXAMPLES

Example 1: Clean boundary
Chunk text: "The transformer architecture introduced self-attention mechanisms that process all tokens in parallel."
Start boundary: sentence start, no split. End boundary: sentence end, no split.
Score: 1.0

Example 2: Mid-sentence split
Chunk text: "mechanisms that process all tokens in parallel. The next major innovation"
Start boundary: splits "self-attention mechanisms" mid-phrase. End boundary: sentence start, but "The next major innovation" is incomplete without the following clause.
Score: 0.2

Example 3: Concept split
Chunk text: "achieved state-of-the-art results on the GLUE"
End boundary: splits the benchmark name "GLUE benchmark" mid-concept.
Score: 0.4

## RISK LEVEL: [RISK_LEVEL]

If RISK_LEVEL is "high", include a confidence field for each boundary assessment and flag any assessment where confidence is below 0.7 for human review.

To adapt this template for your pipeline, replace the placeholders with real trace data. The [RETRIEVED_CHUNKS] placeholder should receive a structured list of chunk objects, each containing at minimum a chunk ID, the chunk text, and optionally the surrounding context. If your trace system does not capture surrounding context, set [SURROUNDING_WINDOW_SIZE] to 0 and omit [SURROUNDING_CONTEXT]. The [RISK_LEVEL] placeholder controls whether the output includes confidence scores and human-review flags. Set it to high when boundary quality directly impacts answer accuracy in regulated or customer-facing workflows. Before deploying, validate that the output JSON matches the schema exactly using a JSON Schema validator, and run at least five known-clean and five known-broken chunk examples through the prompt to calibrate your scoring thresholds.

This prompt is designed to operate on a single production trace record containing the user query and the full text of each retrieved chunk. The implementation harness should extract these fields from your trace store—typically a logging database, observability platform, or vector database query log—and inject them into the [RETRIEVED_CHUNKS] and [USER_QUERY] placeholders. Because chunk boundary assessment is a diagnostic task rather than a user-facing feature, the harness should run as an offline batch job or a triggered analysis step after trace ingestion, not as part of the real-time retrieval pipeline. The output is a structured JSON object with per-chunk boundary scores and a summary quality rating, which your harness must parse and store back into the trace record or a separate quality metrics table for later aggregation and alerting.

Wire the prompt into a Python or TypeScript function that accepts a trace ID, fetches the trace payload, and calls your model endpoint with the rendered prompt. Validate the response against the expected output schema before accepting it: each chunk entry must include chunk_index, boundary_score (a float between 0.0 and 1.0), boundary_issues (an array of strings), and starts_mid_sentence / ends_mid_sentence (booleans). If the model returns malformed JSON, missing required fields, or scores outside the valid range, implement a retry loop with a maximum of two additional attempts, appending the validation errors to the retry prompt as [PREVIOUS_ERRORS]. After three consecutive failures, log the trace ID to a dead-letter queue for manual review and emit a warning metric. For model choice, use a model with strong structured output capabilities—GPT-4o, Claude 3.5 Sonnet, or equivalent—and set temperature=0 to maximize scoring consistency across traces.

Log every assessment result with the trace ID, model version, prompt version, timestamp, and the full boundary quality payload. This audit trail is essential for detecting scoring drift when you change chunking strategies or upgrade the assessment model. For high-volume production environments, consider batching multiple trace assessments into a single prompt call by providing an array of trace objects, but cap the batch size at 5-10 traces to avoid context window overflow and degraded assessment quality. The summary overall_boundary_quality field (e.g., 'good', 'fair', 'poor') should feed into a monitoring dashboard that tracks the percentage of traces with poor boundary quality over time. Set an alert threshold—for example, trigger an investigation when more than 15% of traces in a rolling window score 'poor'—so that chunking regressions are caught before they silently degrade retrieval quality. Avoid running this assessment on every trace indefinitely; sample traces strategically (e.g., 10% of production traffic or all traces flagged by a relevance failure detector) to manage cost while maintaining statistical coverage.