Novel Entity Detection Prompt for Knowledge Gaps

This prompt is designed for knowledge system engineers who need to identify when an input references entities, concepts, or facts that the system does not recognize. It belongs in a pipeline where a retrieval system or knowledge graph has already attempted entity linking, and the model acts as a secondary detector for coverage gaps. The ideal user is an MLOps engineer or knowledge base maintainer who needs structured, actionable output to prioritize knowledge base updates, not a general user performing entity extraction.

Use this prompt when you need structured output that flags novel entities, provides context for human reviewers, and generates actionable suggestions for knowledge base updates. It is most effective in batch processing or asynchronous review workflows where a human will triage the results. The prompt requires a pre-processed input that includes the original text and the results of a failed or low-confidence entity linking attempt. Do not use this prompt for general entity extraction where coverage is not the primary concern, or for real-time applications where latency prohibits a thorough gap analysis.

Before implementing this prompt, ensure your pipeline can provide the necessary context: the raw input text and the specific entities or concepts your primary system failed to resolve. The output is a structured JSON payload designed for ingestion into a review queue or knowledge base update tool. Avoid using this prompt as a standalone entity extractor; it is a diagnostic tool for system improvement, and its value is in the structured escalation and suggestion it provides, not in replacing your primary entity linking system.

This template is designed to be dropped into a knowledge system pipeline where the model must distinguish between known and unknown entities. It instructs the model to act as a strict novelty detector, comparing input text against a provided knowledge boundary and outputting a structured escalation payload when gaps are found. The prompt uses square-bracket placeholders that must be replaced with your specific domain taxonomy, knowledge boundary definitions, and output schema before use.

textCopy
You are a novelty detection agent for a knowledge management system. Your task is to analyze the provided text and identify any named entities, technical concepts, or factual claims that fall outside the defined knowledge boundary. Do not guess or infer beyond the boundary.

## KNOWLEDGE BOUNDARY
[KNOWLEDGE_BOUNDARY_DEFINITION]

## INPUT TEXT
[INPUT_TEXT]

## INSTRUCTIONS
1. Extract all named entities, domain-specific concepts, and factual claims from the input text.
2. For each extracted item, determine whether it is fully covered by the knowledge boundary above.
3. Classify each item as KNOWN or NOVEL.
4. For NOVEL items, provide:
   - The exact text span from the input
   - The entity or concept type (from the allowed taxonomy)
   - A confidence score (0.0 to 1.0) indicating how certain you are that this is genuinely outside the boundary
   - A brief diagnostic note explaining why it falls outside the boundary
5. If no novel items are found, return an empty list with a confirmation that all entities are known.

## ALLOWED ENTITY TYPES
[ENTITY_TYPE_TAXONOMY]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "analysis_timestamp": "ISO-8601 timestamp",
  "total_entities_extracted": integer,
  "known_entities": integer,
  "novel_entities": [
    {
      "text_span": "exact text from input",
      "entity_type": "type from taxonomy",
      "confidence_novel": float,
      "diagnostic_note": "reason for novelty classification",
      "suggested_canonical_form": "proposed normalized form or null",
      "context_snippet": "surrounding sentence for human review"
    }
  ],
  "escalation_recommended": boolean,
  "escalation_priority": "LOW | MEDIUM | HIGH | CRITICAL",
  "reviewer_notes": "summary for human reviewer or null"
}

## CONSTRAINTS
- Do not mark an entity as NOVEL if any synonym, abbreviation, or variant is covered by the knowledge boundary.
- If you are uncertain about novelty (confidence below 0.7), mark as NOVEL and note the uncertainty in the diagnostic note.
- For CRITICAL priority escalations, include a specific recommendation for immediate human review.
- Never fabricate knowledge boundary coverage to reduce escalation volume.

Before integrating this prompt, replace each placeholder with concrete, version-controlled content. [KNOWLEDGE_BOUNDARY_DEFINITION] should be a precise description of what the system knows—this could be a list of covered domains, a reference to an indexed corpus, or explicit entity lists. [ENTITY_TYPE_TAXONOMY] must be a closed set of allowed types (e.g., PERSON, ORGANIZATION, PRODUCT, TECHNICAL_TERM, REGULATORY_REFERENCE). [INPUT_TEXT] is the runtime text to analyze. After adapting the template, run it against a golden dataset of known entities and known gaps to verify that the model correctly distinguishes between the two before deploying to production.

The Novel Entity Detection prompt is not a standalone chat interaction—it is a classifier and escalation trigger that should sit inside a structured pipeline. The typical integration pattern is: (1) receive a text input or document, (2) run the prompt against a defined knowledge boundary or entity catalog, (3) parse the structured output, and (4) route any detected novel entities to a review queue or knowledge base update workflow. The prompt expects a [CONTEXT] block containing the text to analyze and an [ENTITY_CATALOG] or [KNOWLEDGE_BOUNDARY] block defining what the system already knows. The output should be a strict JSON schema with fields for detected_novel_entities, confidence_scores, context_excerpts, and escalation_recommendation. Do not treat this as a free-text generation task—the value comes from structured, machine-readable output that downstream systems can act on without human parsing.

Wiring the prompt into an application requires several guard layers. First, implement output validation using a JSON schema validator that checks for required fields, correct types, and enum compliance on escalation_recommendation (e.g., REVIEW, IGNORE, UPDATE_CATALOG). If validation fails, retry once with the same input and a repair instruction appended to the prompt. Second, add a confidence threshold gate: if any novel entity has a confidence score below 0.7, route the entire detection batch to a human review queue rather than auto-updating the knowledge base. Third, implement deduplication against existing review items to prevent the same novel entity from flooding the queue repeatedly. For model choice, use a model with strong instruction-following and JSON output discipline—GPT-4o, Claude 3.5 Sonnet, or equivalent. Avoid smaller models that may hallucinate entity boundaries or produce malformed JSON under the structured output constraint. If your entity catalog is large, consider chunking the input and running detection in parallel windows, then merging results with a deduplication pass.

Logging and observability are critical because this prompt directly influences knowledge base updates. Log every detection call with: input hash, model version, detected entities, confidence scores, escalation decision, and whether the detection resulted in a catalog update or human review. This creates an audit trail for debugging false positives and measuring drift in the model's entity recognition behavior over time. Set up an eval harness that runs weekly against a golden dataset of known novel entities and known in-catalog entities. Measure precision (did we flag only true novelties?), recall (did we miss any?), and escalation accuracy (did high-confidence true positives route to auto-update while ambiguous cases went to review?). If precision drops below 85% or recall below 90%, freeze automatic catalog updates and route all detections to human review until the prompt or model is recalibrated. Never allow this prompt to write directly to a production knowledge base without either a confidence gate or a human-in-the-loop checkpoint.