Guardrails

Intercept and sanitize user prompts before they reach the LLM to prevent prompt injection, jailbreaking, and off-topic queries.

• Jailbreak detection: Pattern-matching and classifier-based filters that block known attack vectors like DAN prompts or token smuggling
• Topic restriction: Semantic similarity checks that reject queries outside approved domains
• PII redaction: Automated masking of personally identifiable information before inference
• Canonicalization: Normalizing inputs to a standard format to reduce variability

Example: A customer support bot rejects prompts containing ignore previous instructions or attempts to discuss competitor products.

Validate and correct LLM responses after generation but before delivery to the user, catching hallucinations and policy violations.

• Factual consistency checks: Compare generated claims against a grounding knowledge base using NLI-based evaluation
• Regex and structural validation: Enforce output formats like valid JSON, phone numbers, or specific schemas
• Toxicity and safety classifiers: Secondary models that score outputs for harmful content
• Hallucination detection: Methods like SelfCheckGPT that sample multiple responses to identify stochastically unstable facts

Example: A medical summarization system discards any generated statement not entailed by the patient record.

Control the flow and state of multi-turn conversations, preventing the LLM from derailing into unsafe or irrelevant territory.

• State machine enforcement: Define allowed conversational states and transitions, blocking the bot from revisiting completed steps
• Context window management: Prune or summarize conversation history to prevent context overflow attacks
• Fallback policies: Graceful degradation to human handoff or canned responses when confidence is low
• Rate limiting and circuit breakers: Prevent abuse and cascading failures in high-load scenarios

Example: A banking assistant enforces a strict authentication → inquiry → resolution → confirmation flow with no ability to skip steps.

Ground LLM outputs in trusted, external data sources in real-time to prevent factual drift and enable auditable citations.

• Dynamic fact-checking: Query a vector database or knowledge graph for each generated claim and flag contradictions
• Citation injection: Automatically append source attributions to generated text using Citation Recall and Citation Precision metrics
• Grounding score thresholds: Reject responses that fall below a minimum alignment score with retrieved documents
• Canonical entity resolution: Link named entities to unique knowledge base IDs to prevent entity-level hallucinations

Example: A legal research tool verifies every case citation against a validated court opinion database before displaying it.

Define guardrails as code using domain-specific languages or configuration files, enabling version control, testing, and CI/CD integration.

• Colang (NVIDIA NeMo Guardrails): A modeling language for defining dialog flows, safety rails, and fact-checking logic
• YAML/JSON-based rules: Declarative policies that non-ML engineers can audit and modify
• A/B testing of constraints: Deploy multiple guardrail configurations to measure hallucination rate impact
• Hot-reloading: Update safety policies without restarting inference services

Example: A compliance team updates prohibited topics in a YAML file, and the change takes effect in production within seconds.

Use Uncertainty Quantification (UQ) signals to dynamically activate guardrails only when the model's confidence is low, minimizing latency overhead.

• Semantic entropy thresholds: Trigger fact-checking only when multiple sampled responses diverge in meaning
• Expected Calibration Error (ECE) monitoring: Track miscalibration in production and adjust intervention thresholds
• Epistemic vs. aleatoric uncertainty: Distinguish between correctable knowledge gaps and irreducible data noise
• Selective verification: Apply expensive verification steps only to high-risk or low-confidence outputs

Example: A financial report generator runs full fact-checking on revenue figures (high-risk) but skips verification of boilerplate disclaimers.

A core metric evaluating whether all factual claims in a generated text are supported by a source document. It measures the alignment between the output and the provided grounding context.

• Natural Language Inference (NLI): Often used to automate this check by classifying the relationship between source and summary as entailment, contradiction, or neutral.
• Guardrail Integration: A factual consistency check is a primary validation step in a guardrails framework to prevent hallucination.

A metric evaluating whether a model can correctly link a generated claim to the specific segment of a source document that supports it. It is critical for verifiable generation.

• Citation Recall: The proportion of generated claims that are supported by a cited source.
• Citation Precision: The proportion of citations that actually support the corresponding claim.
• Guardrail Role: Guardrails enforce attribution by requiring a valid citation before a statement is allowed to pass to the user.

The practice of evaluating AI models against inputs specifically designed to cause misclassification, hallucination, or jailbreaking. This is the offensive security counterpart to defensive guardrails.

• Prompt Injection: Crafting inputs that override system instructions.
• Jailbreaking: Bypassing safety alignment to produce restricted content.
• Data Poisoning: Corrupting training data to create backdoors.
• Guardrails must be continuously tested against these evolving attack vectors.

A distribution-free, model-agnostic statistical framework that generates prediction sets with a rigorous mathematical guarantee of containing the true output at a user-specified confidence level.

• Uncertainty Quantification: Provides a formal basis for a guardrail to decide when a model is too uncertain to answer.
• Abstention Logic: If the prediction set is too large or empty, the guardrail can trigger a fallback response instead of risking a hallucination.