Guardrails

Intercepts and validates user messages before they reach the LLM. This layer performs canonicalization (standardizing input format), jailbreak detection (identifying adversarial prompts designed to bypass safety rules), and topic classification to route or block off-policy queries. For example, a customer support bot can use an input rail to detect and reject prompt injection attempts like 'Ignore all previous instructions' before the model ever processes them.

Scrutinizes the LLM's raw output before it is returned to the user. This rail enforces factual consistency by checking generated claims against a knowledge base, applies formatting schemas (e.g., ensuring valid JSON or stripping markdown), and executes toxicity classification to block harmful content. A common use case is a dialog rail that forces the model to respond in a strict 'Yes/No' format, re-prompting the LLM if the output deviates.

Governs the multi-turn flow of a conversation by maintaining a state machine. It defines canonical paths a dialogue can take, preventing the model from going off-topic or violating a predefined conversation script. For instance, a dialog rail can enforce that a banking bot must complete identity verification steps in a fixed sequence before disclosing account details, automatically steering the user back if they attempt to skip ahead.

A specialized output rail that performs evidence extraction and entailment scoring against a trusted document store. It breaks the generated text into atomic claims, retrieves relevant evidence, and uses a Natural Language Inference (NLI) model to verify each claim. If a claim is contradicted or unsupported, the rail can either mask it, ask the model to regenerate, or append a disclaimer. This is critical for grounding RAG systems in high-stakes domains like medicine or law.

Controls and validates any function calls or API executions the LLM requests. This rail ensures the model only calls permitted tools with valid parameters, blocking unauthorized actions like sending emails or querying restricted databases. It acts as a least-privilege enforcement layer, sanitizing arguments before they hit an API and validating the response. For example, it can prevent a model from executing a SQL query that contains a DROP TABLE statement, even if hallucinated.

A modeling language used in frameworks like NeMo Guardrails to define flows, rails, and actions in a human-readable format. Colang allows developers to write declarative rules (e.g., 'define user ask about politics -> bot refuse to respond') without writing procedural code. This separates the safety logic from the application code, enabling non-engineers to audit and modify AI behavior. It supports defining canonical forms, detecting user intent, and orchestrating complex multi-step interactions.

Guardrails prevent conversational AI from discussing competitors, making financial promises, or responding to off-topic queries. A topical guardrail restricts the bot to approved support domains, while a safety guardrail blocks toxic language and prompt injection attempts.

• Input rails: Sanitize user messages before they reach the LLM
• Output rails: Validate responses against company policy before display
• Dialog rails: Enforce multi-turn conversation flows like account verification

Medical AI assistants use factual consistency guardrails to prevent fabricated diagnoses. A retrieval-augmented guardrail cross-references every generated statement against an approved medical knowledge base, blocking unverified claims.

• Hallucination detection: Flags outputs not grounded in medical literature
• Scope limitation: Redirects emergency symptoms to human triage
• PII masking: Automatically redacts patient identifiers from logs

Regulated financial institutions deploy guardrails to ensure AI-generated advice complies with SEC and FINRA regulations. A compliance rail scans outputs for prohibited language like guaranteed returns or specific investment recommendations.

• Jurisdictional routing: Applies region-specific regulatory rules
• Disclaimer injection: Automatically appends required legal text
• Sentiment guardrail: Blocks overly optimistic or pessimistic language

Internal developer tools use guardrails to prevent code generation that introduces security vulnerabilities. A code safety rail blocks patterns like SQL injection, hardcoded credentials, or deprecated API calls before the suggestion reaches the developer.

• License compliance: Flags code snippets matching proprietary repositories
• Secret detection: Prevents generation of API key patterns
• Sandbox validation: Executes generated code in isolated environments

Social platforms and publishing tools implement multi-layered guardrails to enforce community standards at scale. Self-checking rails use the same LLM to critique its own draft output before publication, identifying policy violations.

• Jailbreak resistance: Detects and blocks adversarial prompt patterns
• Multimodal scanning: Applies safety checks to generated images and text
• Appeal routing: Flags borderline cases for human review with context

AI tutors employ pedagogical guardrails that prevent simply giving away answers while ensuring age-appropriate content. The system enforces Socratic questioning patterns and blocks responses outside the curriculum scope.

• Difficulty calibration: Adjusts explanation complexity to student level
• Safety gating: Blocks personally identifiable information requests
• Progress tracking: Ensures prerequisite concepts are mastered before advancing

A metric quantifying the frequency at which a language model generates factually incorrect, nonsensical, or unfaithful information not supported by its training data or provided context. Guardrails directly target this metric by intercepting and blocking outputs that exceed a defined hallucination threshold. Key measurement approaches include:

• Human evaluation against source documents
• NLI-based automated scoring using entailment models
• Factual consistency checks against retrieved evidence

An evaluation score that measures the degree to which a generated text is factually consistent with and can be directly inferred from the provided source document or context. Guardrails use faithfulness metrics as a real-time gating function: if a generated sentence cannot be entailed by the retrieved evidence, the guardrail blocks or rewrites it. This is distinct from relevance—a faithful answer can be irrelevant, but an unfaithful answer introduces fabricated facts.

A method for reducing hallucination where a language model first drafts a response, then generates a series of independent verification questions to fact-check its own initial output, and finally produces a corrected answer. Guardrails can implement CoVe as a programmatic loop:

• Step 1: Generate initial response
• Step 2: Extract factual claims
• Step 3: Verify each claim against a knowledge base
• Step 4: Regenerate with corrections This self-critique pattern is a core guardrail design pattern.

The automated identification of statements within a text or between a text and an evidence source that are logically incompatible. Guardrails deploy contradiction detection as a safety net: if a model's output contradicts a previously established fact, company policy, or retrieved document, the guardrail flags or suppresses it. This relies on Natural Language Inference (NLI) models trained to classify premise-hypothesis pairs as entailment, neutral, or contradiction.

A numerical value, often derived from token probabilities or logit outputs, representing a model's internal estimate of the likelihood that its generated output is correct. Guardrails consume confidence scores as a routing signal: low-confidence outputs can be escalated to human review, while high-confidence outputs pass through. However, raw confidence scores are often poorly calibrated—guardrails must apply calibration techniques to make them reliable decision thresholds.