Cosine Similarity Guard

The guard operates by converting both the reference source text and the generated output into high-dimensional vector embeddings using a sentence transformer model. It then calculates the cosine of the angle between these two vectors. A score of 1 indicates identical semantic direction, while 0 indicates orthogonality (no relationship). This mathematical approach captures contextual meaning rather than just keyword overlap.

A critical configuration parameter is the similarity threshold (e.g., 0.85). If the calculated cosine similarity score falls below this predefined value, the guard triggers a blocking action. Typical actions include:

• Hard Failure: The output is discarded, and an error is returned.
• Retry with Context: The generation pipeline is re-invoked with stricter instructions.
• Human Review Queue: The flagged content is routed to a Dead Letter Queue for manual inspection.

This guard directly combats hallucination rate by enforcing a strict grounding score requirement. Unlike an entailment check which verifies logical inference, cosine similarity provides a holistic semantic distance measure. It is highly effective at catching 'drift' where the model adds plausible but unsupported details, ensuring the output remains within the bounds of the provided source material.

The primary cost is the real-time generation of embeddings for both the source and the output. This introduces latency proportional to the model size and sequence length. Optimization strategies include:

• Caching: Pre-computing and storing embeddings for static reference documents.
• Model Distillation: Using a smaller, faster student model for the similarity check.
• Batching: Grouping multiple validation requests for parallel processing on a GPU.

Cosine similarity measures overall topical alignment but is not a substitute for a faithfulness metric. It can fail to detect subtle factual inversions (e.g., 'The cat sat on the mat' vs. 'The mat sat on the cat') if the vector direction remains similar. A semantic drift monitor should be used in conjunction to track gradual shifts in topic distribution over large-scale generation runs.

This guard is typically implemented as a post-generation validation step within a Policy-as-Code framework. It acts as a circuit breaker in the data flow: if the similarity score is too low, the pipeline halts to prevent ungrounded content from reaching the end-user. It is a key component of Continuous Compliance Monitoring for automated content systems.

A quantitative metric that evaluates how well a generated statement is supported by a specific source document or verified knowledge base. Unlike cosine similarity—which measures semantic closeness—a grounding score explicitly verifies factual anchoring by checking entailment relationships between the output and the reference text. A high grounding score indicates claims are directly inferable from the provided context, making it essential for Retrieval-Augmented Generation (RAG) architectures where hallucination prevention is critical.

A Natural Language Inference (NLI) task that determines whether a hypothesis statement logically follows from a given premise. In content guardrails, entailment checks serve as binary validators: does the generated sentence contradict, neutralize, or entail the source? This is more rigorous than cosine similarity because it evaluates logical consistency rather than vector proximity. Deployed as a secondary filter after semantic similarity checks, it catches subtle factual distortions that similar embeddings might miss.

A composite score measuring the degree to which generated summaries or answers contain only claims directly inferable from the source document. Key components include:

• Precision: Proportion of generated claims found in source
• Recall: Proportion of source claims captured in output
• Hallucination Rate: Frequency of unsupported statements Faithfulness metrics provide a more granular alternative to cosine similarity by decomposing semantic alignment into auditable sub-scores, enabling targeted debugging of content generation pipelines.

A system that tracks the gradual shift in meaning or contextual relevance of generated content over time. While cosine similarity guards provide instantaneous threshold checks, semantic drift monitors analyze embedding trajectories across multiple generations to detect topic divergence. This is critical for long-running autonomous agents that may slowly deviate from their original intent. Alerts trigger when the cumulative vector distance from the initial reference exceeds a configurable threshold.

A monitoring component that identifies input data points significantly different from the model's training distribution. When combined with cosine similarity guards, OOD detectors provide a complementary safety layer: similarity checks validate output against references, while OOD detectors flag inputs that may cause unpredictable model behavior. Techniques include:

• Mahalanobis distance in feature space
• Reconstruction error from autoencoders
• Energy-based models for density estimation

A metric measuring the alignment between a model's predicted confidence and its actual empirical accuracy. A well-calibrated model that outputs 0.9 confidence should be correct 90% of the time. In the context of cosine similarity guards, calibration scores validate whether the similarity threshold itself is trustworthy—ensuring that a 0.85 similarity score reliably corresponds to acceptable output quality rather than providing a false sense of security.