Metadata Filtering

The practice of attaching permission metadata to individual text chunks rather than entire documents, enabling fine-grained access control in RAG systems.

• Each chunk inherits metadata from its parent document: { "doc_id": "acme-q3", "clearance": "confidential", "owner": "legal" }
• Vector stores like Weaviate and Qdrant support payload-based filtering directly on chunk embeddings.
• Critical for least privilege retrieval: a user may access the summary section of a document but not the financial projections chunk.

Transparently modifying a user's natural language query to inject metadata constraints that enforce data access policies during retrieval.

• Original query: "What is our Q3 revenue forecast?"
• Rewritten query with injected filters: filter: { "department": "finance", "clearance": { "$lte": "top-secret" } }
• Prevents prompt injection attacks where malicious retrieved content attempts to bypass authorization logic.
• Implemented at the Policy Enforcement Point (PEP) before the query reaches the vector store.

Access Control Lists attached directly to vector embeddings or their metadata, enforcing authorization at the granularity of the vector database index.

• Each embedding carries an ACL field: { "acl": ["user:alice", "group:engineering"] }
• The retrieval engine intersects the user's identity claims with the chunk's ACL before returning results.
• Supports ephemeral tokens and Just-In-Time (JIT) access where credentials are provisioned only for the duration of a single retrieval operation.

A security posture that re-evaluates access policies throughout a session rather than relying on a single authentication event.

• If a user's risk profile changes mid-session, retrieval rights are immediately revoked.
• Metadata filters are dynamically adjusted based on real-time signals: device posture, geolocation, time of day.
• Prevents data spillage where a compromised session token could be used to exfiltrate sensitive chunks after initial authentication.

The policy paradigm that directly powers metadata filtering. ABAC evaluates user attributes (department, clearance), resource attributes (document metadata tags), and environmental conditions (time, location) against a policy engine. In a vector database, this translates to dynamically generating a boolean filter clause—such as department == 'legal' AND classification != 'top-secret'—that is appended to the semantic search query.

A security strategy where the search space is restricted before the vector similarity search executes. By applying metadata filters upfront, the query only scans authorized vector partitions. This is highly efficient but requires strict alignment between the filter logic and the metadata schema.

• Pros: Prevents data leakage, reduces computational load.
• Cons: Requires precise index structuring; rigid filters can kill recall if too aggressive.

An authorization technique where the initial top-K results are re-ranked or redacted after the vector search completes. The system retrieves a broad set of candidates and then strips out documents the user is not permitted to see. This is often used as a safety net when metadata is inconsistent.

• Key Risk: If the initial K is too small, authorized documents might not be in the candidate set at all.

A defense-in-depth strategy combining pre-retrieval and post-retrieval filtering. The initial query is scoped using coarse-grained metadata filters, and the raw results are then passed through a fine-grained authorization layer. This minimizes the risk of exposing sensitive data while maintaining high recall for authorized content. It is the recommended pattern for zero-trust architectures.

The process of applying permission checks to individual text segments rather than whole documents. In a vector database, this means attaching an Access Control List (ACL) directly to the metadata of each chunk. Metadata filtering then enforces authorization at the granularity of the semantic index, ensuring a user sees only authorized paragraphs from a larger document.

The secure transmission of the end-user's authenticated context through the RAG pipeline. Without proper identity propagation, the retrieval engine cannot know who is asking the question and cannot apply user-specific metadata filters. This typically involves passing a JSON Web Token (JWT) or session cookie from the API gateway down to the vector database query layer.