Metadata Filtering

The execution strategy for applying filters dramatically impacts performance and accuracy.

• Pre-Filtering: Applies strict boolean conditions before the vector search. This guarantees all results match the filter but can be slow if the filter is not selective.
• In-Filtering: Integrates the filter into the Approximate Nearest Neighbors (ANN) graph traversal, checking metadata at each hop. This is faster for non-selective filters but may return fewer than the requested top_k results.

Metadata filtering relies on standard database operators applied to structured attributes.

• Equality: category == "legal_document"
• Range: date >= "2024-01-01"
• Set Membership: status IN ["active", "pending"]
• Boolean Logic: (source == "FDA") AND (risk_level > 7) These operators allow the construction of complex, precise scoping rules.

To avoid a full scan, metadata fields used in filters must be indexed.

• Single Field Indexes: Create a dedicated index on frequently filtered attributes like timestamp or user_id.
• Compound Indexes: Combine multiple fields into a single index for queries like (tenant_id, document_type).
• Selectivity Matters: Indexes on high-cardinality fields (e.g., user_id) are far more effective than low-cardinality fields (e.g., is_active).

A logical isolation technique where documents are assigned to separate, non-overlapping namespaces based on a primary metadata key.

• Multi-Tenancy: Assign each client a unique namespace to ensure absolute data isolation without complex filter logic.
• Performance: Searching within a single namespace is faster than applying a tenant filter across a global index, as the search space is physically constrained from the start.

An advanced pattern where a Large Language Model (LLM) translates a natural language query into a structured query object.

• Mechanism: The LLM extracts both a semantic search string and a metadata filter from the user's input.
• Example: "Show me high-priority incidents from last week" becomes query: "system outage" and filter: { priority: "high", date: { $gte: "2024-05-20" } }.
• Benefit: This bridges the gap between human conversational queries and machine-executable structured filters.

The effectiveness of a filter is defined by its selectivity.

• High Selectivity: A filter like user_id == "12345" that matches a tiny fraction of the dataset is highly efficient.
• Low Selectivity: A filter like language == "en" that matches 95% of the data provides minimal performance gain.
• Best Practice: Combine a low-selectivity filter with a high-selectivity one to quickly prune the search space before executing the vector similarity calculation.