Filtered Search

A Filtered Search query is fundamentally a two-part operation: a vector similarity search (e.g., k-NN) and a metadata filter (e.g., user_id = 123 AND status = 'active'). The database's query planner must decide the optimal order of operations—applying the filter before, after, or interleaved with the vector search—to satisfy both the semantic intent and the logical constraints efficiently. This execution directly impacts the final recall and query latency.

Pre-filtering applies the metadata conditions first to create a filtered subset of the dataset, then performs the vector similarity search only within that subset.

• Advantage: Guarantees all results satisfy the filter; efficient for highly selective filters that drastically reduce the search space.
• Disadvantage: Can miss highly relevant vectors that are excluded by the initial filter, potentially causing a significant drop in recall. Performance degrades if the filter is not backed by an efficient secondary index (e.g., B-tree).

Post-filtering performs the approximate nearest neighbor search first, retrieving a candidate set (often larger than k), then applies the metadata filters to the candidates.

• Advantage: Preserves high recall from the vector search; ideal for non-selective filters.
• Disadvantage: May return fewer than k results (or even zero) if the filter is highly selective, requiring the system to perform a fallback search. This can lead to unpredictable latency and requires careful tuning of the initial candidate pool size.

Advanced vector databases implement single-stage execution where filters are integrated directly into the vector index traversal. For graph-based indexes like HNSW, this means pruning graph connections during search based on metadata. For inverted indexes like IVF, it involves searching only within list segments that satisfy the filter. This approach minimizes the trade-off between recall and filter adherence, providing more deterministic P99 latency than pre- or post-filtering.

To optimize Filtered Search, the underlying vector index is often augmented with secondary metadata indexes. Common patterns include:

• Composite Indexing: Co-locating vector data with its metadata on disk (e.g., within the same IVF list) to reduce random I/O during filtered searches.
• Bitmap Indexes: Creating bitmaps for common filter fields (e.g., tenant_id, category) to enable extremely fast set intersection operations during candidate generation.
• Partitioning/Sharding: Physically partitioning data by a key filter attribute (e.g., by customer_id) to naturally isolate search scopes.

The system's query planner dynamically selects the optimal execution strategy based on:

• Filter Selectivity: The estimated fraction of the dataset that will pass the filter.
• Index Availability: Presence of secondary indexes on filter attributes.
• Query Parameters: The required k and the search radius.
• System Load: Current resource utilization. The planner may use heuristics or cost-based models to choose between pre-filtering, post-filtering, or a single-stage approach, directly impacting throughput (QPS) and overall system efficiency.

Filtered search is the core mechanism for user-specific recommendations. A query finds items similar to a user's preferences, but metadata filters ensure results are relevant to the current context.

• Example: "Find movies like Inception that are available in my region and rated PG-13."
• The vector search identifies films with similar themes and style.
• Filters on region_availability and content_rating metadata apply business logic, ensuring the results are actionable for the specific user.

In Retrieval-Augmented Generation systems for enterprises, filtered search enforces data security and relevance.

• Example: An employee asks, "What were our Q3 sales figures?"
• The system performs a vector search for semantically relevant documents.
• A critical pre-filter on user_department = "Sales" AND document_access_tier <= user_clearance_level ensures the LLM only retrieves and reasons over documents the user is authorized to see, preventing data leaks.

Enables 'search within results' for online catalogs by blending semantic understanding with hard product attributes.

• Example: A customer searches for "comfortable running shoes for long distances."
• The vector query finds items with descriptions semantically close to the query.
• Post-filters on category = "Running Shoes", price <= 150, and color = "blue" refine the results to match exact requirements. This provides a better experience than keyword-only search, which cannot understand intent like "comfortable for long distances."

Essential for software serving multiple customers (tenants) from a single database instance. It guarantees absolute data isolation.

• Every search query is automatically scoped with a tenant_id filter.
• Example: A user in tenant_acme searches their support tickets for "login error."
• The system performs a vector similarity search on ticket content but always appends tenant_id = "acme" to the filter. This prevents data from tenant_xyz from ever being retrieved, a fundamental security requirement.

Retrieves relevant information within a specific time window or document version, common in legal, financial, and content management systems.

• Example: A journalist researches "company mergers in the tech sector."
• A vector search finds relevant news articles.
• A range filter publication_date BETWEEN '2023-01-01' AND '2023-12-31' ensures results are from the relevant fiscal year.
• Another use case: filtering vector search to document_version = "latest_approved" to avoid retrieving outdated or draft content.

Monitors streaming data by comparing new entries to a history of 'normal' patterns, but only within a specific operational context.

• Example: Detecting fraudulent financial transactions.
• Each new transaction is converted to a behavioral embedding.
• A filtered search finds similar historical transactions, but with filters for account_id = current_account and transaction_time >= NOW() - INTERVAL '1 hour'.
• If the similarity to recent, legitimate transactions is low, an alert is triggered. The filters ensure the comparison is context-aware.

The two primary execution strategies for combining vector search with metadata filters, representing a fundamental latency-recall trade-off.

• Pre-Filtering: Applies metadata filters first to create a subset, then performs ANN search within it. Optimal when filters are highly selective, as it reduces the search space dramatically. Risk: can miss relevant vectors excluded by the initial filter.
• Post-Filtering: Performs a broad ANN search first, then applies metadata filters to the results. Preserves recall on the vector similarity aspect. Risk: may return too few final results if the initial candidates don't match the filter, requiring a larger and slower initial k.
• Modern Approach: Advanced databases use query planning to dynamically choose or blend these strategies based on filter selectivity and index statistics.

The two fundamental query types that Filtered Search modifies. They define how similarity is measured and results are bounded.

• k-NN Search (k-Nearest Neighbors): Retrieves a fixed number (k) of the most similar vectors to the query. Example: "Find the 10 most similar product embeddings." Filtered Search often executes as a k-NN search within a filtered subset.
• Range Search: Retrieves all vectors within a specified distance radius (epsilon) from the query vector. Example: "Find all document embeddings within a cosine similarity of 0.8." Filtered Search can use range search when the application logic requires a similarity threshold.
• Hybrid Use: A filtered k-NN search is most common, as it provides predictable result set sizes for downstream processing.

The internal database process that determines the most efficient execution path for a Filtered Search query. It is critical for optimizing latency and resource use.

• Function: Analyzes the query's filter predicates, index types, and data statistics to choose between pre-filtering, post-filtering, or a hybrid approach.
• Relies On: Metadata selectivity estimation (how many records a filter will pass) and index performance profiles (the cost of searching different index types).
• Advanced Feature: In distributed databases, the query planner also handles shard pruning, directing the search only to shards that contain relevant data based on filter values.

The dual evaluation metrics for any search system, measuring completeness and accuracy. Filtered Search directly impacts both.

• Recall: The fraction of all relevant items (defined by ground truth) that are successfully retrieved. A pre-filtering strategy with very restrictive filters can severely harm recall if relevant items are excluded early.
• Precision: The fraction of retrieved items that are actually relevant. Post-filtering can hurt precision if the initial ANN search retrieves many irrelevant vectors that are later filtered out, wasting compute.
• Trade-off: Optimizing Filtered Search involves tuning parameters (like ef_search in HNSW or nprobe in IVF) and choosing a strategy to balance these metrics for the specific use case.

A broader paradigm that combines multiple search techniques. Filtered Search is a key subset of Hybrid Search.

• Definition: Integrates vector similarity search with keyword (full-text) search and/or metadata filtering.
• Typical Architecture: Uses a sparse vector (e.g., BM25) for keyword matching and a dense vector (embedding) for semantic matching. Scores are combined using weighted fusion (e.g., Reciprocal Rank Fusion).
• Relation to Filtered Search: Filtered Search (vector + metadata) is one component. Full Hybrid Search is vector + keyword + metadata. This provides a robust retrieval system capable of handling precise keyword matches, semantic intent, and business logic constraints simultaneously.