Filter Expression

Filter expressions are built using standard Boolean logic to combine multiple conditions. This includes:

• AND: All conditions must be true (status = 'active' AND price > 100).
• OR: At least one condition must be true (category = 'electronics' OR category = 'books').
• NOT: Inverts a condition (NOT department = 'archived').
• Parentheses: Used to explicitly group and control the order of evaluation, ensuring complex logic is executed correctly.

Expressions support operators to compare metadata values against the query input. Core operators include:

• Equality/Inequality: =, != (e.g., language = 'en').
• Numeric Comparison: <, <=, >, >= (e.g., timestamp >= 1710000000).
• Set Membership: IN checks if a value is in a list (user_id IN [101, 202, 303]).
• Range Queries: Often expressed with AND (e.g., age >= 21 AND age <= 65) or a dedicated BETWEEN operator if supported. These are essential for filtering by dates, scores, or other ordinal values.

Filters are applied to structured metadata fields associated with each vector. The expression must correctly reference the field's name and data type.

• Field Names: Typically defined during schema creation (e.g., product_id, created_at).
• Data Type Awareness: The expression syntax and operators must match the field type. Comparing a string field with a numeric operator (category > 10) will cause an error.
• Nested Fields: Some databases support querying into nested JSON or map structures using dot notation (e.g., metadata.user.role = 'admin').

The filter is applied within the Approximate Nearest Neighbor (ANN) search process. There are two primary execution strategies:

• Pre-filtering: The metadata filter is applied first to create a candidate set, and then the ANN search runs only on those filtered vectors. Guarantees precision but can be slow if the filter is very restrictive.
• Post-filtering: The ANN search runs first to find the top-K similar vectors, and then the filter is applied to those results. Very fast but may return fewer than K results if many are filtered out. Advanced databases use single-stage filtered search algorithms to optimize this trade-off.

While the core logic is universal, the concrete syntax varies by database. Common patterns include:

• SQL-like WHERE clauses: WHERE status = 'published' AND category IN ('ai', 'ml').
• JSON-based DSL: A structured JSON object defining the filter tree.
• Language-specific builders: Fluent interfaces in SDKs (e.g., collection.query.filter('price').lt(50).filter('in_stock').eq(true)). The expression is typically serialized into the request payload of the Query API call alongside the query vector and search parameters.

The efficiency of filtered search depends heavily on metadata indexing.

• Inverted Indexes: Built on metadata fields to enable fast lookups for equality and set membership checks.
• Composite Indexes: Combine vector and metadata indexes for optimal single-stage filtered search performance.
• Selectivity: The fraction of total vectors that pass the filter. Low-selectivity (restrictive) filters benefit from pre-filtering strategies, while high-selectivity filters work well with post-filtering. Performance degrades without appropriate metadata indexes, causing full collection scans.

A retrieval paradigm that combines vector similarity search with structured metadata filtering and often keyword search. It uses a filter expression to pre-filter or post-filter candidate vectors, ensuring results are both semantically relevant and meet precise business criteria (e.g., product_category = 'electronics' AND price < 100). This approach is fundamental for production RAG systems.

The process of narrowing vector search results based on structured attributes stored alongside embeddings. A filter expression defines the logical conditions (e.g., =, >, IN) applied to these metadata fields.

• Key-Value Pairs: Metadata is typically stored as scalar values (strings, numbers, booleans).
• Indexed Fields: For performance, metadata fields used in frequent filters are often indexed separately from the vector index.

The primary API endpoint for executing similarity searches. It accepts the query vector, distance metric, top_k parameter, and the filter expression as part of the request payload. The API orchestrates the interaction between the vector index and the metadata store to execute the hybrid search. Performance is measured in query latency and queries per second (QPS).

Two execution strategies for applying a filter expression:

• Pre-filtering: The filter is applied first to identify all vectors matching the metadata criteria, then a nearest neighbor search is performed only within that subset. Guarantees top_k filtered results but can be slow if the filter is broad.
• Post-filtering: A nearest neighbor search is performed first, then results are filtered. Faster initial search but may return fewer than top_k results after filtering. Advanced databases use conditional search to optimize this trade-off.

The logical structure of a filter expression, composed of operands (metadata field names and values) and operators. Supports:

• Comparison Operators: =, !=, >, <, >=, <=
• Logical Operators: AND, OR, NOT
• Set Operators: IN, NOT IN Expressions are evaluated to true or false for each vector's metadata. Complex nesting is often supported (e.g., (status = 'active' AND region IN ('US', 'EU')) OR is_premium = true).

The defined structure of a collection (or table) in a vector database, which dictates what metadata fields are available for use in filter expressions. The schema specifies:

• Field Name: The key used in the filter expression.
• Data Type: e.g., string, int64, float, bool. Determines which comparison operators are valid.
• Index Type: Whether the field is indexed for fast filtering. A strict schema ensures filter expressions are validated at query time, preventing errors.