Index API

The primary function for instantiating a new searchable index. This involves specifying the index type (e.g., HNSW, IVF, DiskANN), the vector dimensionality, and the distance metric (e.g., cosine similarity, L2 Euclidean). Configuration parameters, such as the number of connections per node in an HNSW graph or the number of centroids for IVF, are set here to balance search speed, accuracy, and memory usage.

Initiates the computational process of organizing ingested vectors into the optimized search structure. For some algorithms like Inverted File (IVF), this is a training phase that analyzes a sample of data to find optimal partitions. This operation can be resource-intensive and is often performed asynchronously. The API provides endpoints to monitor build progress and finalize the index for querying.

Manages the real-time addition, modification, or deletion of vectors within a live index. Key operations include:

• Upsert: Insert a new vector or update an existing one by ID.
• Delete: Remove a specific vector or batch of vectors.
• Incremental Indexing: Some indices (e.g., HNSW) support dynamic insertion without a full rebuild, while others may require periodic re-indexing for optimal performance.

Provides read-only endpoints for operational visibility. Returns metadata and metrics such as:

• Index size (number of vectors).
• Memory/disk footprint.
• Build status and configuration parameters.
• Performance counters for query latency and throughput. This data is essential for capacity planning, performance tuning, and triggering maintenance operations.

Controls the full lifecycle state of an index. Core operations include:

• Load/Unload: Move an index between persistent storage and memory for querying. Unloading frees RAM; loading is required before search.
• Snapshot & Backup: Create point-in-time copies of the index data for disaster recovery.
• Versioning: Maintain multiple versions of an index for A/B testing or rollback scenarios.
• Deletion: Permanently remove an index and its associated data.

Allows search behavior to be optimized at query execution without rebuilding the index. Common adjustable parameters include:

• Search k: The number of nearest neighbors to return.
• EF (Search Scope): In HNSW, controls the size of the dynamic candidate list, trading accuracy for speed.
• Nprobe: In IVF, specifies how many nearest partitions to search. Fine-tuning these via the API is critical for achieving target latency and recall in production.

Beyond creation, Index APIs standardize several management operations:

• Describe Index: Retrieve configuration parameters (dimension, metric, index type).
• Update/Reconfigure: Some databases allow live tuning of certain parameters (e.g., Qdrant's update_collection).
• Delete Index: Permanently remove the index and all associated vectors.
• Status Check: For databases with asynchronous index builds, an endpoint to poll for completion status.
• Statistics: Get metrics like vector count, index size, and segment information for monitoring.

These operations form the essential lifecycle management toolkit for production vector data.

A vector index is the specialized data structure created and managed by an Index API. It organizes high-dimensional vectors to enable fast Approximate Nearest Neighbor (ANN) search. Common algorithms include:

• HNSW (Hierarchical Navigable Small World): A graph-based index known for high recall and speed.
• IVF (Inverted File Index): A clustering-based index that partitions the vector space for coarse-to-fine search.
• PQ (Product Quantization): A compression technique that reduces memory footprint while maintaining search accuracy. The Index API provides endpoints to create, configure, and rebuild these underlying structures.

These are the configuration options specified when creating an index via the Index API. They control the trade-off between search speed, accuracy (recall), and memory usage. Key parameters include:

• M / ef_construction: HNSW parameters controlling graph connectivity and search scope during build.
• nlist: The number of clusters (Voronoi cells) for an IVF index.
• nbits: The number of bits used for each quantized sub-vector in a Product Quantization (PQ) index.
• metric_type: The distance metric (e.g., L2, cosine, inner_product) used to measure vector similarity within the index.

The index state refers to the lifecycle phase of a vector index, managed through the Index API. Common states include:

• Empty: Created but contains no vectors.
• Building: Actively constructing the index data structure from ingested vectors.
• Ready: Successfully built and available to serve nearest neighbor queries.
• Failed: The build process encountered an error.
• Dropped: The index has been deleted. APIs often provide a GET /indexes/{id}/status endpoint to poll this state, crucial for managing asynchronous index operations.

Index rebuild is an Index API operation that reconstructs the underlying vector index from scratch. This is necessary when:

• The underlying data distribution changes significantly.
• Performance degrades due to fragmentation from many updates/deletes.
• Index parameters need to be changed (e.g., switching from L2 to cosine distance). A rebuild is typically a heavy, asynchronous operation. The API may offer a POST /indexes/{id}/rebuild endpoint, which returns a job ID for status tracking.

Asynchronous indexing is a pattern where an Index API call (like build or rebuild) returns immediately with an acknowledgment, rather than blocking until completion. The client receives a job ID or token to poll for status later. This is essential for:

• Handling large datasets where index construction takes minutes or hours.
• Preventing client timeouts in long-running operations.
• Enabling better resource management on the database server. Completion is often signaled via a webhook callback to a user-defined URL.

Index statistics are metrics exposed by the Index API about the health and performance of a vector index. These are critical for observability and capacity planning. Common statistics include:

• Vector Count: Total number of indexed vectors.
• Index Size: Memory/disk footprint of the index.
• Build Duration: Time taken for the last successful build.
• Query Performance: Latency percentiles (p50, p99) for searches against this index. Monitoring these stats helps engineers tune parameters and scale infrastructure.