This topic describes the types of vector indexes supported by OceanBase AI Database and their characteristics.
Index types
OceanBase AI Database supports the following types of vector indexes:
- Dense index
- Sparse index
- Semantic index
Dense index
OceanBase AI Database supports dense vector indexes, including the HNSW series and IVF series. For convenience, they are referred to as dense indexes in the following text.
Index type |
Description |
|---|---|
| HNSW | The maximum dimension of an index column is 4,096. HNSW indexes are in-memory indexes that must be fully loaded into memory. |
| HNSW_SQ | The HNSW_SQ index provides similar construction speed, search performance, and recall to the HNSW index, but reduces the total memory usage to 1/2 to 1/3 of the original. |
| HNSW_BQ | The recall of the HNSW_BQ index is slightly lower than that of the HNSW index, but it significantly reduces memory usage. The BQ quantization compression algorithm (Rabbitq) can compress a vector to 1/32 of its original size. As the vector dimension increases, the memory optimization effect of the HNSW_BQ index becomes more pronounced. |
| IVF | The IVF index implemented based on database tables does not occupy resident memory. |
| IVF_PQ | The IVF_PQ index implemented based on database tables does not occupy resident memory. It applies PQ quantization technology on top of IVF. The recall of this index is slightly lower than that of the IVF index, but its performance is higher. Additionally, the PQ quantization compression algorithm can typically compress a vector to 1/16 to 1/32 of its original size in most scenarios. |
Sparse index
OceanBase AI Database supports memory-based sparse vector indexes, which are referred to as memory sparse indexes for convenience. Memory sparse indexes are an efficient index type provided by OceanBase AI Database for sparse vector data (vectors with most elements being zero). They require the complete index to be loaded into memory and support DML and real-time search.
Notice
Memory sparse indexes are an experimental feature and are not recommended for production use.
Semantic index
OceanBase AI Database supports semantic indexes, which utilize its built-in embedding capabilities to greatly simplify the process of using vector indexes. It achieves transparency of vector concepts for users: you can directly write the original data to be stored (such as text or image URLs), and OceanBase AI Database will automatically convert it into a vector and create an index internally. During search, you only need to provide the original search content, and OceanBase AI Database will automatically perform embedding and search the vector index, significantly improving ease of use. In addition to text, it also supports creating semantic indexes for image URLs, enabling scenarios such as image search by image or text. Currently, multiple semantic indexes can be created on a single column, which can use different models, distance algorithms, or index parameter combinations.
Notice
Semantic indexes are an experimental feature and are not recommended for production use. The multi-index feature on a single column is only for testing and cannot be used in production.
Considerations and limitations
- Distance algorithms: Dense vector indexes support L2, inner product (IP), and cosine distance as index distance algorithms.
- Distance functions: Vector index search supports calling some distance functions. For details, see Use SQL functions.
- Filter conditions: Vector search supports filter conditions. Filter conditions can be scalar conditions or spatial relationships, such as ST_Intersects. Multi-value indexes, full-text indexes, and global indexes cannot be used as prefilters.
- Hybrid search: You can create vector indexes and full-text indexes on the same table. Vector indexes include dense and memory sparse indexes.
- Offline DDL: For information about the support of vector indexes for offline DDL, see Online DDL and Offline DDL Operations.
- Columnstore index: Columnstore vector indexes are not supported in the current version.