The INDEX_VECTOR_MEMORY_ADVISOR function is used to estimate the memory usage for a vector index that has not been created.
Syntax
FUNCTION INDEX_VECTOR_MEMORY_ADVISOR (
IN IDX_TYPE VARCHAR(65535),
IN NUM_VECTORS BIGINT UNSIGNED,
-- DIM_COUNT for dense indexes, SPARSE_AVG_LENGTH for sparse indexes
IN DIM_COUNT/SPARSE_AVG_LENGTH INT UNSIGNED,
IN DIM_TYPE VARCHAR(65535) DEFAULT 'FLOAT32',
IN IDX_PARAMETERS LONGTEXT DEFAULT NULL,
IN MAX_TABLET_VECTORS BIGINT UNSIGNED DEFAULT 0)
Parameters
Parameter |
Description |
|---|---|
| idx_type | Index type:
|
| num_vectors | The volume of vector data. |
| dim_count/sparse_avg_length | For a dense index, enter DIM_COUNT, indicating the dimensionality of the vector. For sparse indexes, enter SPARSE_AVG_LENGTH, indicating the average number of non-zero dimensions in each vector. When the table is not created, estimate and fill in this value based on business expectations. If SPARSE_AVG_LENGTH Set to 0, the memory usage is estimated based on the default value of 120. |
| dim_type | The data type of the vector element. Default and only supported value: FLOAT32, which is case-insensitive. |
| idx_parameters | Index parameters. The vector index parameter string used during table creation, for example, distance=l2, type=hnsw, lib=vsag. For more parameters, see Vector index memory management. |
| max_tablet_vectors | The maximum vector data volume of a partition in a partitioned table. Default value: 0, indicating that the maximum partition vector data volume equals the total vector data volume. Using a partitioned table can reduce the minimum memory requirement when the total data volume is the same. |
The return value is a string describing the estimated memory usage.
Notice
In OceanBase AI Database, when creating the HNSW_BQ index, the system automatically chooses between parallel or serial mode based on memory availability. The recommended value returned by the INDEX_VECTOR_MEMORY_ADVISOR function is the maximum memory requirement for creating the index in serial mode. If the available memory only meets this value, index creation time may be prolonged due to the inability to create it in parallel.
To ensure the successful rebuilding of a vector index, it is recommended to reserve more memory than the actual memory occupied by the index after construction. The recommended memory value is related to the maximum number of partition vectors:
- For HNSW and HNSW_SQ indexes, returns the recommended memory value.
- For the HNSW_BQ index, the recommended memory value and the actual memory usage after the index is created are returned.
- For IVF and IVF_PQ indexes, the recommended memory value and the actual memory usage after index creation are returned.
Examples
Recommended memory for querying the HNSW index:
SELECT dbms_vector.index_vector_memory_advisor('HNSW',1000000,768,'FLOAT32','M=10,DISTANCE=L2');
+------------------------------------------------------------------------------------------+
| dbms_vector.index_vector_memory_advisor('HNSW',1000000,768,'FLOAT32','M=10,DISTANCE=L2') |
+------------------------------------------------------------------------------------------+
| Suggested minimum vector memory is 7.2 GB |
+------------------------------------------------------------------------------------------+
1 row in set
Query the recommended memory value for the HNSW_BQ index and the actual memory usage after the index is created:
SELECT dbms_vector.index_vector_memory_advisor('HNSW_BQ',1000000,768,'FLOAT32','M=10,DISTANCE=L2');
+---------------------------------------------------------------------------------------------------------+
| dbms_vector.index_vector_memory_advisor('HNSW_BQ',1000000,768,'FLOAT32','M=10,DISTANCE=L2') |
+---------------------------------------------------------------------------------------------------------+
| Suggested minimum vector memory is 4.1 GB, memory consumption when providing search service is 347.5 MB |
+---------------------------------------------------------------------------------------------------------+
1 row in set
Query the recommended memory value for creating an IVF index and the actual memory usage after the index is created:
SELECT dbms_vector.index_vector_memory_advisor('IVF_FLAT',1000000,768,'FLOAT32','SAMPLE_PER_NLIST=1024,DISTANCE=L2');
+---------------------------------------------------------------------------------------------------------------+
| dbms_vector.index_vector_memory_advisor('IVF_FLAT',1000000,768,'FLOAT32','SAMPLE_PER_NLIST=1024,DISTANCE=L2') |
+---------------------------------------------------------------------------------------------------------------+
| Suggested minimum vector memory is 452.7 MB, memory consumption when providing search service is 384.0 KB |
+---------------------------------------------------------------------------------------------------------------+
1 row in set
Query the recommended memory value for the SINDI index and the actual memory usage after the index is created:
SELECT dbms_vector.index_vector_memory_advisor('SINDI',10000,3,'FLOAT32','TYPE=SINDI,LIB=VSAG,DISTANCE=INNER_PRODUCT');
+-----------------------------------------------------------------------------------------------------------------+
| dbms_vector.index_vector_memory_advisor('SINDI',10000,3,'FLOAT32','TYPE=SINDI,LIB=VSAG,DISTANCE=INNER_PRODUCT') |
+-----------------------------------------------------------------------------------------------------------------+
| Suggested minimum vector memory is 73.6 MB |
+-----------------------------------------------------------------------------------------------------------------+
1 row in set