The INDEX_VECTOR_MEMORY_ADVISOR function estimates the memory usage of a vector index that has not been created yet.
Syntax
FUNCTION INDEX_VECTOR_MEMORY_ADVISOR (
IN IDX_TYPE VARCHAR(65535),
IN NUM_VECTORS BIGINT UNSIGNED,
-- DIM_COUNT for dense indexes, SPARSE_AVG_LENGTH for sparse indexes
IN DIM_COUNT/SPARSE_AVG_LENGTH INT UNSIGNED,
IN DIM_TYPE VARCHAR(65535) DEFAULT 'FLOAT32',
IN IDX_PARAMETERS LONGTEXT DEFAULT NULL,
IN MAX_TABLET_VECTORS BIGINT UNSIGNED DEFAULT 0)
Parameters
Parameter |
Description |
|---|---|
| idx_type | The index type. Valid values:
|
| num_vectors | The number of vectors. |
| dim_count/sparse_avg_length | For dense indexes, specify DIM_COUNT to indicate the dimension of the vectors. For sparse indexes, specify SPARSE_AVG_LENGTH to indicate the average number of non-zero dimensions in each vector. If you have not created a table, you must estimate and specify this parameter based on your business requirements. If you set SPARSE_AVG_LENGTH to 0, the system uses 120 as the default value. |
| dim_type | The data type of the elements in the vectors. The default value is FLOAT32, which is also the only supported value. The value is case-insensitive. |
| idx_parameters | The index parameters. This parameter specifies the index parameters used when you create a table, for example, distance=l2, type=hnsw, lib=vsag. For more information, see Manage memory usage of a vector index. |
| max_tablet_vectors | The maximum number of vectors in a partition of a partitioned table. The default value is 0, which indicates that the maximum number of vectors in a partition is equal to the total number of vectors. Using a partitioned table can reduce the minimum memory requirement when the total data volume is the same. |
The return value is a string that describes the estimated memory usage.
Notice
In OceanBase Database, when you create an HNSW_BQ index, the system automatically selects the parallel or serial method for creating the index for a partitioned table based on the available memory. The recommended value returned by the INDEX_VECTOR_MEMORY_ADVISOR function is the maximum memory requirement when creating the index in serial mode. If the available memory meets only this value, the index creation time may be extended because the system cannot use the parallel method.
To ensure the successful execution of vector index rebuilding, the recommended memory reservation value should be greater than the actual memory usage of the vector index after the index is created. The recommended memory value is related to the maximum number of vectors in a partition:
- For HNSW and HNSW_SQ indexes, the function returns the recommended memory value.
- For HNSW_BQ indexes, the function returns the recommended memory value and the actual memory usage after the index is created.
- For IVF and IVF_PQ indexes, the function returns the recommended memory value and the actual memory usage after the index is created.
Examples
Query the recommended memory value for an HNSW index:
SELECT dbms_vector.index_vector_memory_advisor('HNSW',1000000,768,'FLOAT32','M=10,DISTANCE=L2');
+------------------------------------------------------------------------------------------+
| dbms_vector.index_vector_memory_advisor('HNSW',1000000,768,'FLOAT32','M=10,DISTANCE=L2') |
+------------------------------------------------------------------------------------------+
| Suggested minimum vector memory is 7.2 GB |
+------------------------------------------------------------------------------------------+
1 row in set
Query the recommended memory value and the actual memory usage after the index is created for an HNSW_BQ index:
SELECT dbms_vector.index_vector_memory_advisor('HNSW_BQ',1000000,768,'FLOAT32','M=10,DISTANCE=L2');
+---------------------------------------------------------------------------------------------------------+
| dbms_vector.index_vector_memory_advisor('HNSW_BQ',1000000,768,'FLOAT32','M=10,DISTANCE=L2') |
+---------------------------------------------------------------------------------------------------------+
| Suggested minimum vector memory is 4.1 GB, memory consumption when providing search service is 347.5 MB |
+---------------------------------------------------------------------------------------------------------+
1 row in set
Query the recommended memory value and the actual memory usage after the index is created for an IVF index:
SELECT dbms_vector.index_vector_memory_advisor('IVF_FLAT',1000000,768,'FLOAT32','SAMPLE_PER_NLIST=1024,DISTANCE=L2');
+---------------------------------------------------------------------------------------------------------------+
| dbms_vector.index_vector_memory_advisor('IVF_FLAT',1000000,768,'FLOAT32','SAMPLE_PER_NLIST=1024,DISTANCE=L2') |
+---------------------------------------------------------------------------------------------------------------+
| Suggested minimum vector memory is 452.7 MB, memory consumption when providing search service is 384.0 KB |
+---------------------------------------------------------------------------------------------------------------+
1 row in set
Query the recommended memory value and the actual memory usage after the index is created for a SINDI index:
SELECT dbms_vector.index_vector_memory_advisor('SINDI',10000,3,'FLOAT32','TYPE=SINDI,LIB=VSAG,DISTANCE=INNER_PRODUCT');
+-----------------------------------------------------------------------------------------------------------------+
| dbms_vector.index_vector_memory_advisor('SINDI',10000,3,'FLOAT32','TYPE=SINDI,LIB=VSAG,DISTANCE=INNER_PRODUCT') |
+-----------------------------------------------------------------------------------------------------------------+
| Suggested minimum vector memory is 73.6 MB |
+-----------------------------------------------------------------------------------------------------------------+
1 row in set