Question 1: What should I do if the memory usage of a tenant exceeds the upper limit?
ERROR 4030 (HY000): OB-4030:Over tenant memory limits
When you see the preceding error message, first check whether the MemStore memory limit is exceeded. If yes, check whether an excessive amount of data is written to the MemStore and whether throttling is disabled. This error is reported when a large amount of data is written and the minor compaction speed cannot keep up with the data write speed. Run the following statement to view the current memory usage:
obclient> SELECT /*+ READ_CONSISTENCY(WEAK),query_timeout(100000000) */ TENANT_ID,SVR_IP,
round(ACTIVE_SPAN/1024/1024/1024,2) ACTIVE_GB,
round(MEMSTORE_USED/1024/1024/1024,2) TOTAL_GB,
round(FREEZE_TRIGGER/1024/1024/1024,2) FREEZE_TRIGGER_GB,
round(MEMSTORE_USED/FREEZE_TRIGGER*100,2) percent_trigger,
round(MEMSTORE_LIMIT/1024/1024/1024,2) MEM_LIMIT_GB
FROM oceanbase.GV$OB_MEMSTORE
WHERE tenant_id >1000 OR TENANT_ID=1
ORDER BY tenant_id,TOTAL_GB DESC;
The emergency measure for this issue is to increase the tenant memory. For more information, contact OceanBase Technical Support for assistance. After the issue is temporarily resolved, analyze the root cause. If the issue is caused because throttling is not implemented, implement throttling and roll back the operation that increases the tenant memory. If the issue is caused because the tenant memory fails to support business growth, set a proper memory size for the tenant based on the minor compaction frequency. If the MemStore memory does not exceed the limit, run the following statement to check which memory module exceeds the limit:
obclient> SELECT tenant_id, svr_ip, sum(hold) module_sum
FROM oceanbase.GV$OB_MEMORY
WHERE tenant_id>1000 AND hold<>0 AND
CTX_NAME NOT IN ('KVSTORE_CACHE_ID','MEMSTORE_CTX_ID')
ORDER BY tenant_id, module_sum DESC;
The criterion for determining whether a memory module exceeds the limit is module_sum > (the tenant memory_size - the tenant MemStore). If the memory of a memory module exceeds the limit, you may need to adjust the memory of that module first, for example, adjust the value of ob_sql_work_area_percentage. If the tenant memory is too small, you also need to increase the tenant memory.
System variable |
Description |
Default value |
Applicable scope |
|---|---|---|---|
ob_sql_work_area_percentage |
The workspace memory of the tenant, which is used by blocking operators such as SQL sorting. | 5% | Global |
Question 2: What should I do if the plan cache hit rate is lower than 90%?
For an OLTP system, the plan cache hit rate must reach at least 90%. Run the following statements to view the plan cache hit rate:
obclient> SELECT hit_count,executions,(hit_count/executions) as hit_ratio
FROM V$OB_PLAN_CACHE_PLAN_STAT
WHERE (hit_count/executions) < 0.9;
obclient> SELECT hit_count,executions,(hit_count/executions) AS hit_ratio
FROM V$OB_PLAN_CACHE_PLAN_STAT
WHERE (hit_count/executions) < 0.9 AND executions > 1000;
Check for similar statements, such as statements with a random number of parameters following in or not in, that consume a lot of memory unnecessarily. If no such statements exist, the issue may be caused by insufficient memory due to a sudden increase in traffic or sessions. In this case, you need to increase the tenant memory.
Question 3: What should I do if the log contains "fail to allocate memory" or "allocate memory fail"?
The log contains the tenant_id (tenant ID) and context (memory module) information. You can use the following statement to query the information about the specific memory module:
obclient> SELECT * FROM oceanbase.GV$OB_MEMORY WHERE CTX_NAME =xxx AND tenant_id = xxx;
As described in Question 1, if the memory of a memory module exceeds the limit, you must adjust the memory of that module first. If the tenant memory is too small, increase the tenant memory. For more information, contact OceanBase Technical Support for assistance.
Question 4: What should I do if the memory usage of tenant 500 exceeds the upper limit?
The tenant whose ID is 500 is an internal tenant of OceanBase AI Database, also known as tenant 500. In normal cases, this error may be reported multiple times. The most possible reason is that the server has used up its system memory, or the remaining memory of the server is already pre-allocated and you cannot increase the memory of tenant 500.
To resolve this issue, check the memory usage of tenant 500 and the memory usage of the server. The memory usage of tenant 500 is not recorded in the V$OB_MEMORY view. Therefore, you must query the oceanbase.GV$OB_MEMORY view and check the memory usage of the system. If the server has used up its system memory, check whether this is an occasional situation. If yes and it has no impact on your business, you can wait for the system to release memory. If your business is affected, or normal system processes occupy or pre-allocate memory, you can increase the memory. For more information, contact OceanBase Technical Support for assistance.
SELECT * FROM oceanbase.GV$OB_MEMORY WHERE tenant_id=500;
// Command for checking system memory usage