Description
This alert is triggered when the size of memory used by the OceanBase sys tenant (ID = 500) exceeds the threshold. This sys tenant is used for statistical purpose only and does not refer to a particular tenant.
Principle
The following table describes the key parameters that are involved in the monitoring and alerting logic.
Parameter |
Value |
|---|---|
| Metric | ob_tenant500_mem_hold_gigabyte The value of the metric indicates the size of memory occupied by the sys tenant (ID = 500).When this value is greater than the threshold, this alert is triggered. The default threshold is 100 GB. |
| Source | select /*+ MONITOR_AGENT READ_CONSISTENCY(WEAK) */ sum(hold) as hold, sum(used) as used from __all_virtual_memory_info where tenant_id = 500 and svr_ip = ? and svr_port = ? and mod_name <> 'OB_KVSTORE_CACHE_MB'select /* MONITOR_AGENT */ sum(hold) as hold, sum(used) as used from GV$OB_MEMORY where tenant_id = 500 and svr_ip = ? and svr_port = ? and context <> 'OB_KVSTORE_CACHE_MB' |
| Collected metric (unit: GB) | tenant500_mem_hold |
| Metric expression | sum(ob_tenant500_memory_hold_bytes{@LABELS}) by (@GBLABELS) / 1073741824 |
| Collection cycle | 60 seconds |
Alert rule
Metric |
Default threshold (unit: GB) |
Duration |
Detection cycle |
Time before clearance |
|---|---|---|---|---|
| ob_tenant500_mem_hold_gigabyte | 100 | 0 seconds | 60 seconds | 5 minutes |
Alert information
Trigger method |
Alert level |
Scope |
|---|---|---|
| Metric expression | Critical | Server |
Alert templates
Overview: ${alarm_target} ${alarm_name}
Details: Cluster: ${ob_cluster_name}, Host: ${host}, Alert: ${alarm_name}. The memory used is ${value} GB, exceeding the threshold of ${alarm_threshold} GB.
Overview example: ob_cluster=C1-1000:svr_ip=xxx.xxx.xxx.xxx. The memory used by the OceanBase sys tenant (ID = 500) exceeds the threshold.
Details example: Cluster: obcluster-1, Host: xxx.xxx.xxx.xxx, Alert: The memory used by the OceanBase sys tenant (ID = 500) exceeds the threshold. The memory used is 101.0 GB, exceeding the threshold of 100.0 GB.
Impact on the system
If the OceanBase sys tenant (ID = 500) consumes too many memory resources, the OBServer memory will be depleted, causing tenants in the OBServer to malfunction.
Possible cause
Some known defects cause the memory usage of the OceanBase sys tenant (ID = 500) to increase continuously in specific scenarios.
Suggested solutions
Go to the Resource Management page of the sys tenant of the OceanBase cluster, and view the Memory curve chart in the Resource Usage Tendency section to check for any unexpected memory usage.
If the memory usage is expected, increase the alert threshold to avoid the alert from being frequently generated.
In the left-side navigation pane, choose System Management > Alerts . In the Alert Item Configuration tab, find the target alert, and then click Edit to modify the alert threshold.
If the memory usage exceeds the expectation, restart the observer process to prevent the OBServer from malfunctioning due to insufficient memory.
Find the target OBServer in the OBServers list on the Overview page of the OceanBase cluster, and then click Restart to restart the observer process.