Description
This alert is triggered when the percentage of the memory space allocated to all tenants in an OceanBase cluster exceeds the threshold.
Principle
The following table describes the key parameters that are involved in the monitoring and alerting logic.
Parameter |
Value |
|---|---|
| Metric | ob_mem_assigned_percent |
| Source | SQL: select /*+ READ_CONSISTENCY(WEAK)*/ mem_assigned,mem_total from __all_virtual_server_stat where svr_ip = @svr_ip and svr_port = rpc_port(); |
| Collected metrics | mem_assigned and mem_total |
| Metric expression | 100 * avg(mem_assigned{metric_group="all_virtual_server_stat",@LABELS}) by (@GBLABELS) / avg(mem_total{metric_group="all_virtual_server_stat",@LABELS}) by (@GBLABELS) |
| Collection cycle | 60 seconds |
The value of the metric ob_mem_assigned_percent indicates the percentage of memory space allocated by an OceanBase cluster to all tenants in it. When the percentage exceeds the default threshold of 98%, this alert is triggered.
Alert rule
Metric |
Default threshold (unit: %) |
Duration |
Detection cycle |
Time before clearance |
|---|---|---|---|---|
| ob_mem_assigned_percent | 98 | 0 | 60 seconds | 5 minutes |
Alert information
Trigger method |
Alert level |
Scope |
|---|---|---|
| Metric expression | Caution | Server |
Alert templates
Overview
- Template: ${alarm_target} ${alarm_name}
- Example: ob_cluster=obcluster-1:svr_ip=xxx.xxx.xxx.xxx OceanBase memory allocated to tenants exceeds the threshold.
Details
- Template: Cluster: ${ob_cluster_name}, Host: ${host}, Alert: ${alarm_name}. The OceanBase memory allocated to tenants is ${value_shown}%, exceeding the threshold of ${alarm_threshold}.
- Example: Cluster: obcluster-1, Host: xxx.xxx.xxx.xxx, Alert: OceanBase memory allocated to tenants exceeds the threshold. The OceanBase memory allocated to tenants is 99.0%, exceeding the threshold of 98.0%.
Alert recovery
- Template: Alert: ${alarm_name}, OceanBase memory allocated to tenants: ${value_shown}%
- Example: Alert: OceanBase memory allocated to tenants exceeds the threshold. OceanBase memory allocated to tenants: 92%
Impact on the system
The customer needs to apply for more resources, or new tenants may not be able to be created because of insufficient resources.
Possible causes
The free memory space in the OceanBase cluster is insufficient. Further allocation of the memory space may cause it to be over-allocated.
An OBServer in the cluster is deleted.
Suggested solutions
Check whether an OBServer in the cluster is deleted.
Check the OBServers list on the Overview page of this cluster for any abnormal OBServer.
If any, the ob_cluster_exists_inactive_server alert will be triggered at the same time. Read the corresponding topic to resolve this alert. Then, check whether the ob_mem_assigned_percent_over_threshold alert still persists.
If not, the alert may have been caused by the fact that the free memory space in the OceanBase cluster is insufficient. Go to the next step.
Check whether the free memory space in the OceanBase cluster is insufficient.
Perform any of the following steps to resolve this problem.
If you have an unwanted tenant in the cluster, delete it. For more information, see Delete a tenant.
If the memory space used by a tenant is smaller than the memory space allocated to it, scale in the unit type of the tenant. For more information, see Manage unit specifications.
Add OBServers to the cluster to scale it out. For more information, see Add an OBServer.