Description
This alert indicates that some tenants have experienced log stream degradation. For tenants with arbitration service enabled, which are typically 2F or 4F architectures, log stream degradation occurs when a single replica fails in a 2F architecture or two replicas fail in a 4F architecture. This ensures that the tenant can still provide services even if the majority of replicas are unavailable. When this happens, OCP generates an alert, indicating which tenants have experienced log stream degradation.
Principle
Parameter |
Value |
|---|---|
| Monitoring metric | ob_tenant_log_stream_degraded_count |
| Metric source | select /*+ MONITOR_AGENT */ t2.tenant_id, t2.tenant_name, sum(t1.degraded_list <> '') as degraded_count from V$OB_LOG_STAT t1 inner join DBA_OB_TENANTS t2 on t1.tenant_id = t2.tenant_id and t2.tenant_type != 'META' where svr_ip= ? and svr_port= ? group by t1.tenant_id |
| Sampling metric | ob_tenant_log_stream_degraded_count |
| Monitoring expression | sum(log_stream_degraded_count{@LABELS}) by (@GBLABELS) |
| Sampling interval | 60 seconds |
Monitoring metric log_stream_degraded_count indicates the number of degraded log streams for a tenant. An alert is triggered when this value is greater than 0.
Rules
Monitoring expression |
Default threshold |
Duration |
Detection interval |
Elimination interval |
|---|---|---|---|---|
| log_stream_degraded_count | 0 | 0 seconds | 60 seconds | 5 minutes |
Alert information
Alert triggering method |
Alert level |
Scope |
|---|---|---|
| Based on the expression of the monitoring metric | Critical | Tenant |
Alert template
- Alert overview
- Template: ${alarm_target} ${alarm_name}
- Example: ob_cluster_name=obcluster:ob_cluster_id=4:tenant_name=mysql OceanBase tenant has degraded log streams
- Alert details
- Template: Cluster: ${ob_cluster_name}, Tenant: ${tenant_name}, Alert: ${alarm_name}. Log streams of tenant ${tenant_name} have been degraded. Number of degraded log streams: ${value_shown}.
- Example: Cluster: obcluster, Tenant: mysql, Alert: OceanBase tenant has degraded log streams. Log streams of tenant mysql have been degraded. Number of degraded log streams: 3.
- Alert recovery
- Template: Alert: ${alarm_name}, Number of degraded log streams for OceanBase tenant: ${value_shown}
- Example: Alert: OceanBase tenant has degraded log streams, Number of degraded log streams for OceanBase tenant: 0
Impact on the system
When log streams are degraded, the tenant becomes unavailable for a few seconds, which may slightly affect the business system.
Possible causes
- A failure occurred at the IDC where the F replica of the tenant is located.
- The OBServer hosting the log stream of the F replica of the tenant experienced a failure, such as a crash.
Procedure
Log in to the OceanBase cluster as root@sys and execute the following command to query the degraded log streams and the OBServer hosting them.
select t2.tenant_id, t2.tenant_name, t1.degraded_list from V$OB_LOG_STAT t1 inner join DBA_OB_TENANTS t2 on t1.tenant_id = t2.tenant_id and t2.tenant_type != 'META';Based on the OBServer information queried in Step 1, check the operational status of these OBServer nodes.
If the OBServer node experienced a failure, recommend restarting the OBServer node. After the OBServer node is restarted, the alert will be resolved.