Description
This alert is triggered when a log backup is delayed.
Principle
OCP-Server runs a timed task to check the log backup tasks every minute. When the log backup latency exceeds the threshold, this alert is triggered. The log backup tasks include:
Logical log backup
Physical log backup
Note
You can specify the threshold by modifying the value of Alert Threshold for Log Backup Delays in the Configure Alert Threshold section of the backup strategy.
Alert rule
Metric |
Default threshold |
Duration |
Detection cycle |
Time before clearance |
|---|---|---|---|---|
| None | None | 0 seconds | 60 seconds | 5 minutes |
Alert information
Trigger method |
Alert level |
Scope |
|---|---|---|
| Timed task of OCP | Critical | Service |
Alert templates
Overview: ${alarm_target} ${alarm_name}
Details: Cluster: ${ob_cluster_name}[${ob_cluster_id}], Host: ${host}, Alert: A log backup is delayed by ${log_backup_latency} seconds, exceeding the threshold of ${threshold} seconds, Checkpoint: ${checkpoint}, Current time: ${current_time}, Tenant list: ${tenants}.
Overview example: ob_cluster=C1-1000. A log backup is delayed.
Details example: Cluster: obcluster-1[10006], Host: xxx.xxx.xxx.xxx, Alert: A log backup is delayed by 30 seconds, exceeding the threshold of 5 seconds, Checkpoint: 2020-01-05 02:00:00, Current time: 2020-01-15 01:00:00, Tenant list: tenant1,tenant2.
Impact on the system
If the log backup is significantly delayed, the backup data is not up-to-date and the system cannot restore to the latest state or perform new physical data backup tasks.
Possible causes
This problem is commonly found in the following scenarios:
An error occurred while performing a log backup task.
Something is wrong with the backup media.
Suggested solutions
If the alert is triggered during a logical log backup task, check the agentserver.log of the backup service for troubleshooting.
If the alert is triggered during a physical log backup task, check the OBServer logs to find the specific cause. Perform the following steps to find the logs:
Log on to the OCP console. On the Clusters page, find the target cluster in the Clusters list and click its name. Then, the Overview page of the cluster appears.
Click the More icon in the upper-right corner and select Download Logs .
Specify the time range, which must contain the alert time.
For the log type, select OBServer Logs .
Click Download .
Wait for the OBServer logs to be downloaded.
Open the observer.log file and search for WARN- and ERROR-level logs.