What are the alert rules?

This topic describes the scenarios in which OceanBase Migration Service (OMS) generates alerts.

Downtime alerts

• Alert objects: servers deployed with OMS

• Judgment method: OMS has an agent (oms_drc_supervisor) on each deployment server. The agents report heartbeats to the OMS control database. Then, OMS compares the heartbeat time with the current time to determine whether a server is available.

• Impact on business: If high availability (HA) is not enabled, related tasks fail. If HA is enabled, tasks related to the server are migrated to an available server. If no server is available, OMS cannot run properly. If migration is triggered, a migration alert is reported to prompt you to check the migration result.

• Troubleshooting method:

  • If the oms_drc_supervisor application is interrupted, log in to the host and run supervisorctl restart oms_drc_supervisor to restart the application. If the application fails to recover, check the error log under /home/admin/logs/supervisor.

  • For other reasons, such as network problems, on-site O&M intervention is required.

  • If HA is enabled on multiple servers, you can enable the enableHost parameter and perform migration. In this case, only JDBCWriter/Connector is migrated.

OMS server resource level alerts

• Alert objects: servers deployed with OMS

• Judgment method: The agent (oms_drc_supervisor) deployed on each OMS server regularly collects the CPU, memory, and disk usage of the server.

• Impact on business: Tasks cannot be started on the server, or the migration and synchronization performance may be impaired.

• Troubleshooting method: First, run top to check whether the resource consuming processes are OMS processes. If not, perform troubleshooting accordingly. If yes, perform the following operations:

  • Add an OMS server.

  • Clean up unnecessary processes or disk files.

Migration task failed

• Alert objects: all migration tasks

• Judgment method: The status of a migration task is displayed as "Failed".

• Impact on business: Data migration is interrupted.

• Troubleshooting method: Check the cause of failure in the migration task, and then perform troubleshooting accordingly.

Migration task delayed

• Alert objects: running migration tasks that are in the incremental migration phase

• Judgment method: Obtain the maximum delay time of a specific task based on the delay information and the protection level of the migration task that are configured in the alert center. If the task delay time exceeds the expected tolerance level, an alert is sent.

• Impact on business: Data cannot be synchronized from the source to the destination in real time, or data cannot be synchronized from the destination to the source in reverse synchronization.

• Troubleshooting method: Obtain the following logs and perform troubleshooting according to these logs or information on the View Component Monitoring page:

  • Store logs. You can check the latest pull time of Store on the View Component Monitoring page to determine whether the data pulled by Store lags behind .

  • Writer logs.

Synchronization task failed

• Alert objects: all synchronization tasks

• Judgment method: The status of a synchronization task is displayed as "Failed".

• Impact on business: Data synchronization is interrupted.

• Troubleshooting method: Check the cause of failure in the synchronization task, and then perform troubleshooting accordingly.

Synchronization task delayed

• Alert object: running synchronization tasks that are in the incremental synchronization phase

• Judgment method: Obtain the maximum delay time of a specific task based on the delay information and the protection level of the synchronization task that are configured in the alert center. If the task delay time exceeds the expected tolerance level, an alert is sent.

• Impact on business: The incremental data on the source cannot be synchronized to the destination in real time.

• Troubleshooting method: Obtain the following logs and perform troubleshooting according to these logs or information on the View Component Monitoring page.

  • Store logs for store synchronization tasks. You can check the latest pull time of Store on the View Component Monitoring page to determine whether the data pulled by Store lags behind.

  • Connector logs.