Perform a switchover between primary and standby OceanBase Database tenants in a scenario with two IDCs

This topic describes how to perform a switchover between primary and standby OceanBase Database tenants in a scenario with two IDCs

Scenarios

In most cases, your business applications are isolated by tenant. Using primary and standby clusters in this scenario may have the following impacts:

• When a tenant becomes unavailable due to the failure of the OBServer node hosting the tenant, you must perform a failover for the entire cluster, which affects other applications.
• When high availability (HA) is required only for critical business, the system must synchronize the data of all tenants, resulting in significant cross-IDC traffic.

OceanBase Database V4.1 and later support primary and standby tenants instead of primary and standby clusters. Primary and standby tenants support tenant-level log synchronization, switchover, and failover to resolve the preceding problems.

Primary and standby tenants support two log synchronization modes: archiving-based and network-based log synchronization. In archiving-based synchronization, the primary tenant must provide complete backups of baseline data and incremental logs to restore a standby tenant. The standby tenant then continuously retrieves archived logs from the archiving address of the primary tenant and parses and replays the logs. The synchronization delay depends on the network delay and the archiving delay of the primary tenant, and is typically significant.

In dual-IDC deployment, the primary and standby tenants are deployed in two IDCs respectively, as shown in the following figures.

• Primary and standby tenants that apply network-based log synchronization:

  

• Primary and standby tenants that apply archiving-based log synchronization:

  

Prerequisites

The primary and standby tenants for which you want to perform a switchover meet the following conditions:

• The primary and standby tenants are normal.

• The clusters to which the primary and standby tenants belong are running.

• The switchover status of the primary and standby tenants is normal. You can connect to a cluster with the account root@sys and execute the following SQL statement to query the switchover status of a tenant:

  SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, SWITCHOVER_STATUS FROM oceanbase.CDB_OB_TENANTS WHERE TENANT_NAME = ?;
  

• The primary and standby tenants are user tenants. You can execute the preceding SQL statement to query the tenant type.

• The RECOVERY_UNTIL_SCN of the primary tenant is infinite. You can execute the following SQL statement to query this parameter. The value 4611686018427387903 indicates that the RECOVERY_UNTIL_SCN is infinite.

  SELECT RECOVERY_UNTIL_SCN FROM oceanbase.CDB_OB_TENANTS WHERE TENANT_NAME = ?;
  

• All log streams of the primary and standby tenants have leaders. You can execute the following SQL statement to query the leader status. The value 0 indicates that all log streams have leaders.

  SELECT COUNT(1) FROM CDB_OB_LS A LEFT JOIN GV$OB_LOG_STAT B ON A.LS_ID = B.LS_ID
      AND A.TENANT_ID = B.TENANT_ID
      AND B.ROLE = 'LEADER'
  WHERE B.LS_ID IS NULL
      AND A.STATUS NOT IN ('CREATING', 'CREATED', 'TENANT_DROPPING', 'CREATE_ABORT', 'PRE_TENANT_DROPPING')
      AND A.TENANT_ID IN (?, (? - 1));
  

• The primary and standby tenants are not locked.

• All OBServer nodes in the cluster to which the standby tenant belongs are active.

• Versions of the clusters to which the primary and standby tenants belong are the same.

Procedure

Method 1: Perform a switchover for a specified primary tenant

You can switch a specified primary tenant to a standby tenant.

1. Log on to the OceanBase Cloud Platform (OCP) console.

2. In the left-side navigation pane, click Tenants. On the Tenants tab, select the target primary tenant to go to its Overview page.

3. On the page that appears, click the More icon in the upper-right corner and select Switchover from the menu.

4. In the dialog box that appears, confirm the information and click Start Switch .

5. Click the task ID in the dialog box to view the task progress.

Method 2: Perform a switchover for all primary tenants in a cluster

If you require standby tenants to provide services to ensure service continuity during the O&M of a cluster, you can switch all primary tenants in this cluster to standby tenants before the O&M.

1. Log on to the OCP console.

2. In the left-side navigation pane, click Clusters. On the Clusters page, go to the Overview page of the target cluster.

3. Click the More icon in the upper-right corner and select Tenant Switchover from the menu.

4. In the dialog box that appears, select the tenants for which you want to perform a switchover and click Switchover.

5. Click the task ID in the dialog box to view the task progress.

FAQ

The FETCH LOG TIMEOUT error occurred in the Create standby tenant subtask when I created a standby tenant in OCP V4.2.0. What should I do?

The preceding figure shows the error message. The error occurred because the primary tenant had too many clogs, causing an RPC timeout when the standby tenant attempted to fetch clogs. You can increase the value of the standby_db_fetch_log_rpc_timeout parameter for the primary tenant.

The system returned an error message when I performed a switchover, indicating that the log synchronization delay (xx ms) of the tenant is greater than xxx ms, and therefore the operation is not allowed. What should I do?

• Network-based log synchronization:

  1. Check whether log synchronization between the primary and standby tenants is normal. You can check for synchronization exceptions in the tenant topology in OCP.
  2. If the synchronization delay is small and does not increase continuously, check whether a network delay occurs or CPU resources of the standby tenant are insufficient.
  3. If you identify the specific cause in the preceding step and still want to perform the switchover, increase the value of the ocp.standby.tenant.delay.threshold parameter in OCP and then continue with the switchover.

• Archiving-based log synchronization: Check whether the value of the archive_lag_target parameter for the primary tenant is excessively large. This parameter specifies the backup delay for the primary tenant.