OceanBase AI Database is deployed on cost-effective commodity servers. Based on the Paxos protocol, each data copy is written to more than half of the replicas of LogService on three or more servers. Therefore, no data is lost when a minority of replicas are unavailable (one replica can be unavailable in a three-replica cluster), ensuring a recovery point objective (RPO) of 0. If the leader replica fails, a new leader is automatically elected from the remaining follower replicas to continue to provide services within a short time without manual intervention, ensuring a recovery time objective (RTO) of less than 8s. This design achieves a balance between strong consistency and high availability. Users can also adopt flexible deployment modes, such as single-IDC or three-IDC-in-the-same-city deployment, to achieve lossless disaster recovery at all levels.
Root Service
In OceanBase AI Database, Root Service is responsible for managing cluster nodes. Each OBServer reports its process status to Root Service every 2 seconds through heartbeat packets. Root Service monitors these heartbeat packets to obtain the working status of the OBServer processes.
The configuration items related to the OBServer heartbeat status are as follows:
lease_timeThe cluster-level configuration item
lease_timespecifies the heartbeat lease duration. The default value is 10 seconds. For more information about this configuration item, see lease_time.If Root Service does not receive a heartbeat packet from an OBServer within the period specified by
lease_time, Root Service considers the observer process temporarily disconnected and marks its heartbeat status aslease_expired.server_permanent_offline_timeThe cluster-level configuration item
server_permanent_offline_timespecifies the time threshold for heartbeat loss of a node. A node is considered permanently offline after its heartbeat is lost for the specified period, and the data replicas on the permanently offline node must be automatically replenished. The default value is 3600 seconds. For more information about this configuration item, see server_permanent_offline_time.If Root Service does not receive a heartbeat packet from an OBServer within the period specified by
server_permanent_offline_time, Root Service considers the observer process disconnected and marks its heartbeat status aspermanent_offline.
Note
You can query the values of the preceding cluster-level configuration items by using the SHOW PARAMETERS statement. For example: SHOW PARAMETERS LIKE 'server_permanent_offline_time';.
Based on the heartbeat packets, Root Service can obtain the following working statuses of an OBServer:
The OBServer properly sends heartbeat packets, and the disk status in the heartbeat packets is normal. In this case, Root Service considers that the OBServer is working properly.
The OBServer properly sends heartbeat packets, but the disk status in the heartbeat packets is abnormal. In this case, Root Service considers that the observer process still exists, but the disk of the OBServer is faulty. In addition, Root Service switches all leaders on this OBServer to another OBServer.
No heartbeat packet is received from the OBServer within the period specified by
lease_time, and the heartbeat status of the OBServer is marked aslease_expired. In this case, Root Service sets the status of the OBServer toINACTIVEand does not take other measures.No heartbeat packet is received from the OBServer within the period specified by
server_permanent_offline_time, and the heartbeat status of the OBServer is marked aspermanent_offline. In this case, Root Service removes the data replicas on this OBServer from the Paxos member group, and adds the removed data replicas to another active OBServer to ensure that the Paxos member group of data replicas is complete.
Isolate a node
ODP can detect abnormal states of OBServer nodes, such as the stopped state (ACTIVE but with a stop_time field greater than 0) or the INACTIVE state, thereby avoiding routing application traffic to such abnormal nodes.
If a node is unstable, it may occasionally provide service but exhibit various abnormalities such as slow responses. If a client connects to this node, it will experience instability. OceanBase AI Database provides the STOP SERVER command for isolating a node. After isolation, the node no longer provides external services, and ODP does not route requests to it, allowing for safe diagnostic, replacement, or maintenance actions. This makes it a powerful tool for operations and emergency response. The Stop Server operation checks whether all remaining replicas form a majority and whether log synchronization is complete. If these conditions are not met, the operation returns failure; if they are met, the system returns success after all leaders on the failed node have been switched over. Successful completion of Stop Server ensures that any maintenance action can be performed, including killing the observer process. For more information about the STOP SERVER command, see Isolate a node.
Node isolation is applicable in single-IDC failure scenarios:
Non-disconnection failures (such as machine instability caused by NIC packet loss) strongly rely on node isolation; otherwise, repeated leader switchover will affect the application.
Disconnection failures (such as direct server downtime) do not strongly rely on node isolation. Instead, the system marks the node status as
INACTIVEin theoceanbase.DBA_OB_SERVERSview to isolate application traffic.
When a node is isolated or a leader fails, an election is triggered. The election service's priority mechanism takes into account user-specified primary zones and the abnormal status of machines, ensuring a better replica is selected as the new leader.
Disaster recovery operations
OceanBase AI Database adopts a storage-and-compute separation architecture. It provides data durability through high-availability LogService, ensuring zero data loss (RPO = 0), and guarantees data high availability through multi-replica compute nodes. Different replica counts correspond to different disaster recovery operations: two-replica deployment supports manual leader switchover, while single-replica deployment supports disaster recovery within or across availability zones (AZs). For more information about replica deployment scenarios, see Disaster recovery deployment solutions.
Note
The cross-AZ disaster recovery feature for single replicas is supported starting from V4.6.2.1.
Disaster recovery operations in a two-replica scenario
For a compute node deployment with two replicas, in addition to isolating nodes, you can also perform an active leader switchover to switch the primary/secondary roles of log stream replicas.
The procedure for performing an active leader switchover is as follows:
Log in to the
systenant of the cluster as therootuser.Here is an example. Use the actual environment settings when you connect to the database.
obclient -h10.xx.xx.xx -P2883 -uroot@sys#obdemo -p***** -AFor more information about how to connect to a database, see Overview of connection methods.
Query the zones, servers, and log stream replicas of tenants.
obcllient [(none)]> SELECT a.TENANT_ID,a.LS_ID,a.SVR_IP,a.SVR_PORT,a.ZONE,a.role,b.TENANT_NAME,b.TENANT_TYPE FROM oceanbase.CDB_OB_LS_LOCATIONS a, oceanbase.DBA_OB_TENANTS b WHERE a.TENANT_ID=b.TENANT_ID;The query result is as follows:
+-----------+-------+----------------+----------+-------+----------+-------------+-------------+ | TENANT_ID | LS_ID | SVR_IP | SVR_PORT | ZONE | role | TENANT_NAME | TENANT_TYPE | +-----------+-------+----------------+----------+-------+----------+-------------+-------------+ | 1 | 1 | xx.xx.xx.237 | 2882 | zone1 | LEADER | sys | SYS | | 1 | 1 | xx.xx.xx.238 | 2882 | zone2 | FOLLOWER | sys | SYS | | 1001 | 1 | xx.xx.xx.237 | 2882 | zone1 | LEADER | META$1002 | META | | 1001 | 1 | xx.xx.xx.238 | 2882 | zone2 | FOLLOWER | META$1002 | META | | 1002 | 1 | xx.xx.xx.237 | 2882 | zone1 | LEADER | mysql001 | USER | | 1002 | 1 | xx.xx.xx.238 | 2882 | zone2 | FOLLOWER | mysql001 | USER | | 1002 | 1001 | xx.xx.xx.237 | 2882 | zone1 | LEADER | mysql001 | USER | | 1002 | 1001 | xx.xx.xx.238 | 2882 | zone2 | FOLLOWER | mysql001 | USER | +-----------+-------+----------------+----------+-------+----------+-------------+-------------+ 8 rows in setFor more information about the
CDB_OB_LS_LOCATIONSview, see CDB_OB_LS_LOCATIONS.Perform a leader switchover.
ALTER SYSTEM SWITCH REPLICA role LS [=] ls_id SERVER [=] 'ip:port' TENANT = tenant_name;The parameters are described as follows:
role: the role of the replica. Valid values areLEADERandFOLLOWER.ls_id: the ID of the log stream replica to be modified.ip:port: the IP address and RPC port of the server where the log stream replica to be modified resides.tenant_name: the tenant to which the log stream replica to be modified belongs.
Here is an example:
obclient [(none)]> ALTER SYSTEM SWITCH REPLICA LEADER LS = 1001 SERVER = 'xx.xx.xx.238:2882' TENANT = mysql001;After the operation is completed, query the view again to check whether the replica role is modified.
obclient [(none)]> SELECT * FROM oceanbase.CDB_OB_LS_LOCATIONS WHERE TENANT_ID = 1002;The query result is as follows:
+----------------------------+----------------------------+-----------+-------+----------------+----------+----------+-------+----------+-------------------------------------------------------------------+----------------------+--------------+--------------+---------+ | CREATE_TIME | MODIFY_TIME | TENANT_ID | LS_ID | SVR_IP | SVR_PORT | SQL_PORT | ZONE | ROLE | MEMBER_LIST | PAXOS_REPLICA_NUMBER | REPLICA_TYPE | LEARNER_LIST | REBUILD | +----------------------------+----------------------------+-----------+-------+----------------+----------+----------+-------+----------+-------------------------------------------------------------------+----------------------+--------------+--------------+---------+ | 2022-12-26 18:28:49.389601 | 2022-12-26 18:28:54.441143 | 1002 | 1 | xx.xx.xx.237 | 2882 | 2881 | zone1 | LEADER | xx.xx.xx.237:2882:1,xx.xx.xx.238:2882:1 | 2 | FULL | | FALSE | | 2022-12-26 18:28:49.390567 | 2023-01-11 12:02:37.970540 | 1002 | 1 | xx.xx.xx.238 | 2882 | 2881 | zone2 | FOLLOWER | NULL | NULL | FULL | | FALSE | | 2022-12-26 18:29:01.334732 | 2023-01-11 11:59:31.899581 | 1002 | 1001 | xx.xx.xx.238 | 2882 | 2881 | zone2 | LEADER | xx.xx.xx.237:2882:1,xx.xx.xx.238:2882:1 | 2 | FULL | | FALSE | | 2022-12-26 18:29:01.335629 | 2023-01-11 12:02:37.709677 | 1002 | 1001 | xx.xx.xx.237 | 2882 | 2881 | zone1 | FOLLOWER | NULL | NULL | FULL | | FALSE | +----------------------------+----------------------------+-----------+-------+----------------+----------+----------+-------+----------+-------------------------------------------------------------------+----------------------+--------------+--------------+---------+ 4 rows in set
Active leader switchover is applicable to IDC-level and city-wide disaster recovery scenarios. You can switch the leader to a specified Zone or Region to better match the application traffic distribution.
Disaster recovery operations in single-replica scenarios
In a single-replica deployment, after a compute node fails, you need to restart the failed log stream replica on another available node. The disaster recovery operations for different failure scenarios are as follows:
Failure scenario |
Disaster recovery operation |
Applicable to |
Recovery location |
Trigger method |
Operation granularity |
|---|---|---|---|---|---|
| Some compute nodes are unavailable, but there are available nodes in the availability zone (AZ), or other nodes can be added. | REPLACE LS |
User tenant | Within the AZ | Manual execution, one by one | Log stream |
| Some compute nodes are unavailable, and there are no available nodes in the AZ and no other nodes can be added. | ALTER TENANT (change the locality) |
User tenant | Cross-AZ | Manual execution + automatic disaster recovery task initiated by RS | Tenant |
| All compute nodes in the entire AZ are unavailable. | REPLACE TENANT (restore the sys tenant) + change the locality (restore user tenants one by one) |
System tenant and user tenant | Cross-AZ | Manual execution + automatic disaster recovery task initiated by RS | Tenant |
Scenario 1: Some compute nodes are unavailable, but there are available nodes within the availability zone (AZ), or additional nodes can be added.
When you replace a log stream within an availability zone (AZ), the system does not automatically trigger a Replace task. You need to query the cluster to identify the log streams on the failed server (you can query by SVR_IP and SVR_PORT in the CDB_OB_LS_LOCATIONS view), and then execute the REPLACE LS command for these log streams one by one to rebuild the failed log streams.
The syntax for the REPLACE LS command is as follows:
ALTER SYSTEM REPLACE LS [=] <ls_id>
SERVER [=] '<ip>:<rpc_port>'
[TENANT [=] '<tenant_name>'];
The relevant parameters are described as follows:
LS: the log stream ID, which must be an integer, such as1or1001. It cannot be specified as a string.SERVER: the RPC address of the target node, which must be in theip:rpc_portformat. If you specify only the IP address but not the port, a parameter error is returned.TENANT: (Optional) the tenant to which the log stream belongs. When you use thesystenant to operate on a Meta tenant, the format isMETA$<user_tenant_id>, for example,META$1002. If this parameter is omitted, the tenant of the current session is used.
Note
- The clause order is fixed: you must write
LSfirst, thenSERVER, and finally the optionalTENANT. Reversing the order will result in a syntax error. - Incomplete forms such as writing only
REPLACE SERVERorREPLACE TENANTare not supported.
Using the user tenant mysql_tenant (tenant ID 1002) as an example, the procedure is as follows:
Add or enable a new node within the availability zone (AZ), and execute
ALTER SYSTEM ADD SERVERto join the new node to the cluster:ALTER SYSTEM ADD SERVER 'ip:port' ZONE 'zone1';Migrate the units of this tenant to the new node. The
REPLACE LScommand can only be executed if the new node has units of this tenant:ALTER SYSTEM MIGRATE UNIT unit_id DESTINATION 'ip:port';Here,
unit_idis the ID of the unit to be migrated, andip:portis the IP and RPC port of the new node.Execute the following
REPLACE LScommands in sequence:Notice
Before executing the
REPLACE LScommand, the following conditions must be met. Failure to meet any one will result in an error:- The target node is alive, allows migration in, is not permanently offline, and already has a unit of this tenant.
- All machines in the log stream member list are unavailable (
INACTIVE). As long as one machine is still alive, the command will be rejected. - The target node does not already have a replica of this log stream, and no other disaster recovery tasks are currently running on this log stream.
REPLACE LSis recommended to be executed in the sys tenant; when executed in a user tenant, it can only operate on that tenant.First, restore the Meta tenant log stream:
ALTER SYSTEM REPLACE LS = 1001 SERVER = 'ip:port' TENANT = 'META$1002'; ALTER SYSTEM REPLACE LS = 1 SERVER = 'ip:port' TENANT = 'META$1002';Restore the user tenant's log stream:
ALTER SYSTEM REPLACE LS = 1 SERVER = 'ip:port' TENANT = 'mysql_tenant';
If there are other user log streams, proceed to restore them one by one.
A successful command return only indicates that the task has been accepted by the system. It does not mean that the log stream has been restored. Wait until the task is completed before you execute the next command.
To determine whether a task is successfully executed, all of the following conditions must be met:
The
REPLACE REPLICAtask for that tenant andls_idno longer exists in theCDB_OB_LS_REPLICA_TASKSview:SELECT * FROM oceanbase.CDB_OB_LS_REPLICA_TASKS WHERE TENANT_ID = 1002 AND LS_ID = 1 AND TASK_TYPE = 'REPLACE REPLICA';The corresponding historical record in the
CDB_OB_LS_REPLICA_TASK_HISTORYview has aTASK_STATUSofCOMPLETEDand anEXECUTE_RESULTofOB_SUCCESS:SELECT TASK_STATUS, EXECUTE_RESULT, TASK_TYPE, GMT_CREATE FROM oceanbase.CDB_OB_LS_REPLICA_TASK_HISTORY WHERE TENANT_ID = 1002 AND LS_ID = 1 AND TASK_TYPE = 'REPLACE REPLICA' ORDER BY GMT_CREATE DESC;
When querying within a user tenant, you can use the
DBA_OB_LS_REPLICA_TASKSandDBA_OB_LS_REPLICA_TASK_HISTORYviews (which contain only data for the current tenant).If the
TASK_STATUSin the historical record is notCOMPLETEDor theEXECUTE_RESULTis notOB_SUCCESS, the task failed. Decide whether to retry based on the error message.After a task succeeds, you can also verify business availability in the following ways:
Confirm that the leader of that log stream is located at the target node:
SELECT TENANT_ID, LS_ID, ROLE, SVR_IP, SVR_PORT FROM oceanbase.CDB_OB_LS_LOCATIONS WHERE TENANT_ID = 1002 AND LS_ID = 1 AND ROLE = 'LEADER';Perform simple read and write operations on that tenant (such as creating tables, inserting data, and querying) to confirm normal business operation.
After all log streams for that tenant are restored, confirm that unit migration is complete (
DBA_OB_UNITSview: fieldsMIGRATE_FROM_SVR_IPandMIGRATE_FROM_SVR_PORTare cleared), and delete the failed server according to operational specifications.
Scenario 2: Some compute nodes are unavailable, and there are no available nodes or additional nodes can be added within the zone (AZ)
Before restoration, add sufficient compute nodes to the new zone (AZ) to host the restored log stream replicas. Also, confirm that all compute nodes for that tenant in the failed zone (AZ) are in the INACTIVE state (queryable via the DBA_OB_SERVERS view). For each user tenant in the cluster, perform the following operations in sequence:
Lock the tenant.
ALTER TENANT mysql_tenant LOCK;Expand the tenant resource pool's
ZONE_LIST. A tenant with single-replica deployment has only one resource pool. By adding zones from the new zone (AZ) to the resource pool'sZONE_LIST, resources are provided for the restored log stream replicas.ALTER RESOURCE POOL mysql_tenant_pool ZONE_LIST = ('zone1', 'zone2');Directly change the tenant's locality from the failed zone (AZ) to the new zone (AZ). After the locality change, RS automatically initiates a single-replica disaster recovery task, starting the tenant's log stream replica in the new zone (AZ).
ALTER TENANT mysql_tenant LOCALITY = 'F@zone2';Determine whether the disaster recovery task is complete. You can determine this by using any of the following methods:
The
JOB_STATUScolumn in theDBA_OB_TENANT_JOBSview for the corresponding task isSUCCESS.SELECT * FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = 1002;The
PREVIOUS_LOCALITYcolumn for the tenant in theDBA_OB_TENANTSview isNULL.SELECT TENANT_ID, TENANT_NAME, LOCALITY, PREVIOUS_LOCALITY FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_ID = 1002;
Note
If there are unfinished resource specification change tasks for the tenant before the single-replica disaster recovery task is initiated, the
JOB_STATUSinDBA_OB_TENANT_JOBSis marked asSUCCESSonly after the previous task is completed. In this case, you can determine the result based on whetherPREVIOUS_LOCALITYin theDBA_OB_TENANTSview isNULL.Modify the tenant's
PRIMARY_ZONEto change the primary availability zone (AZ) from the failed AZ to the new AZ.ALTER TENANT mysql_tenant PRIMARY_ZONE='zone2';After the disaster recovery task is complete, narrow the tenant's resource pool by retaining only the zones in the new AZ and removing the zones from the failed AZ.
ALTER RESOURCE POOL mysql_tenant_pool ZONE_LIST = ('zone2');Unlock the tenant and restore business access.
ALTER TENANT mysql_tenant UNLOCK;
Scenario 3: All compute nodes in an availability zone (AZ) are unavailable
Before recovery, add sufficient compute nodes to the new AZ to host the restored log stream replicas. Then perform the recovery in the following order:
Execute
REPLACE TENANTon an idle server in the new AZ to restore thesystenant.ALTER SYSTEM REPLACE TENANT = 'sys' REGION = 'region_name' ZONE = 'zone2' SERVER = 'ip:port' LOGSERVICE_ACCESS_POINT 'ip:port' SHARED_STORAGE_INFO 'shared_storage_info';This statement starts the log streams of the
systenant on an idle server in the new AZ.After the
systenant is restored, modify its locality.ALTER TENANT sys LOCALITY = 'F@zone2';Note
If the operation
ALTER TENANT sys LOCALITY = 'F@zone2';fails, you can restart RS and try the command again.For each user tenant in the cluster, perform steps 1 through 7 from scenario 2 to restore that tenant's log stream replicas from the failed AZ to the new AZ.