Experience high availability and failover

High availability is the reliability foundation of distributed databases. OceanBase Database adopts a shared-nothing distributed architecture, where data is stored as multiple replicas across multiple nodes. Data consistency among replicas is ensured through a multi-Paxos-based distributed consensus protocol. When a node fails, such as a crash, the system can automatically elect a new leader replica to continue providing services without manual intervention, achieving disaster recovery capabilities with RPO=0 and RTO<8 seconds.

This topic covers the following experiences:

• Understand the high availability architecture: Understand core concepts such as zones, replicas, log streams, leader and follower replicas, and the Paxos majority.
• Observe replica and leader distribution: View the locations, roles, and synchronization status of log stream replicas through system views.
• Simulate process exceptions: Forcefully terminate an observer process and observe the change in service availability.
• Experience service recovery with zero data loss: Restart the process to restore service and verify that the data remains completely consistent before and after the failure.

Prerequisites

Before you proceed, ensure the following conditions are met:

Overview of the high availability architecture

The architecture and implementation of OceanBase Database ensure strong data consistency, integrity, and high availability. Understanding the following core concepts will help you get started with the experiences in this topic:

In OceanBase Database, user read and write requests are served by the leader replica of a log stream. All modifications generate logs, which are synchronized to other follower replicas via the Multi-Paxos protocol. If the node hosting the leader fails, a follower is elected as the new leader to continue serving requests. The election service is the cornerstone of high availability: in V4.x, the election service operates at the log stream granularity and no longer relies on external clock synchronization services like NTP. Instead, it ensures consistency through local clocks and an improved lease mechanism.

View replica and leader distribution

Log streams have location information (the nodes on which they are distributed) and role information (LEADER or FOLLOWER). This section shows how to view the distribution of log stream replicas for all tenants in a cluster from the sys tenant through system views.

Procedure

1. Log in to the sys tenant of the cluster as the root user (the password is randomly generated during deployment; for retrieval, see the prerequisites above):

   obclient -h127.0.0.1 -uroot@sys -P2881 -p -A oceanbase
   

2. View the distribution of log stream replicas for all tenants through the CDB_OB_LS_LOCATIONS view:

   obclient [oceanbase]> SELECT TENANT_ID, LS_ID, SVR_IP, ZONE, ROLE, PAXOS_REPLICA_NUMBER, REPLICA_TYPE
                       FROM oceanbase.CDB_OB_LS_LOCATIONS;
   

   A sample return result is as follows (the example environment already has a user tenant named ha_tenant with tables in it, and the actual tenant IDs vary with the creation order. If you have just finished the deployment and have not created any user tenant or business tables yet, fewer rows are returned here, which is normal):

   +-----------+-------+-----------+-------+--------+----------------------+--------------+
   | TENANT_ID | LS_ID | SVR_IP    | ZONE  | ROLE   | PAXOS_REPLICA_NUMBER | REPLICA_TYPE |
   +-----------+-------+-----------+-------+--------+----------------------+--------------+
   |         1 |     1 | 127.0.0.1 | zone1 | LEADER |                    1 | FULL         |
   |      1001 |     1 | 127.0.0.1 | zone1 | LEADER |                    1 | FULL         |
   |      1002 |     1 | 127.0.0.1 | zone1 | LEADER |                    1 | FULL         |
   |      1002 |  1001 | 127.0.0.1 | zone1 | LEADER |                    1 | FULL         |
   +-----------+-------+-----------+-------+--------+----------------------+--------------+
   4 rows in set
   

   The key fields are described as follows:

   Field	Result interpretation
   TENANT_ID	Tenant ID. 1 is the sys tenant. Each user tenant is accompanied by a Meta tenant (an internal system tenant used to store and manage the tenant's own metadata, not directly accessible by users), which can be identified by tenant names starting with META$ in the DBA_OB_TENANTS view. In the example, 1001 is the Meta tenant and 1002 is the user tenant ha_tenant.
   LS_ID	Log stream ID. Each tenant has a system log stream numbered 1. User tenants do not have a user log stream immediately after creation. After a user table is created in a tenant, the system automatically creates user log streams numbered 1001 and later. Business data is stored on these user log streams (in the example, log stream 1001 for tenant 1002 appears after its table is created).
   SVR_IP / ZONE	Node and zone where the replica resides. In a single-node cluster, all replicas are located on the same node and in the same zone.
   ROLE	Replica role, always LEADER: a single-replica log stream has no followers; its sole replica is the leader.
   PAXOS_REPLICA_NUMBER	Number of Paxos replicas, always 1. In a multi-replica cluster, this value is typically 3 (or 5). In this case, the same log stream is represented by multiple rows across different nodes and zones, with only one row having ROLE as LEADER.
   REPLICA_TYPE	Replica type. FULL indicates a full-featured replica, possessing complete data and functionality including RedoLog, MemTable, and SSTable. It is a required replica.

3. View the role and synchronization status of log streams through the V$OB_LOG_STAT view:

   obclient [oceanbase]> SELECT TENANT_ID, LS_ID, SVR_IP, SVR_PORT, ROLE, PAXOS_REPLICA_NUM, IN_SYNC
                       FROM oceanbase.V$OB_LOG_STAT;
   

   When you query this view in the sys tenant, it displays the status of the log streams of all tenants. A sample return result is as follows:

   +-----------+-------+-----------+----------+--------+-------------------+----------+
   | TENANT_ID | LS_ID | SVR_IP    | SVR_PORT | ROLE   | PAXOS_REPLICA_NUM | IN_SYNC  |
   +-----------+-------+-----------+----------+--------+-------------------+----------+
   |         1 |     1 | 127.0.0.1 |     2882 | LEADER |                 1 | YES      |
   |      1001 |     1 | 127.0.0.1 |     2882 | LEADER |                 1 | YES      |
   |      1002 |     1 | 127.0.0.1 |     2882 | LEADER |                 1 | YES      |
   |      1002 |  1001 | 127.0.0.1 |     2882 | LEADER |                 1 | YES      |
   +-----------+-------+-----------+----------+--------+-------------------+----------+
   4 rows in set
   

   The V$OB_LOG_STAT view shows the status of the log stream Palf (the Paxos-based logging module). It is commonly used to check whether a log stream has a leader, its member list, and the replica synchronization status. In the results, if ROLE is LEADER, it indicates that the replica is the primary replica for the log stream; if IN_SYNC is YES, it indicates that the replica is fully synchronized with the leader.

Simulate a process exception and observe service recovery

This section simulates a scenario where an observer process crashes abnormally (for example, when it is forcibly terminated by the operating system), observes the change in service availability, then restarts the process to restore service and verifies data consistency before and after the failure.

Procedure

1. Create an experience tenant and prepare verification data. Execute the following command in the sys tenant to create the tenant ha_tenant (for detailed information about tenants and resource pools, see Experience multi-tenancy and resource isolation). If you already have a business tenant, you can use it directly:

   obclient [oceanbase]> CREATE RESOURCE UNIT ha_unit MAX_CPU=1, MIN_CPU=1, MEMORY_SIZE='2G';
   obclient [oceanbase]> CREATE RESOURCE POOL ha_pool UNIT='ha_unit', UNIT_NUM=1, ZONE_LIST=('zone1');
   obclient [oceanbase]> CREATE TENANT IF NOT EXISTS ha_tenant CHARSET='utf8mb4', PRIMARY_ZONE='zone1', RESOURCE_POOL_LIST=('ha_pool') SET ob_tcp_invited_nodes='%';
   

2. Log in to ha_tenant as the root user (the default password for the root user of a newly created tenant is empty, so press Enter when prompted), then create a test table and write data:

   obclient -h127.0.0.1 -uroot@ha_tenant -P2881 -p -A test
   

   obclient [test]> CREATE TABLE ha_demo_order (id INT PRIMARY KEY, item VARCHAR(50), amount DECIMAL(10,2));
   Query OK, 0 rows affected
   
   obclient [test]> INSERT INTO ha_demo_order VALUES (1, 'keyboard', 149.00), (2, 'mouse', 79.00), (3, 'monitor', 999.00);
   Query OK, 3 rows affected
   
   obclient [test]> SELECT COUNT(*) FROM ha_demo_order;
   +----------+
   | COUNT(*) |
   +----------+
   |        3 |
   +----------+
   1 row in set
   

   Record the current row count 3 as the baseline before the failure. Keep the current obclient session open, because you will observe what happens to it during the failure later on.

3. Open a new terminal, locate the observer process, and forcibly terminate it to simulate an abnormal process crash:

   # Find the observer process.
   ps -ef | grep observer | grep -v grep
   
   # Forcefully terminate the observer process (PID is based on the result of the previous query).
   kill -9 <PID>
   

4. Back in the ha_tenant session, execute the query again:

   obclient [test]> SELECT COUNT(*) FROM ha_demo_order;
   ERROR 2013 (HY000): Lost connection to MySQL server during query
   No connection. Trying to reconnect...
   ERROR 2002 (HY000): Can't connect to OceanBase server on '127.0.0.1' (115)
   

   The connection is interrupted: the client's automatic reconnection attempt fails and the service becomes unavailable. A new connection attempt also fails:

   obclient -h127.0.0.1 -uroot@ha_tenant -P2881 -p -A test
   ERROR 2002 (HY000): Can't connect to OceanBase server on '127.0.0.1' (115)
   

   After the unique node in a single-replica cluster fails, no other replicas can take over the service, and the service remains unavailable until the process is restarted.

5. In the terminal, restart the cluster (obd demo deploys clusters with the fixed name demo; if the cluster is not managed by obd, restart the observer process using the startup command recorded during deployment):

   obd cluster restart demo
   

   Wait for the command to complete and all component statuses to change to running.

6. Back in the ha_tenant session, execute the query again. The client will automatically reconnect and return the results:

   obclient [test]> SELECT COUNT(*) FROM ha_demo_order;
   No connection. Trying to reconnect...
   Connection id:    3221700101
   Current database: test
   
   +----------+
   | COUNT(*) |
   +----------+
   |        3 |
   +----------+
   1 row in set
   

   The query returns normally, with the same number of rows as before the failure, indicating zero data loss. The Connection id in the output is the connection ID of the new session after reconnection (the value varies by environment), and Current database indicates that the original database context is still maintained. This is because, in single-replica mode, a transaction commit is only successful after Redo logs are persisted to disk; an abnormal process crash does not result in any lost committed transactions.

Interpretation

Failover in a multi-replica cluster

In the single-node environment described in this topic, services are interrupted when a node fails and require manual recovery. However, in a typical three-replica cluster in a production environment (where one OBServer node is deployed in each of three zones), the behavior is completely different when a node fails:

In a three-replica cluster, the three replicas of each log stream are distributed across different nodes in three zones, forming a Paxos group. At any given time, only one leader provides read and write services. When the node hosting the leader fails:

1. The remaining two replicas form a majority, ensuring no data loss and that majority logs can still be committed.
2. The election service quickly elects a new leader from the remaining replicas (triggered after a lease expires, typically completed within seconds).
3. Subsequent business requests are routed to the new leader, and services automatically recover without manual intervention.
4. After the failed node recovers and rejoins the cluster, Root Service automatically replenishes the missing replicas, restoring the three-replica state.

Next steps

After you complete the experiences in this topic, you can explore the following:

1. Experience horizontal scaling and elastic scaling: Explore the scaling capabilities at the compute, storage, and tenant resource levels. Observe resource changes and business capacity impacts before and after scaling. For more information, see Experience horizontal scaling and elastic scaling.
2. Experience parallel import and high compression: Use the parallel import capability to complete large-scale data ingestion and compare storage usage before and after compression. For more information, see Experience parallel import and high compression.
3. Deepen understanding of multi-tenancy and resource isolation: Create mutually isolated tenants for different businesses within a cluster. For more information, see Experience multi-tenancy and resource isolation.

References

• For detailed information on the high availability architecture, Multi-Paxos protocol, and election service, see Overview of high availability architecture.
• For detailed information on replica concepts and replica types (full-featured/read-only), see Replica overview.
• For the relationship between partitions, shards, log streams, and replicas, see Log stream and replica overview.
• For descriptions of the views for viewing log stream replica distribution, see DBA_OB_LS_LOCATIONS and CDB_OB_LS_LOCATIONS.
• For descriptions of the views for viewing log stream roles and synchronization status, see V$OB_LOG_STAT.
• For deployment methods of multi-node clusters, see Deploy OceanBase Database in a production environment using the command line.