You can isolate a node for fault isolation or O&M changes. After a node is isolated, new read and write requests will not be routed to it. In this way, the fault is isolated or you can perform lossless O&M changes.
Fault isolation: When an exception occurs on a node, you must isolate this node from the business traffic. For example, if there is a switch malfunction in an IDC, some nodes may encounter packet loss, retransmissions, and even elections without a leader. In this scenario, you can isolate these faulty nodes for quick recovery. After the nodes resume normal, you can start them to restore the node traffic.
O&M changes: When a node requires O&M changes, make sure that the O&M changes are transparent to the business traffic. For example, when a node must be restarted, to avoid affecting the business traffic, you can isolate the node and switch the business traffic from this node to other nodes. After the observer process on the node is restarted, you can start the node.
Isolating a node simply involves diverting the business traffic away from that node without changing the number of voting members in the Paxos group. Different isolation commands provide different levels of security guarantees, and it is necessary to carefully consider the subsequent actions that can be safely executed based on the specific isolation command being executed. Isolation commands are suitable for scenarios where short-term isolation is required but continuous monitoring is necessary. It is important to have contingency plans for secondary failures in advance. If the recovery cannot be achieved within a short period, the node can be completely removed by replacing it. For more information about how to replace a node, see Replace a node.
Three isolation commands with different levels of security guarantees are provided:
STOP SERVERFORCE STOP SERVERISOLATE SERVER
After you isolate a node by using any of the preceding commands, if the value of the STATUS field is still ACTIVE but the value of the STOP_TIME field changes from NULL to the point in time when the node is isolated, the node is in the Stopped state.
STOP SERVER
You can use STOP SERVER to isolate multiple server nodes at a time. The syntax is as follows:
ALTER SYSTEM STOP SERVER 'svr_ip1:svr_port1', 'svr_ip2:svr_port2', ...;
STOP SERVER has the highest security level. The STOP SERVER command can switch the business traffic away from the target node to isolate the node from the business traffic. It can also ensure that other nodes are still the majority in the Paxos group. In this way, you can securely perform all actions on the isolated node. For example, you can run the pstack command, adjust the log level, or stop the observer process. When you need to isolate a fault or stop a node for maintenance, you can use the STOP SERVER command, which is the best choice in this case.
The precheck for the STOP SERVER command is strict to ensure that any actions can be securely performed on the isolated node. Various internal status checks are performed to ensure that after the observer process on the target node is stopped, the replicas on the remaining available nodes in the cluster are still the majority in the Paxos group, thereby ensuring the continuity of database services.
Limitations of the STOP SERVER command are as follows:
You cannot stop nodes in multiple zones at the same time. If you stop nodes in multiple zones at the same time or a node in another zone is already stopped, the error
-4660, cannot stop server or stop zone in multiple zonesis returned. In this case, you can query theDBA_OB_SERVERSandDBA_OB_ZONESviews to check whether nodes in multiple zones are stopped.The
STOP SERVERcommand checks the Paxos member list of all log streams for all tenants and must meet the following conditions:Inactive nodes include those in the
INACTIVE,Stopped(ACTIVEstate with astop_timefield greater than 0), orDELETINGstate.To check the Paxos member list of log streams, all log streams must have a leader. If a log stream is found without a leader, an error
-4179is returned, and information about the problematic log stream is displayed.For example, if log stream 1 of tenant
1001has no leader, the error message isTenant(1001) LS(1) has no leader, stop server not allowed. To resolve this issue, you can query theGV$OB_LOG_STATview to confirm the leader replica information of the log stream and investigate the abnormal leader replica.If the command to change a tenant's locality is executed concurrently with the
STOP SERVERcommand, to prevent misjudgment, stopping the node during locality modification is prohibited.For example, if tenant
1001is reducing its locality by one replica, and the log stream member list is changing from A, B, C to A, B, stopping node B may be misjudged as allowed by theSTOP SERVERcommand, conflicting with the replica reduction operation. To avoid this, theSTOP SERVERcommand returns an error-4179and displays the information about the changing tenant:Tenant(1001) locality is changing, stop server not allowed. To resolve this, you can query theDBA_OB_TENANTSview to confirm whether there is a tenant locality change operation.
After successful execution,
STOP SERVERensures that any invasive actions can still be performed on the isolated node.Log synchronization judgment condition: Compare the latest log timestamps of the follower and leader replicas of a log stream. If the difference is within 5 seconds, the logs are considered synchronized. You can query the
END_SCNfield in theGV$OB_LOG_STATview to obtain the latest log timestamp of each replica for a specified log stream.If this condition is not met, an error
-4179is returned, and information about the problematic log stream is displayed.For example, if log stream 1 of tenant
1001is not synchronized, the error message isTenant(1001) LS(1) log not sync, stop server not allowed. To resolve this, you can query theEND_SCNfield in theGV$OB_LOG_STATview to confirm the synchronization status of the log stream replicas.
# Retrieves the latest log position of a specified log stream replica and converts it into a timestamp format. obclient [(none)]> SELECT SCN_TO_TIMESTAMP(END_SCN) FROM GV$OB_LOG_STAT WHERE TENANT_ID = 1001 and LS_ID = 1;
STOP SERVER is the most secure isolation command and has the strictest prerequisites. In actual application scenarios especially in the case of emergency response to faults, the preceding prerequisites may not be fully met. In this case, the STOP SERVER command will fail. Therefore, OceanBase AI Database provides another two node isolation commands: FORCE STOP SERVER and ISOLATE SERVER. The two commands have less strict prerequisites and limits and apply to more scenarios. However, after a node is isolated by using one of the two commands, the actions that can be performed are limited. The STOP SERVER command is the best choice in all scenarios. If STOP SERVER is inapplicable, you can choose the other two isolation commands.
FORCE STOP SERVER
Compared to the STOP SERVER command, FORCE STOP SERVER skips the log synchronization check. The FORCE STOP SERVER command ensures that business traffic is diverted away from the isolated node, but does not guarantee that the process on the isolated node can be stopped safely.
Typical application scenario for FORCE STOP SERVER: Consider three nodes A, B, and C. Node B has a significant log delay, for example, 1 hour. At this point, node C experiences a hardware failure and requires maintenance downtime. To ensure the process can be stopped safely after isolation, we would normally prefer to use the STOP SERVER command to isolate the node. However, due to the large log delay on node B, the STOP SERVER command cannot execute successfully. We then have to resort to using the FORCE STOP SERVER command to isolate the node first, to prevent the hardware failure from affecting business traffic. Then, we investigate the cause of the log delay on node B. Once node B's logs are real-time again, we attempt to execute the STOP SERVER command again. After successful execution, maintenance downtime can proceed safely.
ALTER SYSTEM FORCE STOP SERVER 'svr_ip1:svr_port1', 'svr_ip2:svr_port2', ...;
Warning
FORCE STOP SERVER is only applicable when log delays exist in nodes other than the one to be isolated. In this case, the purpose of isolating the node is solely for fault isolation, and no invasive actions that could stop the process are allowed.
ISOLATE SERVER
Compared with STOP SERVER and FORCE STOP SERVER, ISOLATE SERVER has the loosest pre-checks and the fastest response.
ALTER SYSTEM ISOLATE SERVER 'svr_ip1:svr_port1', 'svr_ip2:svr_port2', ...;
The logic for relaxing pre-checks for the ISOLATE SERVER command is as follows:
The
ISOLATE SERVERcommand no longer restricts isolating nodes from multiple zones simultaneously. If a node in another zone has already been isolated (by executing any isolation command), you can still execute theISOLATE SERVERcommand.Typical scenario: The cluster has two zones (Z1 and Z2). A node in each zone (Z1 and Z2) fails and needs to be isolated. After the failed node in Z1 is successfully isolated, if you use the
STOP SERVERorFORCE STOP SERVERcommand to isolate the failed node in Z2, an error will be returned indicating that isolating multiple zones at the same time is not allowed. In this case, you can only execute theISOLATE SERVERcommand to divert business traffic away from the failed node. No invasive actions that cause the process to stop are allowed after isolation.The
ISOLATE SERVERcommand no longer checks the log synchronization status of log stream replicas. Even if the corresponding conditions are not met, you can still execute theISOLATE SERVERcommand.Typical scenario: The initial Paxos member list for a log stream includes nodes A, B, and C. After node C's machine crashes and permanently goes offline, the log stream's member list changes to A and B, resulting in a replica deficit. If node B then experiences a hardware failure and you want to isolate it, you can execute the
ISOLATE SERVERcommand to divert business traffic away from the failed node. No invasive actions that cause the process to stop are allowed after isolation.
Limitations of the ISOLATE SERVER command:
When multiple zone nodes are being isolated, ISOLATE SERVER checks the Primary Zone attribute of all tenants. It requires that you cannot isolate nodes from all zones in the region where a tenant's Primary Zone is located simultaneously, to prevent the tenant's read/write service from being switched to another region, which could lead to significantly increased latency due to cross-city access requests.
You can check a tenant's Primary Zone attribute using the DBA_OB_TENANTS view, and combine this with information about isolated zones and nodes from the DBA_OB_ZONES and DBA_OB_SERVERS views to confirm whether the above limitations are being violated.
Warning
The ISOLATE SERVER command is only applicable when there are abnormal node states other than the one to be isolated. In this case, the purpose of isolating the node is merely for fault isolation; no invasive actions that cause the process to stop are allowed.
Summary
In any scenario, the STOP SERVER command should be the first choice. After successful execution, it not only achieves fault isolation but also ensures that any emergency or maintenance actions can be safely performed on the isolated node. If STOP SERVER fails, you can choose the weakened version of the node isolation command, but after weakening the isolation, it cannot guarantee that invasive actions can be safely performed.
FORCE STOP SERVER is applicable when there is log lag in nodes other than the one to be isolated. ISOLATE SERVER is applicable when there are abnormal node states other than the one to be isolated.
References
For more node-related O&M operations, see the following topics: