Restarting is a common O&M task. It is suitable for scenarios such as brief server maintenance or applying system configuration changes that require a restart to take effect. The restart duration must be shorter than the time specified by the cluster-level parameter server_permanent_offline_time. Otherwise, the node will be permanently marked as offline. If the server requires prolonged maintenance, you need to follow the server replacement procedure. For more information, see Replace a node.
Note
The cluster-level parameter server_permanent_offline_time sets the threshold for how long a node's heartbeat can be interrupted before it is considered permanently offline. Data replicas on a permanently offline node must be automatically replenished. The default value is 3600s. For more information about this parameter, see server_permanent_offline_time.
Background information
OceanBase AI Database consists of two components: compute nodes and the log service. Compute nodes use the Paxos protocol to directly write logs into multiple replicas of the log service for persistence, thereby maintaining data consistency among replicas. When a minority of replicas are unavailable (a three-replica log service cluster allows one replica to be unavailable), the read and write operations of compute nodes are not affected, and they can still provide services normally.
The STOP SERVER command enables a lossless restart in a multi-replica architecture. When executed, the STOP SERVER command performs the following operations:
- Marks the node to be restarted as stopped in the Root Service (the node status is set to
ACTIVE, and thestop_timefield is updated to a value greater than 0). The client detects this status and avoids routing business requests to the stopped node.
Once the STOP SERVER command is successfully executed, restarting the node will not trigger leader re-elections or client errors, ensuring complete transparency to business traffic. If the STOP SERVER command fails, you must halt the restart process and investigate the cause. Potential reasons for failure include insufficient replicas or redo log delays.
Procedure
The procedure for restarting a node involves the following steps: stopping services, performing a minor compaction, shutting down the process, starting the process, and restarting services.
This topic provides guidance for restarting one node in a cluster. If you want to restart multiple nodes, you can repeat the same steps for each node.
Log in to the
systenant of the cluster as therootuser.Make sure to replace the sample parameters in the following command with the actual configurations of your database environment.
obclient -h10.xx.xx.xx -P2883 -uroot@sys#obdemo -p***** -AFor more detailed instructions on connecting to a database, see Overview of connection methods.
Run the following command to isolate the node.
During the restart process, service continuity may be interrupted. For example, if the cluster contains only one or two nodes, or if the data of a tenant is distributed across only two nodes, the system may become unavailable during the restart. The
Stop Serveroperation ensures service continuity during the restart by isolating the node. Once the node is successfully isolated, it will no longer provide services. If theStop Serveroperation fails, troubleshoot the issue based on the error messages, adjust the cluster deployment if necessary, and retry the operation. Alternatively, if acceptable, you can skip this step and proceed with stopping the service directly.ALTER SYSTEM STOP SERVER 'svr_ip:svr_port';Parameter descriptions:
svr_ip: the IP address of the node to be stopped.svr_port: the RPC port of the node to be stopped. The default value is 2882.
Example:
obclient [(none)]> ALTER SYSTEM STOP SERVER '172.xx.xx.xx:2882';After successful execution, query the
STATUScolumn of the specified server in theoceanbase.DBA_OB_SERVERSview. The value of this column remainsACTIVE, but theSTOP_TIMEcolumn changes fromNULLto the time when the service was stopped.For more information about how to query the
oceanbase.DBA_OB_SERVERSview, see View a node.Run the following command to perform a minor compaction on the node to shorten the time required for redo log replay after the restart and speed up the restart.
ALTER SYSTEM MINOR FREEZE SERVER = ('svr_ip:svr_port');Parameter descriptions:
svr_ip: the IP address of the node to be restarted.svr_port: the RPC port of the node to be restarted. The default value is 2882.
Example:
obclient [(none)]> ALTER SYSTEM MINOR FREEZE SERVER = ('172.xx.xx.xx:2882');Wait for the minor compaction to complete before proceeding to the next step. For more information about how to check the minor compaction progress, see View minor compaction information.
Stop the observer process.
Log in to the server where the observer process is running as the
adminuser.Navigate to the
/home/admin/oceanbasedirectory using the command line.[admin@xxx /]$ cd /home/admin/oceanbaseRun the following command to view and obtain the process ID of the node.
[admin@xxx oceanbase]$ ps -ef | grep observer | grep -v grep admin 103364 1 99 2022 ? 51-17:24:41 /home/admin/oceanbase/bin/observerIn this example,
103364is the process ID of the node.Stop the observer process.
The sample command is as follows:
[admin@xxx oceanbase]$ kill -9 pidHere,
pidis the observer process ID of the node to be stopped.Example:
[admin@xxx oceanbase]$ kill -9 103364Notice
You can stop only one observer process in a deployment directory. If you want to stop observer processes on multiple nodes, you need to log in to each server in sequence.
Confirm that the observer process has stopped by running the following command:
[admin@xxx oceanbase]$ ps aux | grep observerIf no information is returned, the process has stopped successfully.
(Optional) If necessary, perform maintenance on the server during this step.
Start the observer process.
Log in to the server where the observer process is to be started as the
adminuser.Start the observer process.
[admin@xxx oceanbase]$ cd /home/admin/oceanbase && ./bin/observerNotice
You can start only one observer process in a deployment directory. If you want to start observer processes on multiple nodes, you need to log in to each server in sequence.
After successful execution, query the
START_SERVICE_TIMEcolumn in theoceanbase.DBA_OB_SERVERSview. If the value of this column is notNULL, the observer process has started successfully.
Run the following command to restart the node's services:
ALTER SYSTEM START SERVER 'svr_ip:svr_port';where:
svr_ip: the IP address of the node to be started.svr_port: the RPC port of the node to be started. The default value is 2882.
Example:
obclient [(none)]> ALTER SYSTEM START SERVER '172.xx.xx.xx:2882';After successful execution, query the
STOP_TIMEcolumn in theoceanbase.DBA_OB_SERVERSview. If the value of this column isNULL, the node services have started successfully, and the node is ready to provide services.For more information about querying the
oceanbase.DBA_OB_SERVERSview, see View a node.
References
For more information about node O&M, see the following topics: