Deploy an OceanBase cluster with two replicas and an arbitration service using the CLI

Starting from V4.1, OceanBase Database supports the arbitration service (AS), which addresses the issue of increased round-trip time (RT) in the case of failures of the primary and standby replicas in a geo-disaster recovery (GDR) scenario, reduces cross-region bandwidth usage, and keeps the costs of the third region at a minimum.

This topic will guide you through the deployment of an OceanBase cluster with two replicas and an AS using the CLI.

Prerequisites

Before installing OceanBase Database, make sure that the following conditions are met:

• The OBServer node has been configured. For more information, see Server configuration, (Optional) Configure a clock source, and Initialize the OBServer node using oat-cli.

• You have obtained the RPM package of OceanBase Database. For more information, see Prepare installation packages.

Procedure

Step 1: Deploy an OceanBase cluster

1. Install the RPM packages on the two OBServer servers.

   1. Install the OceanBase Database RPM package.

      Switch to the directory where the OceanBase RPM package is stored:

      [root@xxx /]# cd $rpm_dir
      

      Install the OceanBase RPM package:

      [root@xxx $rpm_dir]# rpm -ivh $rpm_name [--prefix=/home/admin/oceanbase]
      

      Here, $rpm_dir indicates the directory where the RPM package is stored, and $rpm_name indicates the name of the RPM package.

      Note

      After installation, OceanBase Database will be installed in the /home/admin/oceanbase directory by default.

      Here is an example:

      [root@xxx /home/admin/rpm]# rpm -ivh oceanbase-4.2.0.0-100000052023073123.el7.x86_64.rpm
      Preparing...                          ################################# [100%]
      Updating / installing...
          1:oceanbase-4.2.0.0-100000052023073################################# [100%]
      

   2. (Optional) Install OceanBase Client.

      OceanBase Client (OBClient) is a dedicated CLI tool for OceanBase Database. With OBClient, you can connect to a MySQL tenant and an Oracle tenant of OceanBase Database. If you only want to connect to a MySQL tenant, you can also use a MySQL client (mysql) to connect to OceanBase Database.

      Notice

      Due to the dependency of obclient V2.2.1 and earlier on libobclient, you must install libobclient first. Contact your technical support representative to obtain the RPM packages of OBClient and libobclient.

      Here is an example:

      [root@xxx /home/admin/rpm]# rpm -ivh obclient-2.2.1-20221122151945.el7.alios7.x86_64.rpm
      Preparing...                          ################################# [100%]
      Updating / installing...
         1:obclient-2.2.1-20221122151945.el7################################# [100%]
      

      Check whether OBClient has been installed:

      [root@xxx /home/admin/rpm]# which obclient
      /usr/bin/obclient
      

2. Configure the observer process directory.

   1. (Optional) Clear the directory.

      When you deploy OceanBase Database for the first time on a new server, do not clear the directory.

      If either of the following conditions is met, you can choose to clear the old OceanBase directory:

      • You want to clear the OceanBase environment before the installation.
      • The installation environment is messy or the files of the installation environment will affect the next installation.

      [root@xxx /home/admin]# kill -9 `pidof observer`
      [root@xxx /home/admin]# rm -rf /data/1/$cluster_name 
      [root@xxx /home/admin]# rm -rf /data/log1/$cluster_name 
      [root@xxx /home/admin]# rm -rf /home/admin/oceanbase/store/$cluster_name 
      [root@xxx /home/admin]# rm -rf /home/admin/oceanbase/log/* /home/admin/oceanbase/etc/*config*
      

      Here, $cluster_name is the name of the cluster.

      Here is an example:

      [root@xxx /home/admin]# kill -9 `pidof observer`
      [root@xxx /home/admin]# rm -rf /data/1/obdemo
      [root@xxx /home/admin]# rm -rf /data/log1/obdemo
      [root@xxx /home/admin]# rm -rf /home/admin/oceanbase/store/obdemo
      [root@xxx /home/admin]# rm -rf /home/admin/oceanbase/log/* /home/admin/oceanbase/etc/*config*
      

   2. Initialize the OceanBase directory.

      We recommend that you install OceanBase Database data files on an independent disk and then use a soft link to connect the data directory to the home directory of the software.

      Use the following statement to switch to the admin user:

      [root@xxx /home/admin]# su - admin
      

      Use the admin user to execute the following commands to create the required directories:

      mkdir -p /data/1/$cluster_name/{etc3,sstable,slog}
      

      mkdir -p /data/log1/$cluster_name/{clog,etc2}
      

      mkdir -p /home/admin/oceanbase/store/$cluster_name
      

      Use the admin user to execute the following command to create a soft link:

      for t in {etc3,sstable,slog};do ln -s /data/1/$cluster_name/$t /home/admin/oceanbase/store/$cluster_name/$t; done
      

      for t in {clog,etc2};do ln -s /data/log1/$cluster_name/$t /home/admin/oceanbase/store/$cluster_name/$t; done
      

      Here, $cluster_name is the name of the cluster. The slog and data files must be on the same disk.

      Here is an example:

      [root@xxx /home/admin]# su - admin
      -bash-4.2$ mkdir -p /data/1/obdemo/{etc3,sstable,slog}
      -bash-4.2$ mkdir -p /data/log1/obdemo/{clog,etc2}
      -bash-4.2$ mkdir -p /home/admin/oceanbase/store/obdemo
      -bash-4.2$ for t in {etc3,sstable,slog};do ln -s /data/1/obdemo/$t /home/admin/oceanbase/store/obdemo/$t; done
      -bash-4.2$ for t in {clog,etc2};do ln -s /data/log1/obdemo/$t /home/admin/oceanbase/store/obdemo/$t; done
      

      Note

      The obdemo directory is created based on the cluster name and can be customized. It is used when you start the process.

      Check the results:

      -bash-4.2$ cd /home/admin/oceanbase
      -bash-4.2$ tree store/
      store/
      `-- obdemo
         |-- clog -> /data/log1/obdemo/clog
         |-- etc2 -> /data/log1/obdemo/etc2
         |-- etc3 -> /data/1/obdemo/etc3
         |-- slog -> /data/1/obdemo/slog
         `-- sstable -> /data/1/obdemo/sstable
      
      6 directories, 0 files
      

3. Initialize the OceanBase cluster.

   Note

   The example IP address is obfuscated. This is not a requirement during the deployment. You need to fill in the IP address of your server when you deploy OceanBase Database.

   1. Start the observer process.

      Start the observer process on each node as the admin user.

      Notice

      The parameters of the observer process on each replica are not exactly the same. When you start the observer process, specify the IP address and port number of the two (or more) servers where the Root Service is located. You do not need to specify all servers when you create a cluster. After the cluster is created, you can add new servers.

      cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer {-I $ip | -i $devname} -P $rpc_port -p $sql_port -z $zone_name -d /home/admin/oceanbase/store/$cluster_name -r '$ip:2882:2881' -c $cluster_id -n $cluster_name -o "system_memory=30G,datafile_size=500G,config_additional_dir=/data/1/$cluster_name/etc3;/data/log1/$cluster_name/etc2"
      

      The parameters are described as follows:

      Parameter	Description
      -I | -i	-I: the IP address of the server to be started. In a multi-server deployment, 127.0.0.1 cannot be used as the target IP address. We recommend that you use the specified IP address (-I 10.10.10.1) to start the server. -i: the name of the network card. You can view the available network card names by using the ifconfig command. Note You can also specify the IP address and the network card name (-I 10.10.10.1 -i eth0) to start the server at the same time. However, this practice is not recommended.
      -p	the port number of the service. Generally, the value is 2881.
      -P	the port number of the RPC service. Generally, the value is 2882.
      -n	the name of the cluster. Custom cluster names must not be repeated.
      -z	the zone to which the observer process belongs.
      -d	the directory where the cluster is located. This parameter specifies the directory where the cluster was created. Apart from the cluster name $cluster_name, other parameters must not be changed.
      -c	the ID of the cluster. Cluster IDs must not be repeated. Custom cluster IDs can be used.
      -l	the log level.
      -r	the RS list. The format is $ip:2882:2881; semicolons separate multiple RS entries, indicating the information of Root Services.
      -o	the list of cluster startup parameters (configurations). This parameter is optional. You can specify values for multiple configurations, separating the values with commas. You can choose appropriate startup parameters and specify suitable values for each parameter based on your business needs to optimize the performance and resource utilization of the cluster. Some common cluster startup configurations are described as follows: cpu_count: the total number of system CPU cores. system_memory: the memory size reserved for Tenant ID 500, namely, the memory space internally reserved for OceanBase Database. If the server memory is limited, you can decrease the value of this parameter. Note that during performance tests, the memory might be insufficient. memory_limit: the total size of available memory. datafile_size: the size of disk space available for data files to occupy. In other words, this parameter specifies the initial size of the sstable data file in OceanBase Database. Based on the assessment of the available space in /data/1/, we recommend that you set the parameter to at least 100G. datafile_disk_percentage: the percentage of total disk space occupied by data files. datafile_next: the automatic expansion step length of data files on disk. datafile_maxsize: the maximum size of data files on disk to which the automatic expansion can be expanded. config_additional_dir: the additional local storage directories for storing configuration files to ensure redundancy of configuration files. log_disk_size: the size of the redo log disk. log_disk_percentage: the percentage of total disk space occupied by the redo log. syslog_level: the system log level. syslog_io_bandwidth_limit: the upper limit on the disk I/O bandwidth available for system logs. System logs exceeding the bandwidth upper limit will be discarded. max_syslog_file_count: the maximum number of log files that can be stored before the log file recycling mechanism is triggered. enable_syslog_recycle: specifies whether to enable the log file recycling mechanism. It is used in conjunction with the max_syslog_file_count parameter to determine whether the log file recycling mechanism applies to old log files. The datafile_size, datafile_disk_percentage, datafile_next, and datafile_maxsize parameters are used together to enable automatic expansion of data files on disk. For more information, see Enable dynamic expansion for disk data files. For more information about cluster configurations, see Overview of cluster-level configurations.

      Here is an example:

      Start the observer process on OBServer servers. Take 10.10.10.1 and 10.10.10.2 as an example.

      zone1:

      [root@xxx admin]# su - admin
      -bash-4.2$ cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -I 10.10.10.1 -P 2882 -p 2881 -z zone1 -d /home/admin/oceanbase/store/obdemo -r '10.10.10.1:2882:2881;10.10.10.2:2882:2881' -c 10001 -n obdemo -o "system_memory=30G,datafile_size=500G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2"
      

      zone2:

      [root@xxx admin]# su - admin
      -bash-4.2$ cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -I 10.10.10.2 -P 2882 -p 2881 -z zone2 -d /home/admin/oceanbase/store/obdemo -r '10.10.10.2:2882:2881;10.10.10.2:2882:2881' -c 10001 -n obdemo -o "system_memory=30G,datafile_size=500G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2"
      

      Run the following command to check whether the observer process has been started:

      • The netstat -ntlp command. If the process has listened on the 2881 and 2882 ports, the process has been started.
      • The ps -ef|grep observer command. It will return information about the observer process.

      Here is an example:

      -bash-4.2$ netstat -ntlp
      (Not all processes could be identified, non-owned process info
      will not be shown, you would have to be root to see it all.)
      Active Internet connections (only servers)
      Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
      tcp        0      0 0.0.0.0:2881            0.0.0.0:*               LISTEN      11112/observer
      tcp        0      0 0.0.0.0:2882            0.0.0.0:*               LISTEN      11112/observer
      ...        ...    ...                       ...                     ...         ...
      
      -bash-4.2$ ps -ef|grep observer
      admin     11112      0 40 16:18 ?        00:00:17 /home/admin/oceanbase/bin/observer -I 10.10.10.1 -P 2882 -p 2881 -z zone1 -d /home/admin/oceanbase/store/obdemo -r 10.10.10.1:2882:2881;10.10.10.2:2882:2881 -c 10001 -n obdemo -o system_memory=30G,datafile_size=500G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2
      

   2. Perform the cluster bootstrap operation.

      Use OBClient to connect to any server as the root user, and leave the password field empty.

      [root@xxx admin]# obclient -h127.0.0.1 -uroot -P2881 -p******
      
      obclient [(none)]> SET SESSION ob_query_timeout=1000000000;
      Query OK, 0 rows affected
      
      obclient [(none)]> ALTER SYSTEM BOOTSTRAP ZONE 'zone1' SERVER '10.10.10.1:2882',ZONE 'zone2' SERVER '10.10.10.2:2882';
      Query OK, 0 rows affected
      

      Notice

      If an error occurs at this step, check whether the parameters of the observer process are correctly specified, whether the permissions of the observer-related directories are correct, whether the log directory has sufficient space, whether the server clock is synchronized, and whether the server has sufficient memory resources. If any of these checks identify a problem, resolve the problem and then clear the OceanBase directory and deploy OceanBase Database from the very beginning.

   3. Verify whether the cluster has been initialized.

      After the cluster is initialized, execute the SHOW DATABASES; command to check whether the oceanbase database is listed in the query result. If yes, the cluster has been initialized.

   4. Change the password.

      The password of the root user in the sys tenant is empty by default. After the initialization, you need to change the password.

      ALTER USER root IDENTIFIED BY '******';
      

Step 2: Deploy the arbitration service

1. Install the OceanBase RPM package on the arbitration server.

   Switch to the directory where the OceanBase RPM package is stored:

   [root@xxx /]# cd $rpm_dir
   

   Install the OceanBase RPM package:

   [root@xxx $rpm_dir]# rpm -ivh $rpm_name [--prefix=/home/admin/oceanbase]
   

   Here, $rpm_dir represents the directory where the RPM package is stored, and $rpm_name represents the name of the RPM package.

   Note

   By default, OceanBase Database will be installed in the /home/admin/oceanbase directory.

   Here is an example:

   [root@xxx /home/admin/rpm]# rpm -ivh oceanbase-4.2.0.0-100000052023073123.el7.x86_64.rpm
   Preparing...                          ################################# [100%]
   Updating / installing...
      1:oceanbase-4.2.0.0-100000052023073################################# [100%]
   

2. Configure the directory for the arbitration server process.

   Create a clog directory for the arbitration server process on the arbitration server.

   Note

   • When you start the arbitration server process, make sure that the clog directory has been created in the store directory specified with the -d option.
   • Do not use a disk with full space to store the data of the store directory, or the arbitration server's functionality may be abnormal.

   Switch to the admin user:

   [root@xxx /home/admin]# su - admin
   

   As the admin user, run the following commands to create the required directories:

   mkdir -p /home/admin/oceanbase/store/clog
   

3. Start the arbitration server process.

   Start the arbitration server process in arbitration mode as the admin user on the arbitration server. The necessary parameters are specified in the startup statement. Here is an example:

   cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -m arbitration -P $rpc_port -d $obdir/store [{-I $ip | -i $devname}] [-l $log_level]  [-o "parameters_list"]
   

   The parameters are described as follows:

   Parameter	Description
   -m	Specifies the arbitration mode. The value is arbitration.
   -P	Specifies the RPC port number of the arbitration server process. The default value is 2882.
   -d	Specifies the storage directory path. It is usually /home/admin/oceanbase/store. Make sure that the clog subdirectory has been created in this directory.
   -I | -i	Optional. -I: Specifies the IP address of the node to be started. -i: Specifies the network card name. You can use the ifconfig command to view the network card names. Note You can start a node by specifying both the IP address and network card name (for example, -I 10.10.10.1 -i eth0). However, we recommend that you do not specify both parameters.
   -l	Specifies the log level. The default value is WDIAG. The valid values are: {DEBUG, TRACE, WDIAG, EDIAG, INFO, WARN, ERROR}. For more information, see Log levels.
   -o	Specifies a configuration file. Multiple configuration files can be specified, separated by commas. The following configuration files are commonly used when you start the arbitration server process: cpu_count system_memory memory_limit __easy_memory_limit log_disk_size log_disk_percentage syslog_level syslog_io_bandwidth_limit max_syslog_file_count enable_syslog_recycle Note To ensure that the log directory of the arbitration node can be automatically cleared and prevent the disk from being filled up, set enable_syslog_recycle to True and set max_syslog_file_count to an effective value. Otherwise, log files will grow continuously and may occupy all disk space.

   Here is an example:

   Start the arbitration server process on the arbitration server. For example, if the IP address of the arbitration server is 10.10.10.3, you can run the following command to start the arbitration server process.

   -bash-4.2$ cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -m arbitration -P 2882 -d /home/admin/oceanbase/store -I 10.10.10.3 -o "enable_syslog_recycle=True,max_syslog_file_count=100"
   

   You can use the following command to check whether the observer process has been started:

   • Run the netstat -ntlp command. If port 2882 is listened to, the process has been started.
   • Run the ps -ef|grep observer command to return the observer process information.

   Here is an example:

   -bash-4.2$ netstat -ntlp
   (Not all processes could be identified, non-owned process info
   will not be shown, you would have to be root to see it all.)
   Active Internet connections (only servers)
   Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
   tcp        0      0 0.0.0.0:2882            0.0.0.0:*               LISTEN      18231/observer
   ...
   
   -bash-4.2$ ps -ef|grep observer
   admin     18231      0  3 10:33 ?        00:00:00 /home/admin/oceanbase/bin/observer -m arbitration -P 2882 -d /home/admin/oceanbase/store -I 10.10.10.3
   ...
   

Step 3: Add an arbitration service to the cluster

Run the following command to add an arbitration service to the cluster:

ALTER SYSTEM ADD ARBITRATION SERVICE "$arb_server_ip:$arb_server_port";

Here is an example:

obclient [(none)]> ALTER SYSTEM ADD ARBITRATION SERVICE "10.10.10.3:2882";
Query OK, 0 rows affected

obclient [(none)]> SELECT * FROM oceanbase.DBA_OB_ARBITRATION_SERVICE;
+---------------------+---------------------+-------------------------+---------------------+------------------------------+------+
| CREATE_TIME         | MODIFY_TIME         | ARBITRATION_SERVICE_KEY | ARBITRATION_SERVICE | PREVIOUS_ARBITRATION_SERVICE | TYPE |
+---------------------+---------------------+-------------------------+---------------------+------------------------------+------+
| 2023-03-06 17:29:36 | 2023-03-06 17:29:36 | default                 | 10.10.10.3:2882     | NULL                         | ADDR |
+---------------------+---------------------+-------------------------+---------------------+------------------------------+------+
1 row in set

Step 4: Enable the arbitration service for the sys tenant

You can enable the arbitration service for a tenant when you create the tenant. For more information, see CREATE TENANT.

If you do not enable the arbitration service when you create a tenant, you can enable it for the tenant after the tenant is created.

Run the following statement to enable the arbitration service for an existing tenant.

ALTER TENANT $tenant_name [SET] enable_arbitration_service = true;

Here is an example:

Enable the arbitration service for the sys tenant of a cluster.

obclient [(none)]> ALTER TENANT sys SET enable_arbitration_service = true;
Query OK, 0 rows affected

Step 5: Query the status of the arbitration service for a tenant

1. Check whether the arbitration service is enabled for the tenant.

   You can query the DBA_OB_TENANTS view for tenant information and check the value of the arbitration_service_status column for the status of the arbitration service.

   The possible values of arbitration_service_status are as follows:

   • ENABLED: the arbitration service is enabled for the tenant.
   • ENABLING: the arbitration service is being enabled for the tenant.
   • DISABLED: the arbitration service is disabled for the tenant.
   • DISABLING: the arbitration service is being disabled for the tenant.

   Here is an example:

   obclient [(none)]> SELECT TENANT_ID,TENANT_NAME,ARBITRATION_SERVICE_STATUS FROM oceanbase.DBA_OB_TENANTS WHERE tenant_name = 'sys';
   +-----------+-------------+----------------------------+
   | TENANT_ID | TENANT_NAME | ARBITRATION_SERVICE_STATUS |
   +-----------+-------------+----------------------------+
   |         1 | sys         | ENABLED                    |
   +-----------+-------------+----------------------------+
   1 row in set
   

2. Check whether the arbitration service is available for the tenant.

   If the arbitration service is enabled for a tenant, the arbitration service is available for the log streams created at the time when it is enabled. However, for log streams created after it is enabled, there is no guarantee that the arbitration service is available, because creating log streams is executed in non-strict mode and does not guarantee the creation of the arbitration service. In this case, you can run the following SQL statement to check whether the arbitration service is available for all log streams of the tenant.

   ##Query the list of log streams without an arbitration replica in the tenant##
   (SELECT distinct ls_id FROM GV$OB_LOG_STAT WHERE tenant_id = xxx) except
   (SELECT ls_id FROM GV$OB_LOG_STAT WHERE tenant_id = xxx and role = 'LEADER' and arbitration_member = "$arb_server_ip:$arb_server_port");
   

   Here is an example:

   Query the list of log streams without an arbitration replica in the sys tenant. If the query result set is empty, the arbitration service is available for the current tenant.

   obclient [oceanbase]> (SELECT distinct ls_id FROM GV$OB_LOG_STAT WHERE tenant_id = 1) except
   (SELECT ls_id FROM GV$OB_LOG_STAT WHERE tenant_id = 1 and role = 'LEADER' and arbitration_member = "10.10.10.3:2882");
   Empty set
   

Next steps

After the cluster is created, you can create user tenants as needed.

For more information about how to create a user tenant using the CLI, see Create a tenant.

References

• Connect to an OceanBase tenant by using OBClient
• Overview of system configuration items
• ALTER USER
• ALTER TENANT
• Introduction to user tenants
• CREATE TENANT
• Enable the arbitration service
• Disable the arbitration service
• Modify the degradation timeout period
• Replace the arbitration service
• Remove the arbitration service