Performance tuning methods
Ideally, the performance of backup and restore is limited only by the total amount of data and hardware capabilities (CPU, disk, and network). However, under the default configuration, backup and restore may not fully utilize hardware resources. To address this, OceanBase AI Database provides resource isolation strategies and the following configuration items to optimize the performance of backup and restore:
Network configuration: The
sys_bkgd_net_percentageconfiguration item limits the percentage of total network bandwidth that can be utilized by background system tasks (including backup and restore tasks). Setting an appropriate value forsys_bkgd_net_percentageallows backup and restore tasks to make full use of network bandwidth without affecting foreground business operations.CPU and I/O configuration: OceanBase AI Database also provides the Resource Manager resource isolation mechanism, allowing you to limit the CPU and I/O usage of different types of tasks. If resource isolation is configured for backup and restore tasks, set the limits based on actual resources and business needs to avoid bottlenecks in CPU and I/O usage.
Other configurations: When sufficient CPU, I/O, and network resources are available, you can further improve the performance of backup and restore tasks by increasing their concurrency through related configuration items (such as
ha_low_thread_score,log_archive_concurrency,log_restore_concurrency, andha_high_thread_score).
Resource configuration
Cluster-level configuration item sys_bkgd_net_percentage
The sys_bkgd_net_percentage configuration item is used to set the percentage of network bandwidth that can be occupied by background system tasks. The default value is 60% of the machine's NIC speed. When the bandwidth is fully utilized, you can appropriately increase the value of sys_bkgd_net_percentage without affecting business requests.
After the configuration item is successfully set, you can view the log as follows:
Log in to the server where the OBServer node resides as the
adminuser.Go to the installation directory of OceanBase AI Database.
Here is an example of the installation path of OceanBase AI Database:
/home/admin/oceanbase/. In practice, follow the actual environment.[admin@xxx /]$ cd /home/admin/oceanbaseRun the following command to view the NIC speed.
[admin@xxx oceanbase]$ grep -E 'print band limit|succeed to init_bandwidth_throttle' log/observer.log*Here,
observer.logis the observer log when the cluster starts.For example, the query result is as follows:
log/observer.log.20210811100806:[2021-08-11 10:06:32.934433] INFO [SERVER] ob_server.cpp:1783 [76957] [0] [Y0-0000000000000000] [lt=4] [dc=0] succeed to init_bandwidth_throttle(sys_bkgd_net_percentage_=60,ethernet_speed_=1310720000,rate=786432000) log/observer.log.20210811100806:[2021-08-1110:07:42.351813] INFO [COMMON] utility.cpp:1487 [77169][418] [Y9FA64586E9E-0005C93F15DAE715] [lt=11] [dc=0] print band limit(comment= in , copy_KB=0, sleep_ms_sum=0, speed_KB_per_s=0, total_sleep_ms=0,total__bytes=531, rate_KB/s=786432,print_interval_ms=69417)In the first query result,
sys_bkgd_net_percentage_=60indicates that the network bandwidth for background system tasks is 60% of the NIC speed of the machine.network_speed=1310720000indicates that the maximum NIC speed identified by OceanBase AI Database is 1310720000 B/s.rate=786432000indicates that the limited maximum speed is 786432000 B/s, andrate = network_speed * sys_bkgd_net_percentage.In the second query result,
rate_KB/s=786432indicates that the limited maximum speed (rate) identified by OceanBase AI Database is 786432 KB/s.
The NIC speed identified by OceanBase AI Database may be inaccurate. After you view the log and find that the speed identified by OceanBase AI Database does not match the actual speed, you can modify the speed by referring to Check the NIC speed.
After the modification, you can query the V$OB_NIC_INFO view to confirm whether the NIC speed is updated.
Resource isolation of Resource Manager
Resource Manager is the resource isolation mechanism of OceanBase AI Database. Through function-level resource isolation, you can configure resource limits for different background tasks, including CPU, IOPS, and network bandwidth. For more information about resource isolation, see Resource isolation overview.
In function-level resource isolation, the background tasks related to backup and restore are as follows:
- ha_high: High-priority, high-reliability tasks such as replication, rebuild, and restore.
- ha_mid: Medium-priority, high-reliability tasks, such as migration tasks.
- ha_low: Low-priority, high-reliability tasks such as backup and backup cleanup.
You can use the DBA_OB_RSRC_DIRECTIVES view to check the resource isolation plan configured for the current tenant. If the query result is empty, it indicates that no resource isolation plan is configured for the tenant. If the query result contains records of background tasks, first check whether CPU, I/O, network bandwidth, and other resources have reached the bottleneck and whether they comply with the resource isolation limits. If so, we recommend that you modify the resource isolation plan without affecting the foreground business. For more information about how to modify the resource isolation plan, see Update a plan directive.
We recommend that you do not configure a resource isolation plan when performing performance tests for backup and restore.
Data backup
In OceanBase AI Database, data backup is performed on macroblocks. Macroblocks are independent of each other. Theoretically, the performance of the data backup phase is linearly proportional to the backup concurrency.
Configuration item |
Description |
Default value |
Configuration instructions |
|---|---|---|---|
| ha_low_thread_score | A tenant-level configuration item that controls the concurrency of data backup. | 0, which indicates that the default concurrency is 2. |
For small tenants (with up to 4 CPU cores), we recommend that you set this configuration item to the default value. For large tenants, you can set it to 10 first. If the backup speed is too slow, you can double the value as needed. When you perform performance tests for backup and restore, we recommend that you set this configuration item to its maximum value of 100. |
Log archiving
Configuration item |
Description |
Default value |
Configuration instructions |
|---|---|---|---|
| log_archive_concurrency | A tenant-level configuration item that controls the concurrency of log archiving. | 0, which indicates that the number of archive worker threads is calculated based on the tenant's MAX_CPU according to the following adaptive rules:
|
We recommend that you keep the default value for both small and large tenants and let the system calculate the number of worker threads based on the adaptive rules. |
Physical restore
In OceanBase AI Database, data restore consists of scattered macroblock tasks that are independent of log streams and tablets. Theoretically, the performance of data restore is linearly proportional to the restore concurrency. In addition, tablet creation has been changed to multi-threaded execution, which significantly improves restore performance.
Configuration item |
Description |
Default value |
Configuration instructions |
|---|---|---|---|
| log_restore_concurrency | A tenant-level configuration item that controls the concurrency of log restore. | 0, which indicates that the number of cores specified by the tenant's MAX_CPU is used as the concurrency. |
Increasing the value of this configuration item not only adds worker threads but also increases memory resource consumption. We recommend that you set it to the default value of 0. If the restore speed is too slow, appropriately increase the value based on the actual resources of your machine. |
| ha_high_thread_score | A tenant-level configuration item that controls the concurrency of data restore. | 0, which indicates that the default concurrency is 8. |
We recommend that you use the default value in non-performance test scenarios. When you perform performance tests for backup and restore, we recommend that you set this configuration item to its maximum value of 100. |
| _restore_idle_time | A cluster-level hidden configuration item that controls the scheduling interval of RS restore. | 1m, which indicates 1 minute. |
If you set it to 10s, the data restore time can be reduced by several dozen seconds to two minutes. We recommend that you set this configuration item for small tenants with high performance requirements (tenant CPU ≤ 4C). Otherwise, the effect is not significant. |