Experience horizontal scaling and elastic scaling

Elastic scaling is a key capability for OceanBase to evolve from a small-scale experience environment to a larger business capacity. As the business grows or declines, and due to peak differences at different times, the database needs to provide elastic service capabilities. OceanBase Database supports elastic service capabilities through its tenant scaling feature: scaling up can increase computing power and storage capacity, which can be achieved either by enhancing the service capability of a single node or by adding service nodes; scaling down is the reverse operation of scaling up. All adjustments take effect online and are transparent to the business.

This topic covers the following experiences:

• Understand the scaling framework: Learn about the three scaling levels: tenant vertical scaling (adjusting resource specifications), tenant horizontal scaling (adjusting the unit number), and cluster scaling (adding nodes).
• Experience tenant vertical scaling: Increase the resource specification of a tenant online and observe how quickly the change takes effect.
• Compare business capacity before and after scaling: Run stress tests with the same parameters on the same tenant before and after scaling, and observe the changes in QPS and latency.
• Learn how to verify scaling: Confirm resource specifications, unit distribution, and task status through system views.

Prerequisites

Before you start this procedure, ensure that the following conditions are met:

Introduction to the scaling system

Tenant scaling, whether up or down, essentially increases or decreases the tenant's service capabilities, including computing power and storage capacity. OceanBase Database provides two methods for tenant scaling, which, together with node expansion at the cluster level, constitute a complete elastic scaling system:

Tenant vertical scaling can be performed in two ways:

• Modify the resource configuration: Directly adjust the CPU or memory values of the unit config, which affects all resource units that use this specification.
• Switch the resource configuration: Point the resource pool to another unit config (ALTER RESOURCE POOL pool1 UNIT='uc2';). This is useful when multiple tenants share the same unit config and you want to adjust only one of them.

Experience tenant vertical scaling

This section creates a small tenant scale_tenant (1C2G) and runs a round of point query stress tests as the baseline. It then scales the specification online to 2C4G and runs the same stress test again, so that you can compare the business capacity before and after scaling.

Procedure

1. Log in to the sys tenant of the cluster as the root user (the password is randomly generated during deployment; obtain it as described in the prerequisites above), and create the experience tenant:

   obclient [oceanbase]> CREATE RESOURCE UNIT scale_unit MAX_CPU=1, MIN_CPU=1, MEMORY_SIZE='2G';
   obclient [oceanbase]> CREATE RESOURCE POOL scale_pool UNIT='scale_unit', UNIT_NUM=1, ZONE_LIST=('zone1');
   obclient [oceanbase]> CREATE TENANT IF NOT EXISTS scale_tenant CHARSET='utf8mb4', PRIMARY_ZONE='zone1', RESOURCE_POOL_LIST=('scale_pool') SET ob_tcp_invited_nodes='%';
   

   The CREATE TENANT operation is synchronous. The command returns after about 10 seconds, so be patient while it runs.

2. Run the baseline stress test before scaling. Use the point query script to run a stress test against scale_tenant, and note down the QPS and the 95th percentile latency from the results:

   obd test sysbench demo --tenant=scale_tenant --script-name=oltp_point_select.lua --table-size=100000 --threads=16 --rand-type=uniform
   

   Point queries are a typical CPU-intensive workload. If CPU resources in the cluster are tight, the 1C CPU quota becomes the throughput bottleneck. If CPU resources are abundant, the tenant may actually use more CPU than its quota, and the QPS difference between the two rounds may not be significant (see Interpretation below for details).

3. Check the current resource specification to confirm the pre-scaling baseline:

   obclient [oceanbase]> SELECT UNIT_CONFIG_ID, NAME, MAX_CPU, MIN_CPU, MEMORY_SIZE, LOG_DISK_SIZE
                       FROM oceanbase.DBA_OB_UNIT_CONFIGS WHERE NAME='scale_unit';
   

   A sample return result is as follows (UNIT_CONFIG_ID is assigned by the system and varies by environment):

   +----------------+------------+---------+---------+-------------+---------------+
   | UNIT_CONFIG_ID | NAME       | MAX_CPU | MIN_CPU | MEMORY_SIZE | LOG_DISK_SIZE |
   +----------------+------------+---------+---------+-------------+---------------+
   |           1009 | scale_unit |       1 |       1 |  2147483648 |    6442450944 |
   +----------------+------------+---------+---------+-------------+---------------+
   1 row in set
   

   The result shows that the scale_unit resource unit provides 1C of CPU and 2G of memory, and that its log disk space is 6G (three times the memory size by default).

4. Scale the tenant specification up to 2C4G online:

   obclient [oceanbase]> ALTER RESOURCE UNIT scale_unit MAX_CPU=2, MIN_CPU=2, MEMORY_SIZE='4G';
   Query OK, 0 rows affected
   

   If the statement returns within seconds, the scale-out is complete. The tenant's CPU and memory take effect immediately, without data migration or switchover, and the sessions used by stress testing or business operations are not affected. (In this example, this statement and the subsequent scale-down statement were both executed during the stress test, without any errors or interruptions.) The ALTER statement adjusts only the options that you explicitly specify, so LOG_DISK_SIZE remains unchanged. If you also need to adjust the log disk specification, specify it in the statement.

5. Confirm that the resource specifications have changed:

   obclient [oceanbase]> SELECT UNIT_CONFIG_ID, NAME, MAX_CPU, MIN_CPU, MEMORY_SIZE, LOG_DISK_SIZE
                       FROM oceanbase.DBA_OB_UNIT_CONFIGS WHERE NAME='scale_unit';
   

   In the result, MAX_CPU and MIN_CPU have changed to 2, and MEMORY_SIZE has changed to 4294967296 (that is, 4G):

   +----------------+------------+---------+---------+-------------+---------------+
   | UNIT_CONFIG_ID | NAME       | MAX_CPU | MIN_CPU | MEMORY_SIZE | LOG_DISK_SIZE |
   +----------------+------------+---------+---------+-------------+---------------+
   |           1009 | scale_unit |       2 |       2 |  4294967296 |    6442450944 |
   +----------------+------------+---------+---------+-------------+---------------+
   1 row in set
   

   You can also view the actual effective values of resource units on a node through the GV$OB_UNITS view:

   obclient [oceanbase]> SELECT TENANT_ID, SVR_IP, ZONE, MAX_CPU, MIN_CPU, MEMORY_SIZE/1024/1024/1024 AS MEMORY_SIZE_G
                       FROM oceanbase.GV$OB_UNITS
                       WHERE TENANT_ID=(SELECT TENANT_ID FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_NAME='scale_tenant');
   

   +-----------+--------------+-------+---------+---------+----------------+
   | TENANT_ID | SVR_IP       | ZONE  | MAX_CPU | MIN_CPU | MEMORY_SIZE_G  |
   +-----------+--------------+-------+---------+---------+----------------+
   |      1020 | 127.0.0.1    | zone1 |       2 |       2 | 3.000000000000 |
   +-----------+--------------+-------+---------+---------+----------------+
   1 row in set
   

6. Run the same parameter stress test again and compare with the baseline result:

   obd test sysbench demo --tenant=scale_tenant --script-name=oltp_point_select.lua --table-size=100000 --threads=16 --rand-type=uniform
   

7. Compare the results of the two rounds and focus on the following:

   • Both rounds end normally without errors: This shows that the tenant can handle business traffic after scaling, and that the specification change is imperceptible to the business.
   • The QPS difference may not be significant: MAX_CPU is an upper limit rather than a guaranteed value. When the cluster has spare CPU resources, a tenant may actually use more CPU than its quota, so it is normal for the measured QPS to be similar before and after scaling up. The benefit of a higher specification mainly shows when cluster resources are tight and tenants compete for CPU.
   • The 95th percentile latency shows only slight fluctuations within the same order of magnitude: When CPU resources are abundant, the request queue time is already very short, and it remains in the sub-millisecond range before and after scaling up.

Interpretation

Experience tenant scale-down

Downsizing is the reverse of upsizing: it reduces the resource specification back to its original value to release resources:

obclient [oceanbase]> ALTER RESOURCE UNIT scale_unit MAX_CPU=1, MIN_CPU=1, MEMORY_SIZE='2G';

The statement also returns within seconds. In this example, the specification was scaled down from 2C4G to 1C2G during the stress test. The change completed instantly, with no errors and no reconnections during the whole stress test, and the business was not affected.

When you reduce the specification, MEMORY_SIZE is subject to one limitation: on every physical server that hosts a unit of the tenant, the configured MEMORY_SIZE must be greater than or equal to the tenant's current actual memory usage (MEM_USED) on that server. If the new MEMORY_SIZE is less than MEM_USED, the adjustment fails, and you must trigger a minor compaction to release memory before you try again. Therefore, we recommend that you trigger a minor compaction manually before you scale down a tenant.

Experience tenant horizontal scaling

When the resource specification of a single node reaches its upper limit, you can increase the total service capacity of a tenant by adding service nodes and increasing the unit number. The total service capacity of a tenant is roughly equal to the specification of a single unit multiplied by the unit number.

Starting from V4.0.0, OceanBase Database requires that the number of units in each zone of a tenant be the same. To manage the units in each zone in a unified way, the system introduces the unit group mechanism: units with the same ID (UNIT_GROUP_ID) in different zones belong to the same unit group. Increasing or decreasing the unit number essentially means creating or deleting units.

Procedure (multi-node cluster)

The following example shows how to adjust the unit number of tenant mysql001 from 1 to 2:

1. Check the current unit number of the tenant:

   obclient [oceanbase]> SELECT TENANT_ID, TENANT_NAME, UNIT_NUM FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_NAME='mysql001';
   

   In the returned result, if the value of UNIT_NUM is 1, it means the tenant has 1 unit.

2. Adjust the unit number to 2:

   obclient [oceanbase]> ALTER RESOURCE TENANT mysql001 UNIT_NUM = 2;
   

   After the statement is executed, the system adds a new unit to each zone. This new unit forms a new unit group with the original unit, and data is redistributed to the new units through load balancing.

3. Check the execution status of the task:

   obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_TENANT_JOBS WHERE JOB_TYPE='ALTER_RESOURCE_TENANT_UNIT_NUM' AND TENANT_ID=1006;
   

   When the JOB_STATUS of the corresponding record in the returned result is SUCCESS, the adjustment is successful. You can locate this change record based on the fields START_TIME (the time when the task was initiated), SQL_TEXT (the SQL statement corresponding to the task), and EXTRA_INFO (the number of units before and after the change). In the example, the values of TENANT_ID and UNIT_GROUP_ID should be based on the query results in your actual environment.

4. Check the unit distribution of the tenant to confirm that the number of units in each zone has become 2:

   obclient [oceanbase]> SELECT UNIT_ID, ZONE, SVR_IP, UNIT_GROUP_ID, STATUS FROM oceanbase.DBA_OB_UNITS WHERE TENANT_ID=1006;
   

To decrease the unit number, use the ALTER RESOURCE TENANT statement as well. The system randomly selects a unit group, redistributes its data to the other unit groups, and then deletes it. You can also specify the unit group to delete with DELETE UNIT_GROUP:

-- Randomly delete one unit group and adjust the unit number to 1.
obclient [oceanbase]> ALTER RESOURCE TENANT mysql001 UNIT_NUM = 1;

-- Specify to delete the unit group with UNIT_GROUP_ID 1006.
obclient [oceanbase]> ALTER RESOURCE TENANT mysql001 UNIT_NUM = 1 DELETE UNIT_GROUP =(1006);

Cluster expansion

When the overall storage space or computing capability of a cluster is insufficient, you can add an OBServer node to the cluster. For example, to add a node in zone1, execute the following command in the sys tenant:

obclient [oceanbase]> ALTER SYSTEM ADD SERVER 'server_ip:port_num' ZONE 'zone1';

Here, server_ip is the IP address of the node to be added, and port_num is the RPC port of the node (default is 2882). After the new node is added, Root Service gradually balances the existing data to it, which horizontally scales storage space and computing power. The whole process is transparent to business operations.

For clusters managed by OBD, you can also use the obd cluster scale_out command to scale out: create a configuration file that contains only the information of the new node and run this command. OBD then deploys and registers the new node automatically.

Summary of scaling verification methods

Regardless of the scaling method used, you can verify the change results through the following system views:

Next steps

After you complete the experiences in this topic, you can explore the following:

1. Experience parallel import and high compression: Use the parallel import capability to complete large-scale data ingestion, and compare storage usage before and after compression. For more information, see Experience parallel import and high compression.
2. Experience data protection and recovery: Experience data recovery capabilities for misoperations through flashback queries. For more information, see Experience data protection and recovery.
3. Deepen understanding of multi-tenancy and resource isolation: Understand the resource management and isolation mechanisms under the multi-tenant architecture. For more information, see Experience multi-tenancy and resource isolation.

References

• For an introduction to the two approaches to tenant scaling, see Introduction to tenant scaling.
• For the complete procedure to achieve tenant scaling by adjusting resource specifications, see Achieve tenant scaling by adjusting resource specifications.
• For the complete procedure to achieve tenant scaling by adjusting the unit number, see Achieve tenant scaling by adjusting the unit number.
• For conceptual explanations of unit config and resource pool, see Tenant resource management.
• For the method to manually add nodes to a cluster, see Add a node.
• For the operation to expand a cluster using OBD (obd cluster scale_out), see Expansion and component changes.
• For detailed column descriptions of the views, see GV$OB_UNITS, GV$OB_SERVERS and DBA_OB_TENANT_JOBS.
• For detailed parameters of OBD test commands, see Test commands.