Load balancing is the foundation for horizontal scaling in LogService. When a LogService cluster grows large, load balancing distributes log stream replicas evenly across cluster nodes to prevent any single node from becoming overloaded, thereby supporting a larger write workload.
Log stream load balancing is managed by LogServiceManager (LM). LM continuously monitors the replica distribution and load status of all LogService nodes (LNs) in the cluster and automatically generates replica migration tasks to maintain a balanced state.
Balanced state
The definition of a balanced state depends on the load balancing strategy. This section describes the balanced state under the default strategy replica_count_based. For the scheduling objectives of the load_based strategy, see the "Load-based balancing" section below.
Load balancing operates at the granularity of log stream replicas and only considers log streams in the NORMAL state and nodes in the Active state. A balanced state is defined as follows: when there are N log streams and M nodes in an Zone, the number of replicas on each node in that Zone is within the range {⌊N/M⌋, ⌊N/M⌋+1}.
Operations that may disrupt the balanced state include:
- Log stream creation: When a log stream is created, LM preferentially selects the node with the fewest replicas for the initial member list.
- Log stream deletion: After a log stream is deleted, the number of replicas on a node is allowed to fall below ⌊N/M⌋ to avoid frequent triggering of balancing tasks.
- Node addition: If a new node has no log stream replicas, LM automatically generates a balancing task to migrate some replicas to the new node.
- Node deletion or permanent offline: The disaster recovery module is responsible for migrating log stream replicas from the deleted node to active nodes, considering the balance after migration.
Balancing mechanism
This section describes the balancing task generation mechanism under the replica_count_based strategy.
The LM periodically generates balancing tasks, which are then executed by the disaster recovery module for actual replica migration. Balancing tasks are mutually exclusive with disaster recovery tasks: the system does not generate new balancing tasks when the internal table t_stream_replica_task contains records.
When generating a balancing task, nodes are classified based on the number of log stream replicas they host:
- Over-utilized: The node's resource usage is significantly higher than the cluster average, indicating that the node is overloaded.
- Above-average: The node's resource usage is higher than the cluster average but still within a controllable range.
- Below-average: The node's resource usage is lower than the cluster average, indicating some resource redundancy.
- Under-utilized: The node's resource usage is significantly lower than the cluster average, with a lot of idle resources.
Migration tasks are generated in the following priority order:
- Over-utilized → under-utilized
- Over-utilized → below-average
- Above-average → under-utilized
Only replicas in the NORMAL state of a log stream can be migrated.
Balancing strategies
LogService supports two balancing strategies, which are controlled by the cluster parameter load_balance_policy.
Replica count-based balancing (replica_count_based)
This is the default strategy. LM uses the number of log stream replicas as the balancing metric and strives to distribute replicas evenly across LNs. This strategy does not depend on the node I/O capacity configuration, which makes it simple and efficient.
Load-based balancing (load_based)
The load balancer uses the I/O resource utilization of nodes (IOPS and bandwidth) as the scheduling metrics to prevent the node resource utilization from exceeding the max_utilization_threshold threshold. It does not require the resource utilization to be balanced across all nodes. The load balancing module schedules based on the node load and log stream write pressure, following these rules:
- Prioritize migrating the write traffic of overloaded nodes to other members in the member list.
- Migrate log stream replicas from an LN with high resource usage to a node that can accommodate the replicas, ensuring that the resource utilization of the destination node does not exceed the threshold after the replicas are added.
- Prioritize migrating out log stream replicas that consume high I/O resources.
- Prioritize migrating replicas to nodes with low resource utilization.
This strategy requires that the disk_max_iops and disk_max_bandwidth parameters are correctly configured (non-zero) for all nodes in the cluster.
Admission check: When the strategy is set to load_based, the following conditions must be met. Otherwise, the modification is rejected:
- The
disk_max_iopsparameter is greater than 0 for all nodes in the cluster. - The
disk_max_bandwidthparameter is greater than 0 for all nodes in the cluster.
If the preceding prerequisites are not met at runtime when the strategy is load_based, the load balancer automatically downgrades according to the following rule: if any node has a disk_max_iops or disk_max_bandwidth value of 0, the strategy is downgraded to replica_count_based and an ERROR log is recorded. During the downgrade, the displayed configuration value remains load_based. Once the conditions are met, the strategy automatically reverts to load_based without manual intervention.
View the balancing status
You can use the show v-table command of the LS-CTRL tool to view the load information of the cluster. The following three virtual views are supported.
node_load: Displays the IOPS and bandwidth usage of each LN.az_load: Displays the IOPS and bandwidth usage of each AZ.stream_replica_io: Displays the I/O statistics of each log stream replica.
ls_ctrl -H <HOST_ADDRESS> show v-table node_load
ls_ctrl -H <HOST_ADDRESS> show v-table az_load
ls_ctrl -H <HOST_ADDRESS> show v-table stream_replica_io
Note
In the preceding commands, HOST_ADDRESS is the HTTP address (including the port number) of any LogService node, for example, 127.0.0.1:8080. For more information about LS-CTRL commands, see Overview of LS-CTRL.
Manually migrate replicas
You can use the ls_ctrl migrate replica command to migrate a specified log stream replica from a source LN to a target LN. For more information about the command, see migrate replica.
You are advised to disable automatic load balancing by using the enable_rebalance parameter before you manually migrate replicas, to avoid interference between manual migration and automatic balancing tasks. Re-enable automatic load balancing after the migration is complete. For more information about configuring enable_rebalance, see Configure the load balancing strategy.
Related parameters
The following parameters are directly related to load balancing. For more information, see Overview of Log Service parameters.
Parameter |
Scope |
Effective mode |
Default value |
Description |
|---|---|---|---|---|
enable_rebalance |
cluster | dynamic | true | Whether to enable log stream load balancing |
load_balance_policy |
cluster | dynamic | replica_count_based | The load balancing strategy. Valid values: replica_count_based (based on the number of replicas) and load_based (based on load) |
rebalance_interval_sec |
cluster | dynamic | 60 | The interval for triggering load balancing, in seconds |
max_utilization_threshold |
cluster | dynamic | 60 | The upper limit of node resource utilization, in percentage. When the utilization exceeds this limit, replica migration out is triggered |
min_utilization_threshold |
cluster | dynamic | 30 | The lower limit of node resource utilization, in percentage. When the utilization is below this limit, only UnderUtilized events are recorded and no action is triggered |
Considerations
- Load balancing tasks and disaster recovery tasks are mutually exclusive. When a migration task is in progress, the system does not generate new balancing tasks.
- Node load is also considered when selecting destinations for log stream creation and disaster recovery tasks, and nodes with lower load are preferentially selected.
- Currently, the load-based balancing strategy considers only the I/O load (IOPS and bandwidth) and does not balance the CPU or memory usage.