Note
This topic applies to obcdc versions later than V3.2.4.5 and earlier than V4.0, as well as versions later than V4.2.1.
The throttling capability of obcdc can, in most cases, keep the memory usage of obcdc during operation within your expectations. You can use the memory_limit parameter to adjust the expected memory consumption of obcdc.
obcdc has multiple internal modules, between which transaction data flows. You can understand the overall production and consumption model as multiple interconnected producer-consumer models. Throttling involves memory usage control for the following two modules: the data consumption module and the log pulling module.
Throttling for the data consumption module
If data consumption is slow, obcdc decreases the task distribution speed of the data consumption module to prevent excessive memory usage caused by a large amount of unconsumed data. obcdc determines the downstream consumption speed based on the task backlog of the consumption queue and the resource recycling queue. If the backlog ratio exceeds pause_redo_dispatch_task_count_threshold/100 and the memory usage reaches the alert threshold memory_limit * memory_usage_warn_threshold/100, or if the number of redo logs being processed reaches the specified threshold, obcdc considers the downstream consumption speed slow.
You can search the libobcdc.log file for the NEED_PAUSE_REDO_DISPATCH=1 keyword to find the throttling logs of the data consumption module. The REASON field shows the specific reason for throttling.
Throttling for the log pulling module
Throttling for the log pulling module is triggered in the following cases:
The memory usage reaches the alert threshold (
memory_limit * memory_usage_warn_threshold/100).The number of active transactions in memory exceeds the specified threshold, and throttling for the data consumption module has been triggered.
A backlog of tasks occurs in the sequencing module.
A backlog of data persistence tasks occurs.
You can search the libobcdc.log file for the NEED_SLOW_DOWN=1 keyword to find the throttling logs of the log pulling module. The REASON field shows the specific reason for throttling.
The following table describes some of the preceding parameters:
Parameter |
Default value |
Value range |
Description |
|---|---|---|---|
| memory_limit | 8G | [2G, +∞) | The expected memory usage of obcdc. |
| memory_usage_warn_threshold | 85 | [0, 100] | The memory usage threshold for triggering throttling detection. |
| pause_redo_dispatch_task_count_threshold | 80 | [0, 100] | The threshold of the backlog ratio of the consumption queue and the resource recycling queue for triggering throttling for the data consumption module. |
obcdc also has more fine-grained throttling algorithms and parameters that automatically adapt based on the specified value of memory_limit. In rare cases, you may need to fine-tune these parameters listed below. However, in most cases, no adjustment is needed.
The following throttling parameters of obcdc are automatically adjusted. For more information, see obcdc parameters.
redo_dispatcher_memory_limit
extra_redo_dispatch_memory_size
redo_dispatched_memory_limit_exceed_ratio
part_trans_task_active_count_upper_bound
part_trans_task_reusable_count_upper_bound
ready_to_seq_task_upper_bound
storager_task_count_upper_bound
storager_mem_percentage