Note that for parameters of the Boolean type in the following tables, the value 0 indicates False and the value 1 indicates True.
Cluster parameters
Metadata parameters
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| init_log_level | The initial log level at startup. | ALL.*:INFO;SHARE.SCHEMA:INFO | No | The initial log level is used to generate more detailed logs. After startup, the log level is changed to the value of log_level. |
| log_level | The log level after startup. | ALL.*:INFO;SHARE.SCHEMA:WARN | Yes | None |
| max_log_file_count | The maximum number of log files. | 0 | No | The value must be greater than or equal to 0. If the value is 0, the number of log files is not limited. To reduce the number of log files, you must restart obcdc. |
| enable_log_limit | Specifies whether to enable log throttling. | 1 | Yes | 1: Log throttling is enabled. 0: Log throttling is disabled. |
| rootserver_list | The RootService server specified at startup. | \| |
Yes | The information about each server must be in the format of server_ip:server_rpc_port:server_sql_port. |
| cluster_url | The cluster URL specified at startup. | \| |
No | If OceanBase AI Database has a cluster URL, obcdc uses this URL to obtain the RootServer information. High availability is supported for servers. |
| cluster_user | The username in the sys tenant. | Empty (required, to be specified by the user) | No | Specify a user in the sys tenant. This user must have the read privilege on internal tables. |
| cluster_password | The password of the preceding user. | Empty (required, to be specified by the user) | No | None |
| cluster_db_name | The name of the database where the internal tables queried by obcdc reside. | oceanbase | No | You do not need to modify the value. By default, internal tables are in the oceanbase database. |
| tenant_endpoint | The list of servers where the tenant resides. | \| |
No | The information about each server must be in the format of host:sql_port. Separate the information about different servers with vertical bars (\|). |
| tenant_user | The name of a user in the tenant. | \| |
No | The format is user_name@tenant_name or user_name@tenant_name#cluster. |
| tenant_password | The password of the preceding user in the tenant. | \| |
No | None |
| config_fpath | The dump path of the configuration file. | etc/liboblog.conf | No | All obcdc configuration information is dumped into this file. |
| timezone | The time zone of obcdc. | +8:00 | No | This parameter specifies the time zone in which data is formatted and exported, which must be consistent with that of the data source. The default time zone is UTC+8. |
| timezone_info_path | The directory where the time zone metadata file is located. | etc/timezone_info.conf | Yes | When meta_data_refresh_mode is set to data_dict, the timezone_info.conf file in the RPM package must be placed in this directory. |
| skip_ob_version_compat_check | Specifies whether to skip the OceanBase version compatibility check. | 0 | No | obcdc V3.x supports data synchronization only with OceanBase AI Database V2.x and V3.x. In addition, the obcdc version must be later than the OceanBase AI Database version, and both major and minor version numbers are considered. |
| cluster_version_refresh_interval_sec | The interval for querying the minimum version of OBServer nodes in the OceanBase AI Database cluster. | 600 | No | This parameter specifies the interval for refreshing the OceanBase AI Database cluster version, in seconds. |
| log_router_background_refresh_interval_sec | The interval for refreshing log stream metadata. | 10 | Yes | This parameter specifies the interval, in seconds, for refreshing the information about the server corresponding to the log stream in the background. |
Working mode-related parameters
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| working_mode | The working mode. | storage (commercial edition) | No | We recommend that you control the transaction size in your business scenarios. If large transactions exist in your business scenarios, use the persistence mode. In persistence mode, we recommend that you deploy obcdc on an SSD. |
| store_service_path | The path for storing data in persistence mode. | ./storage | No | This parameter is valid only in persistence mode. Relative path: The persistent data is stored in a relative path in the path of the process that calls obcdc. Absolute path: The persistent data is stored in the specified absolute path, such as store_service_path=/data/1. |
Synchronization granularity
obcdc supports data synchronization in multiple dimensions. A single obcdc instance can synchronize the data of one cluster, one or more tenants, several databases, and several tables.
Synchronization granularity |
Single obcdc instance |
Multiple obcdc instances |
|---|---|---|
| Cluster | Supported (a single obcdc instance cannot synchronize data of multiple clusters) | Supported |
| Tenant | Supported | Supported |
| Database | Not supported | Not supported |
| Table | Not supported | Not supported (see the multi-instance parameters below) |
If the traffic of a tenant is heavy, it is difficult for a single obcdc instance to synchronize the data of the entire cluster in real time. In this case, you can use multiple obcdc instances to synchronize the same cluster to share the load, with each instance synchronizing the data of a different tenant.
Allowlist- and blocklist-related parameters
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| enable_white_black_list | Specifies whether to enable the allowlist and blocklist feature in obcdc. | 1 | No | By default, the allowlist and blocklist feature is enabled, and the allowlist and blocklist configurations take effect. If you disable the feature, all data of the tenants specified in the allowlist is synchronized. |
| tb_white_list | The allowlist of tables to be synchronized. | *.*.* | No | The information about each table must be in the format of tenant name.database name.table name, for example, mytenant1.db1.tb1\|mytenant2.db2.\*. |
| tb_black_list | The blocklist of tables not to be synchronized. | \| |
No | The format is the same as above. |
| tablegroup_white_list | The allowlist of table groups to be synchronized. | *.* | No | The information about each table group must be in the format of tenant name.table group name, for example, mytenant1.\*\|or_tenant\*.tg\*. |
| tablegroup_black_list | The blocklist of table groups not to be synchronized. | \| |
No | The format is the same as above. |
| enable_filter_sys_tenant | Specifies whether to filter synchronization data of the sys tenant. | 0 | No | Enable this option when the sys tenant is abnormal but the user tenants are normal. |
Note
obcdc provides data filtering capabilities based on table and table group allowlists and blocklists.
- Only the tables and table groups that are in the allowlist and not in the blocklist are synchronized. By default, all tables and table groups in all tenants are in the allowlist, and the blocklist is empty. We recommend that you configure an allowlist in the production environment to synchronize the data of specified tenants. Do not use the default values, which may pull redundant data.
- The fnmatch function is used to match tables and table groups based on regular expressions. By default, matching is case-insensitive.
Multi-instance-related parameters
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| instance_num | The total number of obcdc instances that share the traffic. | 1 | No | None |
| instance_index | The index of the current obcdc instance in the total number of instances (which can be understood as the hash bucket ID). | 0 | No | The value range is [0, instance_num). Each obcdc instance processes only the data of the log streams that match the following formula: hash(tenant_id, ls_id) % instance_num = instance_idx. |
| enable_global_unique_index_belong_to_multi_instance | Specifies whether to allow synchronization of tables with globally unique indexes in multi-instance scenarios. | 0 | Yes | The value 0 specifies that tables with globally unique indexes are not synchronized in multi-instance scenarios. |
Monitoring-related parameters
Throttling
To prevent out-of-memory (OOM) issues caused by data accumulation in the memory in abnormal cases, obcdc provides throttling capabilities. Throttling is triggered based on the internal state of obcdc. During throttling, obcdc stops pulling new data and resumes when its status recovers. The following table describes the throttling parameters. You can modify them to adjust the throttling level as needed.
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| memory_limit | The memory limit for triggering throttling. | 20G |
Yes | This is one of the throttling reference parameters. obcdc tries to keep the memory usage around the value of memory_limit, but this does not mean that obcdc uses at most memory_limit memory. The minimum value is 2 GB. |
| memory_usage_warn_threshold | The memory usage ratio that triggers log pulling throttling, relative to memory_limit. |
85 | Yes | None |
| system_memory_avail_percentage_lower_bound | The lower limit of available system memory, in percentage. | 10 | Yes | The value range is [10, 80]. |
| part_trans_task_active_count_upper_bound | The maximum number of active partition transactions. | 200000 | Yes | This parameter specifies the number of active partition transaction structs in memory (allocated and not recycled). |
| storager_task_count_upper_bound | The throttling threshold of the number of tasks for the storager module to persist. | 1000 | Yes | The minimum value is 1. |
| storager_mem_percentage | The multiple of the throttling memory occupied by the persistence module. | 1 | Yes | The value 0 specifies automatic adjustment. |
| part_trans_task_reusable_count_upper_bound | The maximum number of partition transactions to reuse. | 10240 | Yes | This parameter specifies the sum of tasks to be output to the user queue, to be output by users, to be returned by users, and pending garbage collection (GC). |
| ready_to_seq_task_upper_bound | The maximum number of tasks to be sequenced. | 2000 | Yes | This parameter specifies the maximum number of tasks to be sequenced, including tasks pending parsing, tasks pending formatting, and tasks pending persistence. |
| pause_fetcher | Specifies whether to forcibly stop pulling logs. | 0 | Yes | The value 0 specifies to forcibly stop pulling logs regardless of the result of the throttling logic. |
| direct_load_inc_thread_num | The number of threads for reading and parsing incremental direct load logs. | 0 | Yes | This is an automatic configuration parameter for memory throttling. Its value can be automatically adjusted based on the memory_limit parameter of obcdc, or you can manually modify the configuration for dynamic adjustment.
NoticeIf you enable the incremental direct load feature and use obcdc for data synchronization, adjust this parameter with caution. Otherwise, memory throttling may be affected and the memory limit may be exceeded. |
| direct_load_inc_queue_backlog_lowest_tolerance | The task backlog threshold that triggers throttling. | 0 | Yes | This is an automatic configuration parameter for memory throttling. Its value can be automatically adjusted based on the memory_limit parameter of obcdc, or you can manually modify the configuration for dynamic adjustment.
NoticeIf you enable the incremental direct load feature and use obcdc for data synchronization, adjust this parameter with caution. Otherwise, memory throttling may be affected and the memory limit may be exceeded. |
The reference throttling conditions are as follows:
- Condition 1: The number of active partitions exceeds the threshold specified by
part_trans_task_active_count_upper_bound. || The memory held by obcdc exceeds the threshold specified bymemory_limit. || The available memory of the system is lower than the threshold specified bysystem_memory_avail_percentage_lower_bound. - Condition 2: The number of partition transaction structs to be reused exceeds the threshold specified by
part_trans_task_reusable_count_upper_bound. || The number of tasks to be sequenced exceeds the threshold specified byready_to_seq_task_upper_bound. - Condition 3: The number of tasks to be persisted exceeds the threshold specified by
storager_task_count_upper_bound. && The memory held by obcdc exceeds the throttling triggering memory (memory_limit) multiplied by the ratio of the memory occupied by the persistence module to the throttling memory (storager_mem_percentage).
The formula for triggering throttling is: ((Condition 1 && Condition 2) || Condition 3) || (pause_fetcher != 0).
The direct_load_inc_thread_num and direct_load_inc_queue_backlog_lowest_tolerance parameters are automatically adjusted as follows:
memory_limit |
2G |
4G |
8G |
16G |
32G |
128G |
|---|---|---|---|---|---|---|
| direct_load_inc_thread_num | 1 | 1 | 1 | 2 | 3 | 5 |
| direct_load_inc_queue_backlog_lowest_tolerance | 4096 | 8192 | 16384 | 32768 | 65536 | 131072 |
When the memory is sufficient, you can appropriately increase the value of direct_load_inc_thread_num to speed up the reading and parsing of incremental direct load logs, and appropriately increase the task backlog threshold that triggers throttling (direct_load_inc_queue_backlog_lowest_tolerance). When the memory is insufficient, you can appropriately decrease the values of direct_load_inc_thread_num and direct_load_inc_queue_backlog_lowest_tolerance, or keep their default values.
Parameters related to the monitoring information output
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| enable_dump_pending_trans_info | Specifies whether to periodically collect and print statistics on the status of distributed transactions held by obcdc. | 0 | Yes | The value 0 specifies not to periodically collect and print the status of distributed transactions held by obcdc. If the value is set to 1, the statistics are printed once every 10 seconds. |
| print_fetcher_slowest_ls_num | The number of partitions with the lowest log-pulling speed to be printed. | 10 | Yes | None |
| print_rpc_handle_info | Specifies whether to print RPC-related information. | 0 | Yes | The value 0 specifies not to print RPC-related information. |
| print_stream_dispatch_info | Specifies whether to print log stream pulling information. | 0 | Yes | The value 0 specifies not to print log stream pulling information. |
| print_ls_heartbeat_info | Specifies whether to print partition heartbeat information. | 0 | Yes | The value 0 specifies not to print partition heartbeat information. |
| print_ls_serve_info | Specifies whether to print partition service information, such as the partition status and the number of partition transactions. | 0 | Yes | The value 0 specifies not to print partition service information, such as the partition status and the number of partition transactions. |
| print_participant_not_serve_info | Specifies whether to print information about the partition transactions of all distributed transactions that are not in service (no data is output). | 0 | Yes | The value 0 specifies not to print information about the partition transactions of all distributed transactions that are not in service (no data is output). |
| print_ls_server_list_update_info | Specifies whether to print the details of the server list when it is updated. | 0 | Yes | The value 0 specifies not to print the details of the server list when the list of partition servers is updated. |
| enable_formatter_print_log | Specifies whether to allow the formatter to output logs when some data fails to be formatted. | 0 | Yes | The value 0 specifies not to allow the formatter to output logs when some data fails to be formatted. |
Feature control-related parameters
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| enable_direct_load_inc | Specifies whether to synchronize data imported in incremental direct load mode. | 0 | No |
Metadata management-related parameters
obcdc V4.1 can obtain metadata, such as information about schemas and log streams, from data dictionaries, without the need to pull logs from OBServer nodes by using RPCs.
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| meta_data_refresh_mode | The mode for obtaining metadata such as information about schemas and log streams. | online | No |
NoticeWhen obcdc synchronizes data from OceanBase AI Database, it ignores the user configuration and forcibly uses the |
Tenant management-related parameters
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| skip_rename_tenant_ddl | Specifies whether to ignore DDL statements that rename a tenant. | 0 | Yes | If a renamed tenant is not included in the obcdc synchronization allowlist, no impact occurs. Otherwise, an error is reported and the process exits normally, because the rename operation violates the allowlist restriction. In this case, we recommend that you modify the allowlist to support the tenants before and after the rename operation. If this parameter is set to true, the DDL statement for renaming the tenant is skipped and data synchronization continues. |
Fetcher module-related parameters
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| ls_count_upper_limit | The number of partition progress values. | 2000000 | No | A maximum of 200 million partitions are supported. |
| check_switch_server_interval_min | The interval, in minutes, of the periodic check on whether to switch the log-pulling server for a partition. | 30 | Yes | None |
SQL query-related parameters
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| access_systable_helper_thread_num | The maximum number of connections in the connection pool for SQL execution. | 64 | No | The value range is [48, 1024]. |
| rs_sql_connect_timeout_sec | The timeout period for SQL connections of RootService. | 40 | Yes | None |
| rs_sql_query_timeout_sec | The timeout period for SQL requests of RootService. | 30 | Yes | None |
| tenant_sql_connect_timeout_sec | The timeout period for SQL connections of a tenant. | 40 | Yes | None |
| tenant_sql_query_timeout_sec | The timeout period for SQL requests of a tenant. | 30 | Yes | None |
| sql_server_change_interval_sec | The time interval, in seconds, for switching the SQL execution server. | 60 | Yes | None |
Filter data based on cluster_id
obcdc allows you to filter data based on the cluster_id blocklist. Although OceanBase AI Database specifies a valid cluster_id range, you can specify cluster_id values when writing data to prevent unwanted data from being synchronized. For more information about how to configure the cluster_id values written to the source, see ob_org_cluster_id.
- Data whose
cluster_idis outside the valid range specified by OceanBase AI Database and OceanBase Migration Service (OMS) is not synchronized. The valid range is [1, 4294901759] U [4294901760, 4294967295]. - Data whose
cluster_idis in thecluster_id_black_listis not synchronized. - Each
cluster_idin thecluster_id_black_listmust range fromcluster_id_value_mintocluster_id_value_max.
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| cluster_id_black_list | The blocklist of cluster IDs for synchronization. | \| |
No | Separate multiple cluster IDs with vertical bars (\|). Each cluster ID must range from cluster_id_value_min to cluster_id_value_max. |
| cluster_id_black_value_min | The lower limit of cluster IDs in the synchronization blocklist. | 2147473648 | No | None |
| cluster_id_black_value_max | The upper limit of cluster IDs in the synchronization blocklist. | 2147483647 | No | None |
Server-level blocklists
obcdc sends SQL and RPC requests to OBServer nodes. OceanBase AI Database provides parameters that allow you to block all requests sent to specific servers.
In addition, because obcdc pulls logs at the partition level, it has a built-in partition-level log-pulling blocklist. Servers that time out or report errors when partition logs are pulled are added to this blocklist. Servers are automatically added to and removed from the blocklist.
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| server_blacklist | A blocklist of OBServer nodes that are blocked from receiving RPC requests from obcdc. | \| |
Yes | The information about each OBServer node must be in the format of svr_ip:rpc_port. Separate the information about multiple OBServer nodes with vertical bars (\|). Logs are not pulled from the OBServer nodes in the blocklist. If an OBServer node cannot provide log services normally, for example, due to network partitioning or high latency, you can manually add the node to the blocklist. |
| sql_server_blacklist | A blocklist of OBServer nodes that are blocked from receiving SQL requests from obcdc. | \| |
Yes | The information about each OBServer node must be in the format of svr_ip:rpc_port. Separate the information about multiple OBServer nodes with vertical bars (\|). |
Partition-level log-pulling server blocklist parameters (generally not modified)
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| blacklist_survival_time_sec | The waiting time, in seconds, of servers in the blocklist. | 30 | No | None |
| blacklist_survival_time_upper_limit_min | The maximum period, in minutes, of servers in the blocklist. | 4 | Yes | None |
| blacklist_survival_time_penalty_period_min | The interval threshold, in minutes, for doubling the time that a server stays in the blocklist when the server is repeatedly added to the blocklist. | 1 | Yes | None |
| blacklist_history_overdue_time_min | The period, in minutes, after which a server is deleted from the blocklist history. | 30 | Yes | The minimum value is 10. |
| blacklist_history_clear_interval_min | The time interval, in minutes, for clearing the information about servers in the blocklist. | 20 | Yes | The minimum value is 10. |
Server cache-related parameters
Parameter |
Description |
Default value |
Real-time modification supported |
|---|---|---|---|
| all_server_cache_update_interval_sec | The time interval, in seconds, for updating the cached server list information. | 5 | Yes |
| all_zone_cache_update_interval_sec | The time interval, in seconds, for updating the cached zone information. | 5 | Yes |
Log pulling-related parameters
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| idle_pool_thread_num | The number of threads that process pending partition-level log-pulling tasks. | 4 | No | The value range is [1, 32]. |
| dead_pool_thread_num | The number of threads that process the log-pulling tasks of recycled partitions. | 1 | No | The value range is [1, 32]. |
| stream_worker_thread_num | The number of threads that dispatch partition-level log-pulling tasks. | 8 | No | The value range is [1, 64]. |
| start_lsn_locator_thread_num | The number of threads that locate the ID of the starting partition log. | 4 | No | The value range is [1, 32]. |
| lob_data_merger_thread_num | The number of threads that process large object (LOB) data. | 2 | No | The value range is [1, +∞). |
| start_lsn_locator_locate_count | The number of requests that complete the task of locating the starting log. | 1 | No | The minimum value is 1. To locate the ID of the starting partition log, a specified number of partition replicas must return the same starting log ID. |
| skip_start_lsn_locator_result_consistent_check | Specifies whether to skip the consistency check on the starting log ID. | No | No | None |
| svr_stream_cached_count | The initial size of the server log stream object pool. | 16 | No | The minimum value is 1. |
| fetch_stream_cached_count | The initial size of the partition log stream object pool. | 16 | No | The minimum value is 1. |
| start_lsn_locator_rpc_timeout_sec | The timeout period, in seconds, of RPC requests for locating the ID of the starting partition log. | 60 | Yes | The minimum value is 1. |
| start_lsn_locator_batch_count | The RPC batch size for locating the ID of the starting partition log. | 2000 | Yes | The minimum value is 1. |
| heartbeat_interval_sec | The interval, in seconds, of RPC requests for partition heartbeats. | 1 | Yes | The minimum value is 1. |
| fetch_log_rpc_timeout_sec | The timeout period, in seconds, of RPC requests for pulling partition logs. | 15 | Yes | The minimum value is 1. |
| progress_limit_sec_for_dml | The upper limit of DML log-pulling progress. | 300 | Yes | The minimum value is 1. To ensure that the progress of all partitions is roughly the same and prevent some partitions from pulling logs too fast, the upper limit of logs provided by the server is set to the minimum progress of all partitions plus the value of progress_limit_sec_for_dml. |
| ls_fetch_progress_update_timeout_sec | The threshold, in seconds, for determining whether an RPC request for pulling partition logs times out. | 3 | Yes | The minimum value is 1. |
| ls_fetch_progress_update_timeout_sec_for_lagged_replica | The timeout threshold, in seconds, for pulling logs from a lagging replica. | 15 | Yes | The minimum value is 1. |
| enable_continue_use_cache_server_list | Specifies whether to use the cached server information. | 0 | No | Enable this option when the sys tenant is abnormal but the user tenants are normal. |
| fetching_log_mode | The log pulling mode. | integrated | No |
Noticeobcdc does not support pulling discontinuous archived logs. |
Parameters related to the log pulling priority
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| region | The region from which obcdc preferentially pulls logs. | default_region | Yes | The region is queried from the internal tables of OceanBase AI Database. obcdc preferentially pulls logs from OBServer nodes in the specified region. |
RPC module-related parameters (modifications not recommended)
Parameter |
Description |
Default value |
Value range |
Real-time modification supported |
|---|---|---|---|---|
| io_thread_num | The number of RPC I/O threads. | 4 | [1, +∞) | No |
| rpc_result_cached_count | The initial size of the object pool for log-pulling RPC results. | 16 | [1, +∞) | No |
| rpc_process_handler_time_upper_limit_msec | The maximum RPC processing time. | 200 | [1, +∞) | Yes |
| rpc_result_count_per_rpc_upper_limit | The maximum number of unprocessed log-pulling RPC results in a single log stream. When this limit is reached, log-pulling RPC requests are paused. | 16 | [1, +∞) | Yes |
Secure communication-related parameters
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| ssl_client_authentication | Specifies whether to enable secure communication. | 0 | No | The default value is 0, which specifies to disable secure communication. |
| ssl_external_kms_info | The secure communication method. | file | No | Only the file mode is supported for external use. The key files are placed in the wallet directory under the obcdc startup directory (create the wallet directory yourself). The directory contains the following key files: ca.pem/client-cert.pem/client-key.pem. |
Sequencer module-related parameters
Parameter |
Description |
Default value |
Value range |
Real-time modification supported |
|---|---|---|---|---|
| sequencer_thread_num | The size of the thread pool for sequencing distributed transactions. | 5 | [1, +∞) | No |
| sequencer_queue_length | The queue of partition transactions to be sequenced. | 102400 | [1, +∞) | No |
Parameters related to the REDO read control module
Parameter |
Description |
Default value |
Value range |
Real-time modification supported |
|---|---|---|---|---|
| redo_dispatched_memory_limit_exceed_ratio | When enable_output_trans_order_by_sql_operation is set to 1, this parameter specifies the size limit of the redo logs to be dispatched and processed. That is, the size of the redo logs allowed to be processed in the memory is redo_dispatched_memory_limit_exceed_ratio * redo_dispatcher_memory_limit. |
2 | [1, +∞) | Yes |
| redo_dispatcher_memory_limit | The size limit of the redo logs to be dispatched and processed. | 512M | [128M, +∞) | Yes |
| extra_redo_dispatch_memory_size | The extra redo quota allocated to skew participants in data skew scenarios to prevent deadlocks that may cause obcdc to output no data. | 4M | [0M, 512M] | Yes |
Parameters related to the row data sorting module
Parameter |
Description |
Default value |
Value range |
Real-time modification supported |
|---|---|---|---|---|
| msg_sorter_task_count_upper_limit | The maximum number of tasks for the row data sorting module in a transaction. | 200000 | [1, +∞) | No |
| msg_sorter_thread_num | The number of threads for the row data sorting module in a transaction. | 1 | [1, 32] | No |
Parameters valid only in persistence mode
Storager module-related parameters
Parameter |
Description |
Default value |
Real-time modification supported |
|---|---|---|---|
| storager_thread_num | The size of the thread pool of the persistence module. | 10 | No |
| storager_queue_length | The number of tasks to be persisted. | 102400 | No |
Parameters related to the Data Processor module
Parameter |
Description |
Default value |
Real-time modification supported |
|---|---|---|---|
| reader_thread_num | The size of the thread pool for reading data from the persistence module. | 10 | No |
| reader_queue_length | The number of tasks waiting for data deserialization. | 102400 | No |
Parameters related to batch redo persistence
Parameter |
Description |
Default value |
Value range |
Real-time modification supported |
|---|---|---|---|---|
| batch_buf_count | The number of memory buffers for batch redo persistence. | 10 | [5, +∞) | No |
| batch_buf_size | The size of the aggregate buffer for batch redo persistence. | 20MB | [2MB, +∞) | No |
Parser module-related parameters
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| dml_parser_thread_num | The size of the thread pool for parsing DML statements. | 5 | No | The minimum value is 1. |
| ddl_parser_thread_num | The size of the thread pool for parsing DDL statements. | 1 | No | The minimum value is 1. |
| skip_reversed_schema_version | Specifies whether to ignore schema version rollback exceptions. | 0 | No | Generally, the schema version monotonically increases during DDL processing. A schema version rollback indicates an exception. This parameter allows you to skip such exceptions. |
Formatter module-related parameters
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| formatter_thread_num | The size of the thread pool for data formatting. | 10 | No | The minimum value is 1. |
| skip_dirty_data | Specifies whether to skip abnormal data. | 0 | Yes | Current exceptions:
|
Parameters related to the binlog record output
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| enable_output_trans_order_by_sql_operation | Specifies whether to output data rows in a transaction in the order in which the SQL statements are executed. | 0 | No | This refers to the internal order in which OceanBase AI Database modifies data, which may not be exactly the same as the order of the SQL statements sent by the client. |
| sort_trans_participants | Specifies whether to sort by distributed transaction participant. | 1 | No | This parameter specifies whether to sort by partition key. When used together with enable_output_trans_order_by_sql_operation, it provides a stable output order for statements. |
| enable_output_hidden_primary_key | Specifies whether to output the hidden primary keys of tables without primary keys. | 0 | No | For a table without a primary key, set enable_output_hidden_primary_key to 1 to output its hidden primary key. |
| enable_convert_timestamp_to_unix_timestamp | Specifies whether to convert a timestamp to a Unix integer value. | 0 | No | None |
| enable_output_invisible_column | Specifies whether to output hidden columns. | 0 | No | None |
| output_heartbeat_interval_msec | The interval for outputting safe checkpoint information. | 1000 | Yes | None |
Resource recycling-related parameters (modifications not recommended)
Parameter |
Description |
Default value |
Real-time modification supported |
Remarks |
|---|---|---|---|---|
| resource_collector_thread_num | The size of the thread pool for resource recycling. | 10 | No | The minimum value is 1. |
| resource_collector_thread_num_for_br | The size of the thread pool for recycling binlog records. | 7 | No | The value must be greater than that of resource_collector_thread_num. |
Notice
We recommend that you do not modify the parameters that are not described in this topic.