OceanBase AI Database separates partitions from log streams. All partition transaction logs of a tenant unit can be recorded in one log stream. This reduces the number of log streams, but a single log stream may contain a large amount of table data, requiring obcdc to pull all log stream data.
By default, the finest data synchronization granularity in OceanBase AI Database is at the tenant level. However, for some users, this granularity is too coarse and may cause high downstream consumption load. To resolve this issue, obcdc introduces table-level allowlist and blocklist in OceanBase AI Database to reduce unnecessary data storage. In addition, the sizes of the allowlist and blocklist parameters of obcdc in OceanBase AI Database are unlimited and are no longer subject to the current 64 KB limitation.
In addition, the allowlist and blocklist feature of obcdc in OceanBase AI Database supports table renaming (rename table). You can rename a table on the allowlist to a name on the blocklist through a DDL operation. After that, the table is no longer synchronized, and this change takes effect without the need to modify the allowlist or blocklist configuration.
Parameters
obcdc is compatible with the allowlist and blocklist feature of OceanBase AI Database and supports data synchronization granularities at the tenant, database, and table levels. obcdc has the following three parameters:
Parameter |
Description |
Default value |
Dynamic change supported |
Remarks |
|---|---|---|---|---|
tb_white_list |
The allowlist of tables to synchronize. | *.*.* |
No | The allowlist is in the triplet format of tenant name.database name.table name. By default, *.*.* specifies to synchronize all tables in the cluster. Multiple triplets must be separated with vertical bars (|), and individual triplets support regular expression matching. |
tb_black_list |
The blocklist of tables not to synchronize. | | | No | The blocklist is in the triplet format of tenant name.database name.table name. By default, *.*.* specifies to synchronize all tables in the cluster. Multiple triplets must be separated with vertical bars (|), and individual triplets support regular expression matching. |
enable_white_black_list |
Specifies whether to enable the allowlist and blocklist. | 1 |
No | N/A |
Examples
The following examples demonstrate how to configure the allowlist:
Synchronize the tb1 table in the db1 database of the mytenant1 tenant and all tables in the db2 database of the mytenant2 tenant:
mytenant1.db1.tb1|mytenant2.db2.*
Synchronize the tb1 and tb2 tables in the db1 database of the mytenant1 tenant:
mytenant1.db1.tb1|mytenant1.db1.tb2
Synchronize the tables prefixed with tb in the db1 database of the mytenant1 tenant:
mytenant1.db1.tb*
Note
- Only tables that are in the allowlist and not in the blocklist can be synchronized. By default, the allowlist contains all tables of all tenants, and the blocklist is empty.
- Regular expression matching (fnmatch) is supported, and matching rules are case-insensitive by default.
obcdc supports the following regular expressions: *, ?, [seq], and [!seq].
Use the allowlist and blocklist
Earlier versions of obcdc limited the size of an individual parameter to 64 KB. Currently, the allowlist and blocklist parameters are not subject to this limitation. You can configure the allowlist and blocklist in the libobcdc.conf file.
######### [working_mode] #########
#working_mode=memory
meta_data_refresh_mode=data_dict
#meta_data_refresh_mode=online
######### [synchronous information] #########
tb_white_list=*.*.*
#tb_white_list=sys.oblog*.*|oblog_tt.*.*
tb_black_list=t1.d1.t1
instance_index=0
instance_num=1
After configuration, you need to restart the obcdc host process. For example, start it by running the following command:
./obcdc_tailf -V -f libobcdc.conf -T $TIMESTAMP
obcdc also supports enabling or disabling the allowlist and blocklist. The parameter is enable_white_black_list, and the default value is 1, which indicates that the allowlist and blocklist are enabled.
#libobcdc.conf
enable_white_black_list=0