This topic describes how OceanBase AI Database reads Paimon-format data, including usage instructions, related configuration parameters, and the external table plugin mechanism. Paimon read capability is integrated as a .so dynamic library. There is no need to statically compile the Paimon SDK into OceanBase AI Database, so the deployment architecture remains unchanged, other features that do not use the plugin are unaffected, and no upgrade mixed-deployment compatibility risks are introduced.
Supported capabilities
OceanBase AI Database accesses Paimon-format data through an external Catalog. You can create a Paimon external Catalog, obtain database and table information from the Catalog, and query Paimon tables using SELECT statements. Supported queries include unfiltered queries, column pruning, comparison predicates, combined AND conditions, and IS NULL conditions. Lake table parallel pruning and parallel query are also supported.
Prerequisites
Before using the Paimon reading feature, ensure the following conditions are met:
lib_ob_paimon.sohas been deployed on each OBServer.- The dynamic library is located in the
LD_LIBRARY_PATHof each OBServer, in the same directory as the OBServer, or in a directory specified by_ob_additional_lib_path. You can also specify an absolute path viaext_plugin_config. - The plugin configurations and dynamic library layout remain consistent across all OBServers to avoid loading failures on some nodes.
- OceanBase AI Database can access the file system or object storage path corresponding to the Paimon warehouse, and the required access permissions have been configured.
- The current operating account has the permissions required to view parameters, modify parameters, create external Catalogs, and query external tables.
The Paimon plugin uses a lazy loading mechanism. After configuration is completed and OBServer is restarted, the plugin is loaded automatically when the related feature is used for the first time.
Configure the plugin
After confirming that the environment meets the prerequisites, the next step is to configure how the plugin is loaded.
View the current configuration
SHOW PARAMETERS LIKE 'ext_plugin_config';
The default configuration declares the loading of the Paimon plugin:
[{"name":"paimon"}]
When path is not explicitly specified, the system parses the dynamic library name according to the lib_ob_<name>.so rule and searches for it in the directories specified by LD_LIBRARY_PATH or _ob_additional_lib_path. The Paimon plugin corresponds to lib_ob_paimon.so.
Specify the plugin configuration
Load the Paimon plugin using the default dynamic library search path:
ALTER SYSTEM SET ext_plugin_config = '[{"name":"paimon"}]';
Load the Paimon plugin using an absolute path:
ALTER SYSTEM SET ext_plugin_config = '[{"name":"paimon","path":"/home/admin/oceanbase/lib/lib_ob_paimon.so"}]';
After modifying ext_plugin_config, you must restart all OBServer nodes. After the restart is complete, you can check the plugin loading status with the following statement:
SELECT * FROM __all_virtual_ext_plugin_info;
If a single plugin fails to load, other plugins can still be loaded normally. The failure reason can be viewed through the error_msg field in this view and the OBServer logs.
Create an external Catalog
After plugins are successfully loaded, you need to create an external Catalog as the entry point for accessing Paimon data. The following example creates an HMS Catalog named hms_catalog to obtain metadata of Paimon tables through Hive Metastore:
CREATE EXTERNAL CATALOG IF NOT EXISTS `hms_catalog`
PROPERTIES (
TYPE = 'HMS',
URI = 'thrift://hms.example.com:9083'
);
View databases and tables
After the Catalog is created, you can browse the databases and tables in it.
After entering the created external Catalog, you can view its databases:
SHOW DATABASES;
Select a target database and view its tables:
USE paimon_test_data_db;
SHOW TABLES;
The syntax used to switch Catalogs and databases should follow the client syntax supported by the target OceanBase AI Database version.
Query Paimon data
After the data entry is set up, you can directly use SQL to query data in Paimon tables. Examples are provided below in ascending order of query complexity.
Query without filtering
SELECT *
FROM all_primitive_types
LIMIT 1;
Filter with AND and NULL conditions
SELECT COUNT(*) AS cnt
FROM all_primitive_types
WHERE col_bool IS NULL
AND col_int IS NULL;
Column pruning and comparison condition filtering
SELECT col_bool, col_int, col_string, col_date
FROM all_primitive_types
WHERE col_int = 50000;
The query returns only the specified columns and applies equality filtering to col_int. Filter conditions that can be pushed down are passed to the Paimon read side through the plugin predicate interface. Conditions that cannot be pushed down are still executed by OceanBase AI Database to ensure result correctness.
Configure parallel pruning
When the amount of table data being queried is large, metadata traversal itself may become a bottleneck. Parallel pruning can process data shards and metadata in parallel using multiple threads, reducing the plan generation time for large table queries. This capability is controlled by tenant-level parameters and is executed using resources from the tenant-level thread pool.
Enable parallel pruning:
ALTER SYSTEM SET enable_parallel_iceberg_pruning = true;
Set the number of threads for lakehouse pruning:
ALTER SYSTEM SET lake_table_pruning_thread_count = 8;
The command names for the parallel pruning switch may vary across different versions. You should refer to the parameter names actually supported by the target version. The number of threads should be set based on the tenant's CPU and concurrent workload to prevent pruning tasks from consuming excessive computing resources.
Configuration parameters
The following table summarizes the main configuration parameters involved in Paimon read features.
Parameter |
Type |
Default value |
Value range |
Level |
Effective |
Introduced in |
Description |
|---|---|---|---|---|---|---|---|
ext_plugin_config |
String | [{"name":"paimon"}] |
JSON array, up to 256 items | Cluster-level | Effective upon restart | V4.6.2.1 | Declares the external table C++ format plugins to load and their dynamic library locations. |
enable_lake_table_parallel_resolving |
BOOL | True |
True or False |
Tenant-level | Effective immediately | V4.6.2.1 | Controls whether to enable parallel pruning for lake tables. |
lake_table_pruning_thread_count |
INT | 4 |
[1, 64] |
Tenant-level | Effective immediately | V4.6.2.1 | Sets the number of threads in the tenant-level pruning thread pool for lake tables. This parameter applies to both Iceberg and Paimon. |
External table plugin mechanism
OceanBase AI Database uses a dynamic plugin mechanism to integrate external table capabilities such as Paimon and ODPS. External table functionality is provided as independent .so dynamic libraries, which are loaded on demand by the OceanBase AI Database main program at runtime, eliminating the need to statically compile third-party SDKs into OceanBase AI Database. The core advantages of this plugin-based design are:
- It isolates third-party dependencies and avoids library version conflicts between different components.
- Plugins can be deployed and upgraded independently without affecting the OceanBase AI Database main process.
- Unused plugins are not loaded, preventing additional resource overhead.
Plugin loading process
Plugin loading is driven by the ext_plugin_config parameter. The overall process is as follows:
- After an OBServer starts, it loads the corresponding dynamic libraries upon first use of external table functionality, based on the plugin list declared in
ext_plugin_config. - During loading, the system resolves the dynamic library path by plugin name and completes version and interface verification.
- Successfully loaded plugins remain resident throughout the OBServer process lifecycle and are not unloaded.
- The failure to load a single plugin does not affect the normal use of other plugins.
Plugin name matching is case-insensitive. Plugins can continue to load their dependent dynamic libraries as needed. For example, lib_ob_paimon.so automatically loads subsequent dependencies required for Paimon reads.
Plugin capabilities
External table plugins provide the following capabilities for OceanBase AI Database:
Capability |
Description |
|---|---|
| Metadata reading | Retrieves table schema information from an external data source, including column names, data types, precision, and other attributes. |
| Type mapping | Automatically converts between external data types, Arrow types, and OceanBase AI Database internal types, and supports nested types such as ARRAY and MAP. |
| Batch data reading | Returns data in columnar batches through the Arrow C Data Interface, balancing performance and compatibility. |
| Predicate pushdown | Passes filter conditions from queries to plugins for execution, reducing data transmission. Supports comparisons, IN, IS NULL, AND/OR/NOT, and other conditions. Conditions that cannot be pushed down are executed by OceanBase AI Database to ensure result correctness. |
| Statistics collection | Provides table-level and column-level statistics, including the number of rows, number of NULL values, number of distinct values, minimum value, and maximum value, to help the query optimizer generate better execution plans. |
| Read-only file access | The plugin accesses external storage through the file system interface provided by OceanBase AI Database and supports only read operations. |
ext_plugin_config configuration rules
The value of ext_plugin_config is a JSON array, with each element corresponding to an external table plugin:
[
{
"name": "<plugin>",
"path": "</abs/lib.so>"
}
]
The configuration rules are as follows:
name: The name of the plugin and is required. The name must be non-empty, up to 64 characters in length, case-insensitive, and is internally treated as lowercase.path: The absolute path of the plugin's dynamic library and is optional.- If
pathis specified, the system directly loads the dynamic library from this path. - If
pathis not specified or is empty, the system parses the dynamic library aslib_ob_<name>.soand searches for it in the dynamic library search path.
- If
- An empty array
[]indicates that no external table plugins are to be loaded. - The array can contain at most 256 entries. Entries exceeding this limit will be truncated and logged.
nameandpathmust comply with JSON escape rules; for example, backslashes should be written as\\.- Invalid JSON or non-array configurations will result in no plugins being loaded during this load and an error log will be recorded.
- Configurations are read only once during the lifetime of a single OBServer process. Modifying parameters during runtime will not change the set of loaded plugins.
- If a
dlopen,dlsym, ABI verification, or file lookup error occurs for a single plugin, only that plugin is skipped and other plugins are not affected.
Examples of using ext_plugin_config
Declare multiple plugins simultaneously:
ALTER SYSTEM SET ext_plugin_config = '[{"name":"paimon"},{"name":"odps","path":"/abs/lib_ob_odps.so"}]';
Disable all external table plugins:
ALTER SYSTEM SET ext_plugin_config = '[]';
All of the above modifications take effect after restarting all OBServer nodes.
Considerations
- Configuration modifications take effect after a restart. After modifying
ext_plugin_config, you must restart all OBServer nodes for the new configuration to take effect. Modifying parameters during runtime will not change the set of loaded plugins. - Configuration and deployment must be consistent across all nodes. The order of plugins, the version of the dynamic libraries, and the layout of dependency libraries must be consistent across all OBServer nodes. Otherwise, queries may fail on some nodes.
- Plugins provide only read-only capabilities. Current external table plugins support only reading external data; they do not support writing to or modifying external data sources through the plugins.