The catalog synchronization feature automatically synchronizes database and table metadata from the OceanBase AI Database internal catalog to Hive Metastore (HMS). This enables computing engines (such as Spark/Daft) and OceanBase DataStudio to discover and query tables in OceanBase AI Database through HMS.
Overview
In the Unified Catalog system, metadata needs to be maintained and managed in a unified way so that third-party engines can obtain the required metadata information directly from HMS. Therefore, OceanBase AI Database needs to actively register tables with HMS, so that these engines can discover and access data in OceanBase AI Database. Catalog data synchronization provides this automatic synchronization capability, eliminating the need for users to manually register each table individually on the HMS side.
After synchronization is enabled, the Internal catalog appears as a federated view: local tables and remote tables (such as Iceberg tables) are displayed together, and users do not need to distinguish their sources. Locally created tables are automatically registered with the remote HMS as placeholder tables.
Overall, all data writes are completed on the OceanBase AI Database side, and the compute engine in lakebase can directly consume the metadata.
Scope of application
- Supports only unidirectional synchronization from the Internal catalog to an external HMS.
- Applies only to new DDL operations after synchronization is enabled; existing objects are not automatically processed.
Core concepts
- Placeholder Table: The
EXTERNAL_TABLEregistered in HMS for a local OceanBase AI Database table, used for discovery by third-party engines. The placeholder table carries OceanBase AI Database lineage information (ob.type,ob.tenant_id,ob.table_id,ob.schema_version, etc.) in itsparameters. HMS adds a protection flag to prevent accidental deletion by external tools. - Federated View: After synchronization is enabled, local tables and remote tables under the Internal catalog are displayed together, and users do not need to care which side the table is actually located on.
- Synchronization Target Catalog: The HMS-type External catalog specified by the tenant parameter
catalog_sync_target, serving as the receiver for synchronization.
Scenarios
- Unified consumption by data platforms: Business tables written to OceanBase AI Database need to be read by computing engines (such as Spark/Daft) or OceanBase DataStudio via HMS.
- Lakehouse integrated queries: Query local tables and remote tables (such as Iceberg tables) within OceanBase AI Database without manual catalog switching.
- Unified metadata governance: View the table structure of OceanBase AI Database through HMS to avoid inconsistencies across multiple metadata sources.
Prerequisites
- Create an HMS-type External catalog using
CREATE EXTERNAL CATALOG. - The OceanBase AI Database cluster can access the Thrift address of HMS.
- Have the privileges to modify the tenant parameter
catalog_sync_target.
Procedure
Create the synchronization target Catalog.
For more syntax details, see CREATE EXTERNAL CATALOG.
CREATE EXTERNAL CATALOG hms_sync_catalog PROPERTIES ( TYPE = 'hms', URI = 'thrift://<hms-host>:<port>' );Enable synchronization.
Configure the synchronization target Catalog. Once enabled, DDL operations on the Internal catalog will be automatically synchronized to HMS.
ALTER SYSTEM SET catalog_sync_target = 'hms_sync_catalog';Disable synchronization.
-- Setting it to null disconnects the synchronization. ALTER SYSTEM SET catalog_sync_target = '';Behavior after disabling:
- Remote tables become immediately invisible (queries no longer fall back to the synchronized external catalog).
- Local tables are not affected
- Registered placeholder tables in HMS are not automatically cleaned up. To remove them, you must manually delete them from HMS.
Resume synchronization.
When resuming synchronization, note the following:
- If you point back to the original catalog, you need to execute the synchronization start statement again in the second step of the configuration method.
- If you point to a different catalog, the synchronization target will change to the HMS corresponding to the new catalog.
- During the disconnection period, tables newly created on the OceanBase AI Database side will not be synchronized to HMS.
- After resuming synchronization, DDL operations performed during the disconnection period are not supported for retrospective synchronization.
Supported DDL operations
The following linkages only take effect under an Internal catalog with synchronization enabled. Operations performed directly in the External catalog follow an independent path and do not trigger synchronization. For detailed syntax on switching catalogs, see SET CATALOG.
CREATE TABLE
When executing
CREATE TABLEunder an Internal catalog, the system first registers a placeholder table in HMS, then creates the local table. This order ensures that HMS serves as a pre-checkpoint for table name uniqueness and allows for compensatory rollback if local table creation fails.ScenarioHandlingA table with the same name already exists in HMS (either a placeholder table or an Iceberg table) Error: Table already exists Registration succeeds in HMS, but local table creation fails Automatic compensatory rollback; deletes the newly registered HMS placeholder table Both sides succeed The table becomes visible in both HMS and OceanBase AI Database DROP TABLE
When executing
DROP TABLEunder an Internal catalog, the system first deletes the local table, then deregisters the HMS placeholder table. This order uses local deletion as the concurrency control point (concurrent DROPS only allow one to succeed).ScenarioHandlingLocal table does not exist Error HMS deregistration fails Error (not ignored silently) CREATE DATABASE
First create a placeholder database in HMS, then create the local database.
ScenarioHandlingA database with the same name already exists in HMS Claim it, skip HMS and create only the local one HMS creation fails for reasons other than "already exists" Error, do not create the local one DROP DATABASE
Delete the HMS and local databases separately.
Note that if a remote table still exists under this database, deletion is strictly prohibited and an error is returned to avoid cascading effects on the referenced remote table.
ALTER TABLE
Currently, changes to columns (
ADD COLUMN,MODIFY COLUMN) by using theALTER TABLEstatement are supported for synchronization to HMS.Unsupported DDL operations
OperationDescriptionRENAME TABLEDirectly intercepted and an error is returned when synchronization is enabled. CTAS(CREATE TABLE AS SELECT)Directly intercepted and an error is returned when synchronization is enabled.
Query semantics
Federated views
After synchronization is enabled, when querying tables in the Internal Catalog, the system follows these rules: it first queries local tables. If a table with the same name does not exist locally, it automatically falls back to querying the synchronized External Catalog. Remote tables are accessed via the External Table path, and this process is transparent to users.
Users do not need to be aware of the actual storage location of tables and can directly query them using a unified
SELECTstatement.Conflict resolution for duplicate names: During synchronization establishment, the system does not perform a full conflict check. If a situation where a local table and a remote table have the same name occurs at runtime (an abnormal state), the system prioritizes accessing the local table, and users must manually clean up the conflicting tables.
SHOW TABLES
Displays local and remote tables in a merged view and automatically deduplicates them. In case of a name conflict (abnormal state), the local table prevails.
SHOW DATABASES
Controlled by the system variable
_enable_catalog_fusion(default istrue):-- When enabled, SHOW DATABASES statements will consolidate and deduplicate both internal and externally synchronized external databases. SET _enable_catalog_fusion = true; SHOW DATABASES;The semantics of
SHOW DATABASES FROM <catalog>remains unaffected.Control for displaying placeholder tables
_show_sync_placeholder_table(GLOBAL/SESSION, default isfalse): When executingSHOW TABLESin the Internal Catalog and Catalog fusion is enabled (_enable_catalog_fusion), this variable controls how tables that have been synchronized to the remote HMS are displayed:false(default): Deduplication mode. If a table with the same name already exists locally, the corresponding synchronized placeholder table is not displayed repeatedly.true: Complete mode. All synchronized placeholder tables are displayed without deduplication.
SET _show_sync_placeholder_table = true; SHOW TABLES FROM hms_sync_catalog.<db>;
Type mapping
The type sets of OceanBase AI Database and Hive do not completely overlap. During synchronization, the following rules apply:
OceanBase AI Database type |
Processing on HMS |
|---|---|
INT / BIGINT / VARCHAR / DECIMAL / TIMESTAMP / CHARand other Hive recognizable types. |
Directly mapped to the corresponding Hive data type |
unsigned / enum / set / bit / datetime(6) / timestamp_ntz / year / time / jsonand other types that Hive does not recognize. |
Downgraded toSTRING, and at the same time in the table attributeparametersThe original OceanBase AI Database type is recorded in the column (for example,ob.col.<col>.original_type=geometry) |
Limitations and constraints
Limitations |
Description |
|---|---|
| Synchronization Direction | Only supports unidirectional synchronization from Internal to External HMS |
| Scope | Effective only for new DDL operations after synchronization is enabled; existing objects are not automatically processed. |
RENAME TABLE |
Not supported when synchronization is enabled. The request is directly intercepted at the entry point and an error is returned. |
CTAS(CREATE TABLE AS SELECT) |
Not supported when synchronization is enabled. The request is directly intercepted at the entry point and an error is returned. |
| Remote table write operations | Remote tables are read-only. To perform write operations, you must switch to the external catalog. For detailed syntax on switching catalogs, see SET CATALOG. |
DROP DATABASEProtection |
Deletion is prohibited when remote tables exist under the database. |
| Name conflict | No complete conflict check is performed during synchronization establishment. During runtime, local data takes precedence. For exceptional cases of duplicate names, manual cleanup by the user is required. |
| Permission control | The current version does not support privilege management for remote tables. |
Example
Complete process.
-- 1. Create an HMS external catalog CREATE EXTERNAL CATALOG hms_sync_catalog PROPERTIES ( TYPE = 'hms', URI = 'thrift://hms-host:9083' ); -- 2. Enable Synchronization SET CATALOG internal; ALTER SYSTEM SET catalog_sync_target = 'hms_sync_catalog'; -- 3. Create a database and table (automatically synchronize to HMS) CREATE DATABASE sync_demo_db; USE sync_demo_db; CREATE TABLE orders (id INT, amount DECIMAL(10,2), created TIMESTAMP); -- 4. View under Internal (Consolidated View) SHOW TABLES; -- 5. View the synchronization results in External Catalog SHOW TABLES FROM hms_sync_catalog.sync_demo_db; -- 6. Query a remote table SELECT * FROM hms_sync_catalog.sync_demo_db.orders; -- 7. Stop Synchronization ALTER SYSTEM SET catalog_sync_target = '';Clean up the environment.
SET CATALOG internal; ALTER SYSTEM SET catalog_sync_target = ''; DROP TABLE IF EXISTS sync_demo_db.orders; DROP DATABASE IF EXISTS sync_demo_db; DROP CATALOG IF EXISTS hms_sync_catalog;
Quick reference for parameters and system variables
Name |
Level |
Default Value |
Description |
|---|---|---|---|
catalog_sync_target |
tenant | empty | The name of the target catalog for synchronization. If this parameter is empty, synchronization is disabled. |
_enable_catalog_fusion |
GLOBAL / SESSION | true |
Controls whether SHOW DATABASES merges and displays the synchronized databases of the external catalog |
FAQ
Q: After enabling synchronization, will existing tables be synchronized to HMS?
No. Synchronization only takes effect on new DDL operations after it is enabled. Existing tables need to be manually registered in HMS or dropped and recreated.
Q: After disabling synchronization, will placeholder tables in HMS be automatically deleted?
No. Placeholder tables remain in HMS, but OceanBase AI Database queries no longer roll back to the remote database. To clean them up, you must delete them manually in HMS.
Q: What should I do if a table is successfully registered in HMS during creation but fails to be created locally?
The system automatically performs compensatory rollback by dropping the newly registered HMS placeholder table. If compensation also fails (in extreme cases), you can retry the cleanup with a regular DROP TABLE.
Q: Can I directly INSERT/UPDATE data into a remote table?
No. Remote tables are read-only. When write operations are required, switch to the corresponding external catalog using SET CATALOG <external_catalog> to perform the operations directly.
Q: Can I RENAME TABLE after enabling synchronization?
No. RENAME TABLE is explicitly prohibited in synchronization mode. If renaming is needed, first disable synchronization, rename the table, and then re-enable synchronization.
Q: Can I execute CTAS after enabling synchronization?
No. CREATE TABLE AS SELECT is prohibited in synchronization mode. If such an operation is needed, disable synchronization first and then execute it.
Q: How do I confirm whether synchronization is effective?
After creating a table in the internal catalog, check it under the external catalog:
SHOW TABLES FROM hms_sync_catalog.<db>;
If the corresponding table is visible, synchronization is successful.