To facilitate queries against data stored in various external data sources, OceanBase AI Database introduces the Catalog feature, which achieves unified management of external data sources through a new data hierarchy.
Catalog types
Catalogs are divided into the following two types:
Internal catalog
An internal catalog is used to manage data stored within OceanBase AI Database. Each tenant has exactly one internal catalog named
internal. Internal catalogs cannot be created, dropped, or modified.External catalog
Connects to external data sources to retrieve metadata of external data. You can directly query external data through an external catalog without performing data import or migration.
Catalog advantages
Multi-level data management architecture
Based on the original Database-Table hierarchy, a Catalog layer is added to form a three-tier structure: Catalog-Database-Table, supporting direct access to the directory tree of external data sources.
Automatic mapping and on-demand querying
By declaring a connection to an external data source using
CREATE EXTERNAL CATALOG, you can directly query external tables without manually creating a mapping table.Unified query experience
When querying external data sources, the syntax is consistent with local tables, supporting cross-source correlation analysis.
Catalog features
Supports multiple data sources
Supports external data sources such as ODPS (MaxCompute) and HMS (Hive, Iceberg).
Privilege isolation
Controls access permissions to external data sources at the Catalog layer to ensure data security.
Performance optimization
Query plan optimization based on Catalog reduces intermediate data transmission and improves cross-source query efficiency.
Scenarios
Joint analysis
Scenarios where joint analysis of data from OceanBase AI Database and external data sources is required.
Data lake/data warehouse integration
Build a unified data lake architecture to directly query massive amounts of data stored in external systems such as ODPS and Hive.
ETL pipeline optimization
Reduce data migration steps by accessing source system data directly through the Catalog, thereby lowering the complexity of ETL development.
Limitations
In the current version, data in catalogs is read-only and cannot be modified. All modification statements on objects under catalogs, such as DROP TABLE and INSERT INTO, are prohibited.