Apache Iceberg is an open-source table format (Lake Table Format). The data files and metadata files of a table are stored in external storage such as S3, S3-compatible object storage, or HDFS. The Catalog manages the table's namespace and metadata pointers.
OceanBase AI Database connects to an external Catalog service (such as HMS Catalog or REST Catalog) through External Catalog, loads the Iceberg table metadata from the corresponding Catalog, and then reads the underlying data files. Iceberg tables cannot be accessed directly through a simple file path. They require metadata information such as metadata.json, snapshots, and manifests provided by the Catalog.
Note
The difference between HMS Catalog and REST Catalog lies in which Catalog service is used to load Iceberg table metadata. The features and column types supported by Iceberg tables are consistent in both access methods. For details on creating and configuring authentication for HMS Catalog, see Load a Hive table through Catalog.
Notice
Currently, only OceanBase AI Database in MySQL-compatible mode supports access to Iceberg tables through External Catalog.
In the Iceberg ecosystem, the roles of each component are as follows:
Concept |
Description |
|---|---|
| Storage | Physical storage locations of data files and metadata files, such as S3, S3-compatible object storage, or HDFS |
| File format | The encoding format of data files. Iceberg tables typically use the Parquet format. For Iceberg tables of V1 or V2, the Parquet format is recommended. |
| Table format (Iceberg) | Table version, partition, schema, and snapshot management mechanisms |
| Catalog | Table name to Iceberg metadata location mapping service |
| External Catalog | Objects used to connect to external Catalogs in OceanBase AI Database |
OceanBase AI Database does not directly parse the object storage directory structure to "discover" Iceberg tables. Instead, it obtains the table definition from an external Catalog service through External Catalog, locates and reads the data files based on the metadata.
Supported capabilities
Supported types
Iceberg is a new generation of open table format specification. HMS Catalog extends its support to manage Iceberg tables with Hive metadata, achieving a unified metadata entry point.
The
CREATE EXTERNAL CATALOGsyntax and parameters for each Catalog type are subject to CREATE EXTERNAL CATALOG and the current version of the product documentation.Advanced features
- Iceberg schema evolution, such as adding and dropping columns
- Iceberg partition transforms, such as
bucket()andyear()partition expressions
Prerequisites
- Iceberg table is ready: The target Iceberg table is registered in the corresponding Catalog, and its metadata is complete and available.
- Storage access: All OBServer nodes have read permission for the Iceberg table's warehouse path. If the underlying storage is HDFS, deploy the Java SDK in advance. For more information, see Deploy Java SDK.
- Catalog service reachable: The OceanBase AI Database cluster can access the HMS Thrift endpoint, the REST Catalog service address, or the object storage or file system path.
- Permissions: The current user must have permissions such as
CREATE CATALOGandUSE CATALOG. If the storage layer requires Kerberos or an AccessKey, configure additional Location or Catalog credentials.
Create an External Catalog for accessing an Iceberg table
Load via HMS Catalog
When Iceberg table metadata is in Hive Metastore, create an External Catalog with TYPE = 'HMS':
CREATE EXTERNAL CATALOG iceberg_hms
PROPERTIES (
TYPE = 'HMS',
URI = 'thrift://hms.example.com:9083'
);
For the creation procedure, Kerberos authentication, and Location configuration, see Access Hive tables through HMS Catalog.
Load through REST Catalog
REST Catalog provides Iceberg metadata through HTTP APIs. When creating an External Catalog, you must specify the Catalog type as REST and configure parameters such as the REST service address, warehouse, and authentication information.
CREATE EXTERNAL CATALOG iceberg_rest
PROPERTIES (
TYPE = 'REST',
URI = '<rest_catalog_api_endpoint>',
WAREHOUSE = '<warehouse_name_or_path>'
-- For other parameters, refer to the CREATE EXTERNAL CATALOG syntax documentation.
);
The complete list of parameters for REST Catalog (such as OAuth2 authentication and the WAREHOUSE format) varies depending on the REST service implementation. For details, see CREATE EXTERNAL CATALOG.
Query an Iceberg table
Switch Catalogs
SET CATALOG iceberg_hms;
USE analytics_db;
Alternatively, use a three-part identifier to avoid context switching:
SELECT * FROM iceberg_hms.analytics_db.events
WHERE event_date = '2025-04-01'
LIMIT 100;
Sample query
-- Aggregate query
SELECT product_id, SUM(sales_amount)
FROM iceberg_hms.analytics_db.sales
WHERE event_time >= '2025-04-01'
GROUP BY product_id;
-- Federated query with internal tables
SELECT o.order_id, e.event_type
FROM internal.trade_db.orders o
JOIN iceberg_hms.analytics_db.events e ON o.user_id = e.user_id;
View metadata
SHOW CATALOGS;
DESC iceberg_hms.analytics_db.sales;
SHOW CREATE TABLE iceberg_hms.analytics_db.sales;
Iceberg table capabilities
The following capabilities apply to Iceberg tables accessed through an External Catalog. The features supported by an Iceberg table are consistent with its column types whether loaded through HMS Catalog or REST Catalog; the main difference lies in which Catalog service retrieves the metadata. If write capabilities differ due to version or Catalog type, the behavior of the current version prevails.
Read capability
Feature |
Supported operations |
|---|---|
SELECT query |
Supported |
| Partition pruning | Supports filtering by partition columns or partition transform expressions. |
| Column pruning | Supported |
| Time travel | Some versions support querying by snapshot or timestamp. The syntax is subject to the documentation of the current version. |
| Position Delete / Equality Delete / Deletion Vector | Read from a table containing deleted files. The specific supported scope is subject to the current version. |
| Schema evolution (adding and dropping columns) | Supports reading evolved schemas. |
Partition transform (such as year(), bucket(), and truncate()) |
Supported |
Write capability
Action |
Supported operations |
|---|---|
INSERT INTO |
Supported in some scenarios. Partition writes are subject to limitations. |
INSERT OVERWRITE |
Not supported |
UPDATE / DELETE |
Not supported |
Notice
Iceberg tables accessed through HMS Catalog are currently read-only. The availability of the INSERT INTO capability for an Iceberg table depends on the External Catalog type and the current version description. Confirm this before use.
Data file format
The underlying data file format for Iceberg tables is primarily Parquet. Metadata files include metadata.json, manifest, manifest-list, and others, which OceanBase AI Database parses during queries.
Considerations
- Distinguish between Catalog and storage: The Catalog provides metadata mapping, while data files remain in external storage such as S3, S3-compatible object storage, or HDFS. Storage credentials must be configured separately.
- Distinguish between table format and file format: Iceberg is a table format, whereas Parquet is its commonly used underlying file format. Do not use these two terms interchangeably.
- Iceberg tables in HMS: Hive 1.2.x, 2.3.x, and 3.1.x do not natively manage Iceberg. However, if engines such as Spark or Flink have registered Iceberg metadata in HMS, OceanBase AI Database can access it through HMS Catalog. For details, see the Iceberg compatibility description in Access Hive tables through HMS Catalog.
- Files not registered in the Catalog: Raw Parquet or ORC files on object storage are not Iceberg tables. Use an File external tables to access them.
- Limitations: The parameters and behavior of REST and FILESYSTEM Catalogs are subject to the documentation of the current product version. Iceberg features not listed in this topic are considered unsupported in the current version or require further confirmation.