Data migration refers to the process of moving data from a source system to OceanBase AI Database, used for database replacement, version upgrade, data warehouse integration, and more.
Migration phases
Data migration is typically planned in the order of "data structure → full data → incremental data" (though some phases may overlap in practice, subject to the tool and window availability):
Stage |
Target |
Common methods |
Typical tools and methods |
|---|---|---|---|
| Schema migration | The target database has the same tables, indexes, and constraints (and necessary partitioning strategies) as the business. | DDL conversion, data type and character set mapping, and rewriting of incompatible syntax | OMS (object synchronization), DBCAT (table structure), manual DDL |
| Full migration | Complete the historical data baseline by the scheduled time | Batch import and export, parallel splitting, and direct load | OMS, obloader + direct load, DataX, MyDumper + import, LOAD DATA |
| Incremental migration | After full synchronization is complete, continuous change synchronization will continue until switchover or long-term dual-active deployment. | Log parsing (Binlog, Redo, etc.), latency tracking, conflict and reinsertion strategies | OMS |
Heterogeneous database migration
Migration scenarios
Heterogeneous data migration is mainly applicable to the following scenarios:
- Database upgrade or replacement: Migrate an existing database system to OceanBase AI Database to achieve better performance, scalability, or cost-effectiveness.
- Data consolidation: Integrate data from multiple different database systems into a unified OceanBase AI Database.
- Architecture upgrade: Migrate from a traditional database to a distributed database architecture.
In these data migration scenarios, attention should be paid to:
- Schema and data type mapping: Precisely define the data structure and type mapping rules between the source and target (OceanBase AI Database).
- Performance optimization: For migrating large volumes of data, adopt efficient migration strategies such as parallel processing, sharding migration, and batch loading to shorten the migration window.
- Data consistency verification: After migration completion, ensure data integrity and accuracy through methods such as hash verification, sampling comparison, or full comparison.
MySQL to OceanBase AI Database migration
Oracle to OceanBase AI Database migration
Other data sources to OceanBase AI Database migration
Data source |
References |
|---|---|
| PostgreSQL | Use OMS to migrate a PostgreSQL database |
| DB2 LUW (table structure) | Use DBCAT to migrate table structures from a DB2 LUW database to OceanBase AI Database |
| CSV | Use DataX to migrate CSV files to OceanBase AI Database |
Homogeneous database migration
Migration scenarios
Homogeneous data migration is mainly applicable to the following scenarios:
- Version upgrade: Migrate data from an old version of an OceanBase AI Database cluster to a new version to obtain new features or improve performance.
- Migration between OceanBase AI Database clusters.
Since both the source and target are OceanBase AI Databases, data format conversion is usually not required for this type of migration. However, attention should still be paid to:
- Data consistency during migration: Ensure data consistency during migration through transaction logs or snapshot mechanisms.
- Downtime control: Select an appropriate migration plan based on business RTO/RPO requirements to minimize or avoid service interruptions.
- Version compatibility: When migrating between versions, pay attention to subtle differences in SQL syntax, system parameters, or internal implementations.
Migration between tenants of OceanBase AI Database
Scenario |
References |
|---|---|
| Data migration between user tenants of OceanBase AI Database |
Big data ecosystem integration
Migration scenarios
- Synchronize offline data warehouse data to a real-time analytics system: Synchronize data from offline data warehouses such as Hive or Spark to real-time data processing platforms (such as Flink or Kafka) to support real-time data processing and streaming analysis.
- Archive offline data to long-term storage: Archive historical data from offline data warehouses to long-term storage (such as cloud object storage S3) to reduce storage costs and meet data retention policies.
- Import offline data into OceanBase AI Database: Migrate or synchronize data from offline data warehouses such as Hive to OceanBase to support unified analysis, data services, or integration with business data.
In these data migration scenarios, pay attention to the following:
- Adapt source data format to target table: Check whether the source file format (such as CSV, Parquet, or ORC) is consistent with the column types, precision, time format, and partition fields of the OceanBase target table. If necessary, perform field mapping and type conversion during table creation or import.
- Efficient transmission of massive data: Use distributed tools such as DistCp to improve transmission efficiency through parallel processing and network optimization.
- Align metadata with table structure: Ensure that the source table definition, partition information, and schema are consistent with the target table (column names, types, partitioning strategy) in OceanBase AI Database, avoiding mismatches where only data is migrated but the target table structure does not match.
Data source |
Document |
|---|---|
| Hive | Use OMS Community Edition to migrate Hive, Use obloader to migrate Hive |
| HBase | Use OMS Community Edition to migrate HBase |
| ORC, Parquet, and CSV files | Use obloader to migrate ORC, Parquet, and CSV files |