Migrate incremental data from OceanBase Database in MySQL-compatible mode to TiDB Database

You can create a data migration task from OceanBase Database in MySQL-compatible mode to TiDB Database to seamlessly migrate incremental data from the source database to the target database.

Prerequisites

• You have created a cluster and a tenant in the source OceanBase database. For more information, see Create an instance and Create a tenant.

• You have created a target database instance.

• You have created dedicated database users for data migration on both the source and target sides and granted them the required permissions. For more information, see User privileges.

Limitations

• Only users with the project role of Project Owner, Project Admin, or Data Services Admin can create new data migration tasks.

• Currently, data migration supports TiDB databases of versions V4.x, V5.4, V6.x, and V7.x, and OceanBase Database in MySQL-compatible mode of versions V2.x, V3.x, and V4.x.

• When migrating data from OceanBase Database in MySQL-compatible mode to TiDB Database, only incremental synchronization is supported; DDL synchronization is not supported.

• The data migration service only supports migrating objects whose database, table, and column names are ASCII-compliant and do not contain special characters (including .|"'`()=;/& and line breaks).

• The data migration service only supports the TiCDC Open Protocol; other protocols are not supported. If you use an unsupported protocol, it may cause a JDBC-Connector exception with a null pointer error.

• The data migration service does not support triggers (Trigger) on the target side. If triggers exist, data migration may fail.

Considerations

• If the source character set is UTF-8, it is recommended that the target use a character set compatible with the source (for example, UTF-8 or UTF-16) to avoid issues such as garbled characters on the target due to character set incompatibility.

• If you modify a unique index on the target side, you need to restart the data migration task; otherwise, data inconsistency may occur.

• Clock desynchronization between nodes, or between the client terminal and the server, may cause inaccurate latency reporting during incremental synchronization.

  For example, if a clock is earlier than standard time, the reported latency may be negative. If a clock is later than standard time, it may cause a positive latency.

• Verify whether the migration precision for column types such as DECIMAL, FLOAT, and DATETIME meets your expectations. If the precision of the target field type is lower than that of the source field type, truncation may occur, leading to data inconsistency between the source and target.

• When creating a new data migration task, you only configured Incremental Synchronization. Data migration requires that local incremental logs in the source database be retained for more than 48 hours.

• If performing incremental synchronization with OceanBase Database as the source and the database table objects contain UDT columns, the incremental synchronization task may fail due to unsupported UDT columns.

Supported source and target instance types

In the following table, the instance types supported by OceanBase Database in MySQL-compatible mode include Dedicated (Transactional) and Self-managed Database.

Data type mappings

Procedure

1. Create a data migration task.

   

   1. Log in to the OceanBase Cloud console.

   2. In the left-side navigation pane, click Data Services > Data Migration.

   3. On the Data Migration page, click the Data Migration tab.

   4. On the Data Migration tab, click Create Task in the upper-right corner.

2. In the Edit Task Name text box, enter a custom name for the migration task.

   We recommend using a combination of Chinese characters, numbers, and letters. The name must not contain spaces and cannot exceed 64 characters in length.

3. On the Configure Source & Target page, configure the parameters.

   1. In the Source section, configure the parameters.

      If you need to reference an existing data source, click Quick Fill on the right side of Source and select the target data source from the drop-down list. After selection, the configurations for the source section will be automatically populated. If you want to save the current configuration as a new data source, click the Save icon in the upper-right corner of the Source section.

      You can also click Manage Data Source in the Quick Fill drop-down list to go to the Data Source page to view and manage data sources. This page provides unified management for different types of data sources. For more information, see Data Sources.

      Parameter	Description
      Cloud Provider	Currently supports Alibaba Cloud, AWS, and Google Cloud.
      Database Type	Select OceanBase MySQL Compatible for the source.
      Instance Type	Includes Dedicated (Transactional) and Self-managed Database.
      Region	Select the region where the source database is located.
      Connection Type	Includes Endpoint and Public IP. For Alibaba Cloud, AWS, and Google Cloud, you can also select Peering Connection. If you select Endpoint, add the account ID displayed on the page to the allowlist of your endpoint service to allow connections from this account's endpoint to the endpoint service. For more information, see Select Private Network Connection. If you select Public IP, add the data source IP address displayed on the page to the allowlist of the OceanBase database instance to ensure connectivity. For more information, see Select Public Network Connection. Note This parameter is displayed only when the instance type is set to Self-managed Database. After you select the regions for the source and target, the page will display the data source IP address that needs to be added to the allowlist. If you select Peering Connection, choose an available peering connection from the drop-down list. Add the IP address displayed on the page to the OceanBase Database instance's inbound security group rules or allowlist to ensure connectivity. For more information, see AWS peering connection, Alibaba Cloud peering connection, and Google Cloud peering connection.
      Peering Connection ID	This parameter is displayed only when Instance Type is set to Self-managed Database and Connection Type is set to Peering Connection. Select an available peering connection from the drop-down list. The list displays only connections that match the current cloud provider and region and are in the Connected state. It also displays the corresponding VPC ID and CIDR block. If no connection is available, click Create to open a new tab and create a data service peering connection.
      Connection Information	This parameter is displayed only when the instance type is set to Self-managed Database. When Connection Type is set to Endpoint, enter the endpoint service name. When Connection Type is set to Public IP, enter the IP address and port number of the database host. When Connection Type is set to Peering Connection, enter the private IP address or hostname and port number of the database host.
      Instance	The ID or name of the instance where the OceanBase database tenant in MySQL compatible mode resides. You can view the ID or name of the target instance on the Instances page. Note When the cloud provider is Alibaba Cloud, cross-account authorized Alibaba Cloud root account instances are also supported. For details, see Alibaba Cloud Account Authorization.
      Tenant	The ID or name of the tenant in MySQL compatible mode of the OceanBase database. You can expand the target instance on the Instances page to view the ID or name of the target tenant under that instance.
      Database Account	The username of the user for the OceanBase database tenant in MySQL compatible mode used for data migration.
      Password	The password of the database user.

      When Instance Type is set to Self-managed Database, you can also decide whether to perform advanced settings based on your actual situation.

      Notice

      If the newly created migration task requires incremental synchronization, you must enable both the sys tenant account and the incremental log proxy service.

      Parameter	Description
      sys Tenant Account	After enabling the sys tenant account, you need to enter the sys account and password. Sys Account: The name of the sys user. This user is mainly used to read incremental logs and database object structure information from OceanBase Database. Please create it under the sys tenant of the business cluster. Password: The password of the sys user.
      Incremental Log Proxy Service	After enabling the incremental log proxy service, you need to fill in the OBLogProxy Connection Information. This parameter is for the OceanBase Database Incremental Log Proxy Service, which provides access and management capabilities for real-time incremental projects as a service, facilitating application access to OceanBase Database incremental logs. It also addresses the need to subscribe to incremental logs when network isolation is present. The format is OBLogProxy IP:OBLogProxy Port.

   2. In the Configure Source & Target step, configure the parameters in the Target section.

      If you need to reference a created and saved data source, click Quick Fill on the right side of Target and select the target data source from the drop-down list. After selection, the configurations in the target area will be automatically populated. If you want to save the current configuration as a new data source, click the Save icon in the upper-right corner of the Target section.

      You can also click Manage Data Source in the Quick Fill drop-down list to go to the Data Source page, where you can view and manage data sources. This page provides unified management for different types of data sources. For more information, see Data sources.

      Parameter	Description
      Cloud Provider	Currently supports Alibaba Cloud, AWS, and Google Cloud. You can choose the same cloud provider as the source or perform cross-cloud data migration. Note The cross-cloud data migration feature is not currently available by default. To use this feature, contact OceanBase Cloud Technical Support.
      Database Type	Select TiDB as the target database type.
      Instance Type	Currently only supports Self-managed TiDB.
      Region	Select the region where the target database is located.
      Connection Type	Includes Endpoint and Public IP. For Alibaba Cloud, AWS, and Google Cloud, you can also select Peering Connection. If you select Endpoint, add the account ID displayed on the page to the allowlist of your Endpoint Service to allow connections from this account's endpoint to the Endpoint Service. For more information, see Select Private Network Connection. When Cloud Vendor is set to AWS and Enable is selected for the Require Acceptance to Use Endpoint parameter when creating the Endpoint Service, the data migration service prompts you to open the AWS console and perform an Accept endpoint connection request operation for the incoming request the first time it accesses the private connection. When the cloud provider is set to Google Cloud, add the authorized project to Published Services. After authorization, no further manual authorization is required when you test the data source connection. If you select Public IP, add the data source IP address displayed on the page to the allowlist of the TiDB database instance to ensure connectivity. For more information, see Select Public Network Connection. Note The data source IP address to be added to the allowlist is displayed on the page only after you have selected the regions for both the source and target. If you select Peering Connection, choose an available peering connection from the drop-down list. Add the IP address displayed on the page to the TiDB instance's inbound security group rules or allowlist to ensure connectivity. For more information, see AWS peering connection, Alibaba Cloud peering connection, and Google Cloud peering connection.
      Peering Connection ID	This parameter is displayed only when Connection Type is set to Peering Connection. Select an available peering connection from the drop-down list. The list displays only connections that match the current cloud provider and region and are in the Connected state. It also displays the corresponding VPC ID and CIDR block. If no connection is available, click Create to open a new tab and create a data service peering connection.
      Connection Information	When Connection Type is set to Endpoint, enter the name of the Endpoint Service. When Connection Type is set to Public IP, enter the IP address and port number of the database host. When Connection Type is set to Peering Connection, enter the private IP address or hostname and port number of the database host.
      Database Account	The username of the TiDB database user used for data migration.
      Password	The password of the database user.

4. Click Test and Continue.

5. On the Select Type & Objects page, configure the parameters.

   Note

   Currently, migrating data from OceanBase Database in MySQL-compatible mode to TiDB Database supports only unidirectional synchronization.

   1. In the Migration Type section, select the migration type for the current data migration task.

      Currently, only Incremental Synchronization and DML Synchronization are supported. For details, see Configure DML/DDL synchronization.

   2. In the Select Migration Objects section, configure the method for selecting migration objects.

      You can select migration objects using two methods: Specify Objects and Match by Rule.

      

   3. In the Select Migration Scope section, select the objects to migrate.

      • When Specify Objects is selected, data migration supports Table-level and Database-level. Database Table Migration allows you to select one or more tables or views within a database as migration objects, while Full Database Migration allows you to select an entire database as the migration object. If Database Table Migration is selected for a database, that database does not support Full Database Migration. If Full Database Migration is selected for a database, that database does not support Database Table Migration.

        After selecting Table-level or Database-level, select the objects you want to migrate from the left and click > to add them to the list on the right.

        Data migration supports importing objects via text, and allows operations such as renaming target objects, setting row filters, viewing column information, and removing single or all migration objects.

        

        Note

        When you select Full Database Migration, only the database name is displayed in the list on the right; specific objects are not supported.

        Operation	Description
        Import Objects	In the right-side list of the selection area, click Import Object in the upper-right corner. For more information, see Import migration objects.
        Rename	Data migration supports renaming migration objects. For more information, see Rename a database or table.
        Row Filter	Data migration supports filtering rows using WHERE conditions. For more information, see Filter data with SQL conditions. You can also view the column information for the migration object in the View Column section.
        Remove/Clear All	Data migration supports removing one or more temporarily selected objects during data mapping to the target. Remove a single migration object In the right-side list of the selection area, click the Remove icon next to the target object to remove it. Remove all migration objects In the right-side list of the selection area, click Clear All in the upper-right corner. In the dialog box, click OK to remove all migration objects.

      • Select Match by Rule. For more information, see Configure database-to-database matching rules.

6. Click Next. On the Migration Options page, configure the parameters for incremental synchronization.

   Parameter	Description
   Write Concurrent Configuration	This parameter specifies the number of concurrent writes to the target during incremental synchronization. The maximum value is 512. A high concurrency level may overwhelm the target and affect business operations.
   Limit Incremental Migration Rate	You can decide whether to enable rate limiting for incremental synchronization based on your actual needs. If enabled, set the target write RPS (the maximum number of data rows that can be synchronized to the target per second during incremental synchronization) and target write BPS (the maximum traffic that can be synchronized to the target per second during incremental synchronization). Note The RPS and BPS values set here are only for throttling. The actual achievable performance during incremental synchronization is subject to factors such as the source, target, and instance specifications.
   Incremental Synchronization Start Timestamp	Specify a point in time after which data should be migrated. The default is the current system time. For more information, see Set an incremental synchronization timestamp.

7. Click Pre-check to have the system perform a precheck on the data migration task.

   During the Pre-check step, data migration checks whether the database user's read and write permissions, network connectivity, and other requirements are met. The data migration task can only be started after all checks pass. If the precheck reports an error:

   • You can troubleshoot and resolve the issue, then re-execute the precheck until it passes.

   • You can also click Skip in the Actions column for the failed precheck item. A dialog box appears, explaining the specific impact of skipping this operation. After confirming you can skip it, click OK in the dialog box.

8. After the precheck passes, click Purchase to go to the Purchase Data Migration Instance page for purchase.

   After successful purchase, you can start the data migration task. For details, see Purchase a data migration instance. If you do not need to purchase a data migration instance immediately, click Save to go to the data migration task details page, where you can manually purchase it later if needed.

   You can click Configure Validation Task in the upper-right corner of the data migration details page to compare data differences between the source and target databases. For details, see Create a data verification task.

   The data migration service supports modifying migration objects while a data migration task is running. For details, see View and modify migration objects. After a data migration task is started, it is executed sequentially according to the selected migration type. For details, see the "View migration details" section in View the details of a data migration task.