SeaTunnel is a distributed and highly scalable data integration platform for transferring data between various data storage systems. SeaTunnel provides easy-to-configure data flow task plugins, allowing users to define the processes of data handling and transformation. For more information about SeaTunnel, refer to SeaTunnel's GitHub page.
Prerequisites
Before migrating data, confirm the following information:
- You have downloaded SeaTunnel.
- You have downloaded the MySQL driver into the lib directory.
- You have installed Java 1.8 or a higher version.
- You have enabled Binlog for OceanBase Cloud MySQL.
This part introduces how to execute SQL queries in OceanBase Cloud MySQL-compatible tenant by using SeaTunnel's MySQL CDC Source Connector. connector-cdc-mysql is a plugin of SeaTunnel, used for capturing and streaming real-time change data (Change Data Capture, CDC) in MySQL databases.
Example
Follow these steps to perform data replication between OceanBase Cloud MySQL-compatible tenants:
Enable the Binlog service in OceanBase Cloud.
The method to enable the Binlog logging service is: Instances -> Tenants -> Binlog Service, click to activate. For more information, see Enable Binlog Service.
Create user for SeaTunnel.
CREATE USER 'username'@'%' IDENTIFIED BY 'password'; GRANT SELECT ON *.* TO 'username';Configure
config/plugin_config.Set
connector-cdc-mysql, runsh bin/install-plugin.sh 2.3.3.Configure SeaTunnel.
modify
config/v2.streaming.conf.template:source { # This is a example source plugin **only for test and demonstrate the feature source plugin** MySQL-CDC { result_table_name = "fake" parallelism = 1 server-id = 5656 username = "username" password = "password" table-names = ["seatunnel.full_types"] base-url = "jdbc:mysql://xx.xxx.x.xx:3306/seatunnel" } # If you would like to get more information about how to configure SeaTunnel and see full list of source plugins, # please go to https://seatunnel.apache.org/docs/category/source-v2 }Start SeaTunnel.
Run this command to start SeaTunnel:
./bin/seatunnel.sh --config ./config/v2.streaming.conf.template -e local
Notice
OceanBase Cloud does not support the SET datatype.
Caused by: org.apache.seatunnel.api.table.catalog.DataTypeConvertException: ErrorCode:[COMMON-07], ErrorDescription:[Unsupported data type] - Convert type: SET to SeaTunnel data type error.
For more information, see SeaTunnel MySQL CDC Documentation.
This part introduces how to synchronize data using SeaTunnel's JDBC OceanBase Source Connector and JDBC OceanBase Sink Connector. The JDBC OceanBase Source Connector supports full read and write on a single table data in OceanBase. For more information, see JDBC OceanBase Source Connector. JDBC OceanBase Sink Connector supports full read and write on multi-table data in OceanBase. For more information, see JDBC OceanBase Sink Connector.
Example
Follow these steps to perform data replication between OceanBase Cloud MySQL-compatible tenants:
Enabled the Binlog service in OceanBase Cloud.
The path to enable the Binlog logging service is: Instances -> Tenants -> Binlog Service, click to activate. For more information, refer to Enable Binlog Service.
Create user for SeaTunnel.
CREATE USER 'username'@'%' IDENTIFIED BY 'password'; GRANT SELECT ON *.* TO 'username';Configure
config/plugin_config.Download OceanBase JDBC driver to
libdirectory. For more information, see OceanBase Client.Configure SeaTunnel.
Modify
config/ob_to_ob:env { parallelism = 1 job.mode = "BATCH" } source { # This is a example source plugin **only for test and demonstrate the feature source plugin** Jdbc { driver = com.oceanbase.jdbc.Driver url = "jdbc:oceanbase://e2e_oceanbase_mysql:3306/seatunnel?useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true&serverTimezone=UTC" user = "username" password = "password" query = "SELECT c_bit_1, c_bit_8 FROM source" compatible_mode = "mysql" } # If you would like to get more information about how to configure seatunnel and see full list of source plugins, # please go to https://seatunnel.apache.org/docs/connector-v2/source/FakeSource } sink { Jdbc { driver = com.oceanbase.jdbc.Driver url = "jdbc:oceanbase://e2e_oceanbase_mysql:3306/seatunnel?useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true&serverTimezone=UTC" user = "username" password = "password" query = "insert into sink(c_bit_1, c_bit_8) values (?, ?);" compatible_mode = "mysql" } # If you would like to get more information about how to configure seatunnel and see full list of sink plugins, # please go to https://seatunnel.apache.org/docs/connector-v2/sink }Start SeaTunnel.
Run this command to start SeaTunnel:
./bin/seatunnel.sh --config ./config/ob_to_ob -e local
Notice
OceanBase Cloud does not support the SET datatype.
Caused by: org.apache.seatunnel.api.table.catalog.DataTypeConvertException: ErrorCode:[COMMON-07], ErrorDescription:[Unsupported data type] - Convert type: SET to SeaTunnel data type error.