With the Flink connector for OBKV-HBase, you can write data from Flink to OceanBase AI Database by using the obkv-hbase-client-java library.
Getting started
You can find the official release version on the Releases page or in the Maven Central repository.
<dependency>
<groupId>com.oceanbase</groupId>
<artifactId>flink-connector-obkv-hbase</artifactId>
<version>${project.version}</version>
</dependency>
To use the latest snapshot version, configure the Maven snapshot repository as follows.
<dependency>
<groupId>com.oceanbase</groupId>
<artifactId>flink-connector-obkv-hbase</artifactId>
<version>${project.version}</version>
</dependency>
<repositories>
<repository>
<id>sonatype-snapshots</id>
<name>Sonatype Snapshot Repository</name>
<url>https://s01.oss.sonatype.org/content/repositories/snapshots/</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
You can also obtain the package by building it from source.
git clone https://github.com/oceanbase/flink-connector-oceanbase.git
cd flink-connector-oceanbase
mvn clean package -DskipTests
SQL JAR
To use this connector directly with Flink SQL, you need to download a JAR file named flink-sql-connector-obkv-hbase-${project.version}.jar, which contains all dependencies.
Examples
Create a table named htable1$family1 in OceanBase AI Database. In the table name, htable1 is the HBase table name, and family1 is the column family in HBase.
USE test;
CREATE TABLE `htable1$family1`
(
`K` varbinary(1024) NOT NULL,
`Q` varbinary(256) NOT NULL,
`T` bigint(20) NOT NULL,
`V` varbinary(1048576) NOT NULL,
PRIMARY KEY (`K`, `Q`, `T`)
);
Java application example
The following example uses a Maven project. Add the required dependencies to the pom.xml file of the application, and then use the following code.
package com.oceanbase;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.api.EnvironmentSettings;
import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
public class Main {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment().setParallelism(1);
StreamTableEnvironment tEnv =
StreamTableEnvironment.create(
env, EnvironmentSettings.newInstance().inStreamingMode().build());
tEnv.executeSql(
"CREATE TABLE t_sink ( "
+ " rowkey STRING,"
+ " family1 ROW<column1 STRING, column2 STRING>,"
+ " PRIMARY KEY (rowkey) NOT ENFORCED"
+ ") with ("
+ " 'connector'='obkv-hbase',"
+ " 'url'='http://127.0.0.1:8080/services?...',"
+ " 'schema-name'='test',"
+ " 'table-name'='htable1',"
+ " 'username'='root@test',"
+ " 'password'='',"
+ " 'sys.username'='root',"
+ " 'sys.password'=''"
+ ");");
tEnv.executeSql(
"INSERT INTO t_sink VALUES "
+ "('1', ROW('r1f1c1', 'r1f1c2')),"
+ "('2', ROW('r2f1c1', 'r2f1c2')),"
+ "('2', ROW('r3f1c1', 'r3f1c2'));")
.await();
}
}
After execution, you can query and verify the results in OceanBase AI Database.
For more information, see OBKVHBaseConnectorITCase.java.
Flink SQL example
Place the required dependency JAR files in the lib directory of Flink. Then, create the destination table in Flink by using the SQL client.
CREATE TABLE t_sink
(
rowkey STRING,
family1 ROW <column1 STRING,
column2 STRING >,
PRIMARY KEY (rowkey) NOT ENFORCED
) with (
'connector'='obkv-hbase',
'url'='http://127.0.0.1:8080/services?...',
'schema-name'='test',
'table-name'='htable1',
'username'='root@test',
'password'='',
'sys.username'='root',
'sys.password'='');
Insert test data.
INSERT INTO t_sink
VALUES ('1', ROW ('r1f1c1', 'r1f1c2')),
('2', ROW ('r2f1c1', 'r2f1c2')),
('2', ROW ('r3f1c1', 'r3f1c2'));
After execution, you can query and verify the results in OceanBase AI Database.
Configuration parameters
Parameter |
Required |
Default value |
Type |
Description |
|---|---|---|---|---|
| url | Yes | String | The configuration URL of the cluster. You can obtain it by executing SHOW PARAMETERS LIKE 'obconfig_url'. |
|
| schema-name | Yes | String | The database name of OceanBase AI Database. | |
| table-name | Yes | String | The HBase table name. Note that in OceanBase AI Database, the table name format is hbase_table_name$family_name. |
|
| username | Yes | String | The username of a non-sys tenant. | |
| password | Yes | String | The password of a non-sys tenant. | |
| sys.username | Yes | String | The username of the sys tenant. | |
| sys.password | Yes | String | The password of the sys tenant user. | |
| hbase.properties | No | String | The attributes of the obkv-hbase-client-java configuration.Separate multiple values with semicolons. |
|
| sync-write | No | false | Boolean | Specifies whether to enable synchronous write. If this parameter is set to true,data is written directly to the database without using a buffer. |
| buffer-flush.interval | No | 1s | Duration | The buffer flush interval. If this parameter is set to '0',periodic flushing is disabled. |
| buffer-flush.buffer-size | No | 1000 | Integer | The buffer size. |
| max-retries | No | 3 | Integer | The number of retries on failure. |