Apache Flink is an open-source stream processing framework designed for real-time data streams and batch processing tasks. Its efficient stream processing capabilities complement the Hadoop ecosystem and provide real-time data processing and low-latency data analytics. Flink can process real-time data from various data sources, enabling users to perform complex stream and batch processing tasks within the same data architecture.
This topic describes how to access OBKV-HBase directly by using the Community Edition of Flink without modifying code.
Prerequisites
Before you use Flink to access OBKV-HBase, make sure that the following conditions are met:
- For native Flink access, use OBKV-HBase Java client 2.2.1 or 1.2.1 and later. Version 2.2.1 is recommended. Mainstream Flink versions use HBase 2.x by default.
- When you use the Community Edition of Flink, exclude its built-in HBase version.
- The Flink version is 1.13 to 1.20.
Create a table
When using the Community Edition of Flink to access an OBKV-HBase wide table, you can create the table in one of the following ways:
- Use the HBase-compatible interfaces to create the table.
- Create the table by using SQL. If you use SQL, follow the OBKV-HBase table creation rules. For more information, see Data operation examples.
Set OBKV-HBase client connection parameters
OBKV-HBase client connection parameters are used to establish a connection between the OBKV-HBase client and the OceanBase cluster. The settings of these parameters differ between public cloud and on-premises deployments. For an on-premises OceanBase AI Database cluster, set the parameters as described in Direct connection mode configuration. For a public cloud OBKV service, set the parameters as described in Cloud mode configuration. For detailed instructions on connecting to an OceanBase cluster by using the OBKV-HBase client, see Connect to a cluster by using the OBKV-HBase Java client.
Regardless of the deployment mode, you can set all parameters by using either of the following methods:
- In the
hbase-site.xmlconfiguration file. - By using the
setmethod of the HadoopConfigurationobject before initializing theConnectionobject in your code.
Direct connection mode configuration
Direct connection mode is suitable for on-premises OceanBase AI Database clusters. It connects directly to the OceanBase AI Database cluster by using Configurl, which is used to obtain the real IP address of the underlying OceanBase server.
Notice
The paramURL contains the XML special character &, which must be converted to &; otherwise, parsing fails.
Method 1: Configure the following parameters in the
hbase-site.xmlconfiguration file:<!-- Assume the current cluster is as follows: ClusterName: obkvcluster TenantName: obkv DatabaseName: test UserName: root User password: SYS_USER_NAME: sysroot SYS user password: --> <configuration> <!-- Replace the HBase implementation class with that of OceanBase AI Database --> <property> <name>hbase.client.connection.impl</name> <value>com.alipay.oceanbase.hbase.util.OHConnectionImpl</value> </property> <!-- OceanBase AI Database connection configuration --> <property> <!-- The URL for obtaining the RSlist from the OBConfig server. For details, see the Config URL section below. --> <name>hbase.oceanbase.paramURL</name> <value>PARAM_URL</value> </property> <property> <!-- Format: userName@tenantName#clusterName --> <name>hbase.oceanbase.fullUserName</name> <value>root@obkv#obkvcluster</value> </property> <property> <!-- The password for userName to access OceanBase AI Database --> <name>hbase.oceanbase.password</name> <value></value> </property> <property> <!-- The username under the sys tenant. Only users in the sys tenant have the permission to access the routing table. --> <name>hbase.oceanbase.sysUserName</name> <value>sysroot</value> </property> <property> <!-- The password for the user under the sys tenant. --> <name>hbase.oceanbase.sysPassword</name> <value></value> </property> </configuration>Method 2: Configure parameters in the Hadoop
Configurationobject in your code by using thesetmethod before initializing theConnectionobject:/* * Assume the current cluster is as follows: * ClusterName: obkvcluster * TenantName: obkv * DatabaseName: test * UserName: root * User password: * SYS_USER_NAME: sysroot * SYS user password: */ Configuration conf = HBaseConfiguration.create(); conf.set("hbase.client.connection.impl", "com.alipay.oceanbase.hbase.util.OHConnectionImpl"); conf.set("hbase.oceanbase.fullUserName", "root@obkv#obkvcluster"); conf.set("hbase.oceanbase.password", ""); // Obtain the RSlist URL from the OBConfig server. For details, see Obtain the Config URL. conf.set("hbase.oceanbase.paramURL", ""); conf.set("hbase.oceanbase.sysUserName", "sysroot"); conf.set("hbase.oceanbase.sysPassword", "");
Cloud mode configuration
Cloud mode applies to OceanBase AI Database clusters deployed on public clouds. It connects to the OceanBase AI Database cluster through ODP.
Method 1: Configure the following parameters in the
hbase-site.xmlconfiguration file:<!-- Assume the current cluster is as follows: DatabaseName: test UserName: root --> <configuration> <!-- Replace the HBase implementation class with that of OceanBase AI Database --> <property> <name>hbase.client.connection.impl</name> <value>com.alipay.oceanbase.hbase.util.OHConnectionImpl</value> </property> <!-- OceanBase AI Database connection configuration --> <configuration> <property> <name>hbase.oceanbase.fullUserName</name> <value>root</value> </property> <property> <name>hbase.oceanbase.password</name> <value></value> </property> <property> <!-- For details, see the section below on obtaining the ODP address.--> <name>hbase.oceanbase.odpAddr</name> <value></value> </property> <property> <name>hbase.oceanbase.odpPort</name> <value>3307</value> </property> <property> <name>hbase.oceanbase.odpMode</name> <value>true</value> </property> <property> <name>hbase.oceanbase.database</name> <value>test</value> </property> </configuration>Method 2: Configure parameters in the Hadoop
Configurationobject in your code by using thesetmethod before initializing theConnectionobject:/* * Assume the current cluster is as follows: * DatabaseName: test * UserName: root */ Configuration conf = HBaseConfiguration.create(); conf.set("hbase.client.connection.impl", "com.alipay.oceanbase.hbase.util.OHConnectionImpl"); conf.set(HBASE_OCEANBASE_FULL_USER_NAME, "root"); // User password. conf.set(HBASE_OCEANBASE_PASSWORD, ""); // For details, see Obtain the ODP address. conf.set(HBASE_OCEANBASE_ODP_ADDR, ""); // The port for OBKV is 3307 (fixed). conf.setInt(HBASE_OCEANBASE_ODP_PORT, 3307); // Cloud mode uses ODP mode (fixed). conf.setBoolean(HBASE_OCEANBASE_ODP_MODE, true); // The name of the database. conf.set(HBASE_OCEANBASE_DATABASE, "test");
Usage examples
Create a dimension table
A dimension table is an important concept in data warehousing and data analytics. It stores dimension information that does not change frequently, such as dates, times, locations, and personnel, and provides context and attribute information for fact tables.
The method for creating a dimension table differs between the Community Edition of Flink and the OBKV Flink connector:
- If you use the Community Edition of Flink to access an OBKV-HBase wide table, see HBase table.
- If you use the OBKV Flink connector to access an OBKV-HBase wide table, see Synchronize data to OBKV-HBase by using the Flink connector.
Create a fact table
A fact table is another important concept in data warehousing and data analytics. It stores detailed information and related numerical data for each transaction or operation that occurs in business processes.
The method for creating a fact table differs between the two approaches:
- If you use the Community Edition of Flink to access an OBKV-HBase wide table, see HBase table.
- If you use the OBKV Flink connector to access an OBKV-HBase wide table, see Synchronize data to OBKV-HBase by using the Flink connector.
Add dependencies
Add the OBKV-HBase dependency:
<!-- OBKV-HBase dependency, includes hbase-client and hadoop-common -->
<dependency>
<groupId>com.oceanbase</groupId>
<artifactId>obkv-hbase-client</artifactId>
<version>2.2.1</version>
</dependency>
If you use Flink SQL, do not add the following dependency to avoid conflicts:
<!-- Flink Connector HBase dependency -->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-hbase-2.2</artifactId>
<version>${flink.connector.hbase.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
</exclusion>
</exclusions>
</dependency>
Flink read/write example
This section describes two read and write methods in Flink: DataStream and Flink SQL.
Flink DataStream read and write
Create a sample table group and tables:
-- Create a table group.
CREATE TABLEGROUP `flink_test` SHARDING = 'ADAPTIVE';
-- Create a table.
CREATE TABLE `flink_test$cf1` (
`K` varbinary(1024) NOT NULL,
`Q` varbinary(256) NOT NULL,
`T` bigint(20) NOT NULL,
`V` varbinary(1024) DEFAULT NULL,
PRIMARY KEY (`K`, `Q`, `T`)
) TABLEGROUP = `flink_test` PARTITION BY KEY(`K`) PARTITIONS 17;
CREATE TABLE `flink_test$cf2` (
`K` varbinary(1024) NOT NULL,
`Q` varbinary(256) NOT NULL,
`T` bigint(20) NOT NULL,
`V` varbinary(1024) DEFAULT NULL,
PRIMARY KEY (`K`, `Q`, `T`)
) TABLEGROUP = `flink_test` PARTITION BY KEY(`K`) PARTITIONS 17;
Use Flink DataStream combined with the HBase Java API for read and write operations. A sample read and write operation is as follows:
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
env.fromElements("row1", "row2")
.flatMap(new RichFlatMapFunction<String, String>() {
private transient Table table;
@Override
public void open(org.apache.flink.configuration.Configuration parameters) throws Exception {
Connection connection = ConnectionFactory.createConnection(ObHTableTestUtil.newConfiguration());
table = connection.getTable(org.apache.hadoop.hbase.TableName.valueOf("flink_test"));
}
@Override
public void flatMap(String s, Collector<String> out) throws Exception {
// Write to HBase
Put put = new Put(Bytes.toBytes(s));
put.addColumn(Bytes.toBytes(family1), Bytes.toBytes(column1), Bytes.toBytes("cf1_" + s + "_value"));
put.addColumn(Bytes.toBytes(family2), Bytes.toBytes(column2), Bytes.toBytes("cf2_" + s + "_value"));
table.put(put);
// Read from HBase
Get get = new Get(Bytes.toBytes(s));
Result result = table.get(get);
String family1Value = Bytes.toString(result.getValue(Bytes.toBytes(family1), Bytes.toBytes(column1)));
String family2Value = Bytes.toString(result.getValue(Bytes.toBytes(family2), Bytes.toBytes(column2)));
// Collect the get result.
out.collect(String.format("Get result from cf1: Row: %s, Qualifier: %s, Value: %s", Bytes.toString(result.getRow()), column1, family1Value));
out.collect(String.format("Get result from cf2: Row: %s, Qualifier: %s, Value: %s", Bytes.toString(result.getRow()), column2, family2Value));
}
@Override
public void close() throws Exception {
table.close();
}
}).print();
Flink SQL read and write
Create a sample table group and tables:
CREATE TABLEGROUP `student_info` SHARDING = 'ADAPTIVE';
CREATE TABLEGROUP `student_hobby` SHARDING = 'ADAPTIVE';
CREATE TABLEGROUP `all_student_info` SHARDING = 'ADAPTIVE';
CREATE TABLE `student_info$cf1` (
`K` varbinary(1024) NOT NULL,
`Q` varbinary(256) NOT NULL,
`T` bigint(20) NOT NULL,
`V` varbinary(1024) DEFAULT NULL,
PRIMARY KEY (`K`, `Q`, `T`)
) TABLEGROUP = `student_info` PARTITION BY KEY(`K`) PARTITIONS 17;
CREATE TABLE `student_hobby$cf1` (
`K` varbinary(1024) NOT NULL,
`Q` varbinary(256) NOT NULL,
`T` bigint(20) NOT NULL,
`V` varbinary(1024) DEFAULT NULL,
PRIMARY KEY (`K`, `Q`, `T`)
) TABLEGROUP = `student_hobby` PARTITION BY KEY(`K`) PARTITIONS 17;
CREATE TABLE `all_student_info$cf1` (
`K` varbinary(1024) NOT NULL,
`Q` varbinary(256) NOT NULL,
`T` bigint(20) NOT NULL,
`V` varbinary(1024) DEFAULT NULL,
PRIMARY KEY (`K`, `Q`, `T`)
) TABLEGROUP = `all_student_info` PARTITION BY KEY(`K`) PARTITIONS 17;
When you create a table mapping for OBKV-HBase, the zookeeper.quorum parameter is not required, but it must be set to a non-empty string, which can be any string. Flink SQL simulates dimension table operations by writing data to the student_info and student_hobby tables. Here is an example:
-- Create a mapping between the Flink table and the HBase table `student_info`.
CREATE TABLE student_info (
id INT,
cf1 ROW<name STRING, age INT>,
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'connector' = 'hbase-2.2',
'table-name' = 'student_info',
'zookeeper.quorum' = 'any_string'
);
-- Create a mapping between the Flink table and the HBase table `student_hobby`.
CREATE TABLE student_hobby (
id INT,
cf1 ROW<hobby STRING>,
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'connector' = 'hbase-2.2',
'table-name' = 'student_hobby',
'zookeeper.quorum' = 'any_string'
);
-- Insert data into student_info and student_hobby.
INSERT INTO student_info VALUES
(1001, ROW('Lily', 13)),
(1002, ROW('John', 12)),
(1003, ROW('David', 13));
INSERT INTO student_hobby VALUES
(1001, ROW('basketball')),
(1002, ROW('badminton')),
(1003, ROW('jogging'));
Use Flink SQL to read data from the student_info and student_hobby tables, merge the results, and write them to the all_student_info table:
-- Create a mapping between the Flink table and the HBase table `all_student_info`.
CREATE TABLE all_student_info (
id INT,
cf1 ROW<name STRING, age INT, hobby STRING>,
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'connector' = 'hbase-2.2',
'table-name' = 'all_student_info',
'zookeeper.quorum' = 'any_string'
);
-- Insert data into all_student_info.
INSERT INTO all_student_info
SELECT
si.id AS id,
ROW(si.cf1.name, si.cf1.age, sh.cf1.hobby) AS cf1
FROM
student_info AS si
LEFT JOIN
student_hobby AS sh ON si.id = sh.id;