Apache Spark is a fast, open-source, general-purpose big data processing engine designed for batch and stream processing of large-scale datasets. It plays a core role in the Hadoop ecosystem and provides more efficient data processing capabilities than the traditional MapReduce model.
This topic describes how to access OBKV-HBase directly by using Spark without modifying your code.
Prerequisites
Before you use Spark to access OBKV-HBase, make sure that the following conditions are met:
- The OBKV-HBase Java client is version 2.2.1 or later, which supports native Spark capabilities.
- When you add Spark-HBase dependencies, exclude the built-in HBase version to avoid conflicts with the HBase version that the OBKV-HBase Java client depends on.
- The Spark version is earlier than 3.0.1.
Create a table
When using Spark to access an OBKV-HBase wide table, you can create the table in one of the following ways:
- Use the HBase-compatible interfaces to create the table.
- Create the table by using SQL. If you create the table by using SQL, follow the OBKV-HBase table creation rules. For more information, see Data operation examples.
Set OBKV-HBase client connection parameters
The OBKV-HBase client connection parameters are used to establish a connection between the OBKV-HBase client and the OceanBase AI Database cluster. The method for setting these parameters differs between public cloud and on-premises deployments. For an on-premises OceanBase AI Database cluster, set the parameters as described in Direct connection mode configuration. For a public cloud OBKV service, set them as described in Cloud mode configuration. For tutorials on connecting to an OceanBase cluster by using the OBKV-HBase client, see Connect to a cluster by using the OBKV-HBase Java client.
Regardless of the deployment mode, you can set all parameters in either of the following ways:
- In the
hbase-site.xmlconfiguration file of the client. - In your code, by using the
setmethod of HadoopConfigurationbefore initializing theConnectionobject.
Direct connection mode configuration
Direct connection mode applies to on-premises OceanBase AI Database clusters. It connects directly to the OceanBase AI Database cluster by using Configurl, which is used to obtain the real address of the underlying OceanBase AI Database servers. The following describes how to configure connection parameters in Spark for direct connection mode:
Notice
The paramURL contains the XML special character &, which must be converted to &; otherwise, parsing will fail.
Method 1: Configure the following parameters in the
hbase-site.xmlconfiguration file:<!-- Assume the current cluster is as follows: ClusterName: obkvcluster TenantName: obkv DatabaseName: test UserName: root User password: SYS_USER_NAME: sysroot SYS user password: --> <configuration> <!-- Replace the HBase implementation class with that of OceanBase AI Database --> <property> <name>hbase.client.connection.impl</name> <value>com.alipay.oceanbase.hbase.util.OHConnectionImpl</value> </property> <!-- OceanBase AI Database connection configuration --> <property> <!-- The URL for obtaining the RSlist from the OBConfig server. For details, see Obtain the Config URL. --> <name>hbase.oceanbase.paramURL</name> <value>PARAM_URL</value> </property> <property> <!-- Format: userName@tenantName#clusterName --> <name>hbase.oceanbase.fullUserName</name> <value>root@obkv#obkvcluster</value> </property> <property> <!-- The password for userName to access OceanBase AI Database --> <name>hbase.oceanbase.password</name> <value></value> </property> <property> <!-- The username under the sys tenant. Only users in the sys tenant have permission to access the routing table. --> <name>hbase.oceanbase.sysUserName</name> <value>sysroot</value> </property> <property> <!-- The password for the user under the sys tenant. --> <name>hbase.oceanbase.sysPassword</name> <value></value> </property> </configuration>Method 2: Configure parameters by using the Hadoop
Configurationobject in your code. Call thesetmethod before initializing theConnectionobject:/* * Assume the current cluster is as follows: * ClusterName: obkvcluster * TenantName: obkv * DatabaseName: test * UserName: root * User password: * SYS_USER_NAME: sysroot * SYS user password: */ Configuration conf = HBaseConfiguration.create(); conf.set("hbase.client.connection.impl", "com.alipay.oceanbase.hbase.util.OHConnectionImpl"); conf.set("hbase.oceanbase.fullUserName", "root@obkv#obkvcluster"); conf.set("hbase.oceanbase.password", ""); // Obtain the RSlist URL from the OBConfig server. For details, see Obtain the Config URL. conf.set("hbase.oceanbase.paramURL", ""); conf.set("hbase.oceanbase.sysUserName", "sysroot"); conf.set("hbase.oceanbase.sysPassword", "");
Obtain the Config URL:
You can obtain the Config URL by using one of the following methods:
Obtain the Config URL from OCP
If you use OCP, you can obtain the Config URL as follows:
a. Log on to the OceanBase AI Database console. b. In the left navigation pane, choose Clusters, scroll down to Cluster List, and click the name of the cluster that you want to access. c. On the cluster details page, find the ConfigURL parameter. You need to specify this parameter during client initialization.

Deploy a Config Server by using OBD and obtain
ObRootServiceInfoUrlIf you do not use OCP, you must deploy a Config Server by using OBD and obtain
ObRootServiceInfoUrl.
Cloud mode configuration
Cloud mode applies to OceanBase AI Database clusters deployed on public clouds. It connects to the OceanBase AI Database cluster through ODP. The following describes how to configure connection parameters for Spark in cloud mode:
Method 1: Configure the following parameters in the
hbase-site.xmlconfiguration file:<!-- Assume the current cluster is as follows: DatabaseName: test UserName: root --> <configuration> <!-- Replace the HBase implementation class with that of OceanBase AI Database --> <property> <name>hbase.client.connection.impl</name> <value>com.alipay.oceanbase.hbase.util.OHConnectionImpl</value> </property> <!-- Configure the OceanBase AI Database connection --> <property> <name>hbase.oceanbase.fullUserName</name> <value>root</value> </property> <property> <name>hbase.oceanbase.password</name> <value></value> </property> <property> <!-- For details, see Obtain the ODP address. --> <name>hbase.oceanbase.odpAddr</name> <value></value> </property> <property> <name>hbase.oceanbase.odpPort</name> <value>3307</value> </property> <property> <name>hbase.oceanbase.odpMode</name> <value>true</value> </property> <property> <name>hbase.oceanbase.database</name> <value>test</value> </property> </configuration>Method 2: Configure parameters by using the Hadoop
Configurationobject in your code. Call thesetmethod before initializing theConnectionobject:/* * Assume the current cluster is as follows: * DatabaseName: test * UserName: root */ Configuration conf = HBaseConfiguration.create(); conf.set("hbase.client.connection.impl", "com.alipay.oceanbase.hbase.util.OHConnectionImpl"); conf.set(HBASE_OCEANBASE_FULL_USER_NAME, "root"); // User password. conf.set(HBASE_OCEANBASE_PASSWORD, ""); // For details, see Obtain the ODP address. conf.set(HBASE_OCEANBASE_ODP_ADDR, ""); // The port for OBKV is 3307 (fixed). conf.setInt(HBASE_OCEANBASE_ODP_PORT, 3307); // Cloud mode uses ODP mode (fixed). conf.setBoolean(HBASE_OCEANBASE_ODP_MODE, true); // The name of the database. conf.set(HBASE_OCEANBASE_DATABASE, "test");
Usage examples
Add dependencies
Add the following OBKV-HBase dependency:
<!-- OBKV-HBase dependency, which includes hbase-client and hadoop-common -->
<dependency>
<groupId>com.oceanbase</groupId>
<artifactId>obkv-hbase-client</artifactId>
<version>2.2.1</version>
</dependency>
To ensure seamless integration between OBKV-HBase and Spark, we recommend that you use Spark 3.0.1 and the hbase-spark dependency. Note the following when you add Spark dependencies:
- When you add the
hbase-sparkdependency, exclude thehbase-clientandhadoop-commondependencies that come with Spark to avoid dependency conflicts with OBKV-HBase. - When you add the OBKV-HBase dependency, exclude the
jackson-databinddependency to avoid conflicts with the version used by Spark, which may cause version detection to fail.
<!-- OBKV-HBase dependency -->
<dependency>
<groupId>com.oceanbase</groupId>
<artifactId>obkv-hbase-client</artifactId>
<version>${hbase.client.version}</version>
<exclusions>
<exclusion>
<artifactId>jackson-databind</artifactId>
<groupId>com.fasterxml.jackson.core</groupId>
</exclusion>
</exclusions>
</dependency>
<!-- Spark dependency -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.12</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- Spark and HBase connector dependency -->
<dependency>
<groupId>org.apache.hbase.connectors.spark</groupId>
<artifactId>hbase-spark</artifactId>
<version>${hbase.spark.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
</exclusion>
</exclusions>
</dependency>
Spark read and write examples
The following examples show how to read data from and write data to OBKV-HBase by using Spark, including the table structure, data read and write, and data read by using Spark SQL.
Table structure
Assume that the following OBKV-HBase table exists:
-- Create a table group.
CREATE TABLEGROUP `spark_demo` SHARDING = 'ADAPTIVE';
-- Create a table.
CREATE TABLE `spark_demo$cf1` (
`K` varbinary(1024) NOT NULL,
`Q` varbinary(256) NOT NULL,
`T` bigint(20) NOT NULL,
`V` varbinary(1024) DEFAULT NULL,
PRIMARY KEY (`K`, `Q`, `T`)
) TABLEGROUP = `spark_demo` PARTITION BY KEY(`K`) PARTITIONS 17;
CREATE TABLE `spark_demo$cf2` (
`K` varbinary(1024) NOT NULL,
`Q` varbinary(256) NOT NULL,
`T` bigint(20) NOT NULL,
`V` varbinary(1024) DEFAULT NULL,
PRIMARY KEY (`K`, `Q`, `T`)
) TABLEGROUP = `spark_demo` PARTITION BY KEY(`K`) PARTITIONS 17;
Read and write data by using Hadoop TableInputFormat/TableOutputFormat and Spark RDD
You can read and write data by combining Hadoop TableInputFormat and TableOutputFormat with Spark RDDs. Because Spark uses the Java API of OBKV-HBase to read and write data, you do not need to configure hbase.zookeeper.quorum. This method makes full use of the distributed capabilities of OceanBase AI Database.
// Write data
public void testSparkReadAfterWriteHBase() throws Exception {
// Initialize SparkContext
SparkConf sparkConf = new SparkConf().setAppName("HBaseWriteExample").setMaster("local");
JavaSparkContext jsc = new JavaSparkContext(sparkConf);
// Configure HBase for write
Configuration hbaseConf = HBaseConfiguration.create();
hbaseConf.set(TableOutputFormat.OUTPUT_TABLE, "spark_demo");
hbaseConf.set("mapreduce.job.output.key.class", ImmutableBytesWritable.class.getName());
hbaseConf.set("mapreduce.job.output.value.class", Put.class.getName());
hbaseConf.set("mapreduce.outputformat.class", TableOutputFormat.class.getName());
// Create RDD data (Key: RowKey, Value: Tuple of (ColumnFamily, ColumnName, Value))
JavaRDD<Tuple2<byte[], Tuple3<String, String, String>>> data = jsc.parallelize(Arrays.asList(
new Tuple2<>(Bytes.toBytes(row1), new Tuple3<>(family1, "name", john_name)),
new Tuple2<>(Bytes.toBytes(row1), new Tuple3<>(family1, "age", john_age)),
new Tuple2<>(Bytes.toBytes(row1), new Tuple3<>(family1, "gender", john_gender)),
new Tuple2<>(Bytes.toBytes(row2), new Tuple3<>(family1, "name", lily_name)),
new Tuple2<>(Bytes.toBytes(row2), new Tuple3<>(family1, "age", lily_age)),
new Tuple2<>(Bytes.toBytes(row2), new Tuple3<>(family1, "gender", lily_gender)),
new Tuple2<>(Bytes.toBytes(row3), new Tuple3<>(family1, "name", walter_name)),
new Tuple2<>(Bytes.toBytes(row3), new Tuple3<>(family1, "age", walter_age)),
new Tuple2<>(Bytes.toBytes(row3), new Tuple3<>(family1, "gender", walter_gender))
));
// Map RDD to HBase format (ImmutableBytesWritable, Put)
JavaPairRDD<ImmutableBytesWritable, Put> hbaseWriteRDD = data.mapToPair(tuple -> {
byte[] rowKey = tuple._1;
String family = tuple._2._1();
String qualifier = tuple._2._2();
String value = tuple._2._3();
Put put = new Put(rowKey);
put.addColumn(Bytes.toBytes(family), Bytes.toBytes(qualifier), Bytes.toBytes(value));
return new Tuple2<>(new ImmutableBytesWritable(rowKey), put);
});
// Write the data to HBase
hbaseWriteRDD.saveAsNewAPIHadoopDataset(hbaseConf);
// Read from HBase
hbaseConf.set(TableInputFormat.INPUT_TABLE, "spark_demo");
Scan scan = new Scan();
scan.addFamily(Bytes.toBytes(family1));
scan.addFamily(Bytes.toBytes(family2));
scan.setCaching(100);
String scanToString = org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.convertScanToString(scan);
hbaseConf.set(TableInputFormat.SCAN, scanToString);
JavaPairRDD<ImmutableBytesWritable, Result> hbaseReadRDD = jsc.newAPIHadoopRDD(
hbaseConf,
TableInputFormat.class,
ImmutableBytesWritable.class,
Result.class
);
// Process the data
hbaseReadRDD.flatMapToPair(tuple -> {
ImmutableBytesWritable key = tuple._1();
Result result = tuple._2();
String rowKey = Bytes.toString(key.get());
List<Tuple2<String, String>> res = new ArrayList<>();
for (Cell c : result.rawCells()) {
String familyValue = Bytes.toString(CellUtil.cloneValue(c));
String column = Bytes.toString(CellUtil.cloneQualifier(c));
Tuple2<String, String> value = new Tuple2<>(rowKey, String.format("Get result from cf1: K: %s, Q: %s, V: %s", Bytes.toString(result.getRow()), column, familyValue));
res.add(value);
}
return res.iterator();
}).collect().forEach(System.out::println);
// Stop SparkContext
jsc.stop();
}
Read data by using SparkSession SQL
Spark SQL supports only reading data from HBase. OBKV-HBase does not support SparkSQLPushDownFilter yet. To disable this filter, set the SparkSession parameter hbase.spark.pushdown.columnfilter to false. The default value is true. Example:
public void testSparkSessionSQLHBase() throws Exception {
SparkConf conf = new SparkConf().setAppName("SparkSQLHBaseExample").setMaster("local[*]");
conf.set("spark.sql.defaultUrlStreamHandlerFactory.enabled", "false"); // Avoid a connection casting exception.
JavaSparkContext jsc = new JavaSparkContext(conf);
// Initialize SparkSession
SparkSession spark = SparkSession.builder()
.appName("SparkSQLHBaseExample")
.config(conf)
.getOrCreate();
// HBase configuration
Configuration hbaseConf = ObHTableTestUtil.newConfiguration();
// Initialize HBaseContext
String tmpHdfsConfigPath = "/your-hdfs-path/tmp/hbase-config";
HBaseContext hbaseContext = new HBaseContext(jsc.sc(), hbaseConf, tmpHdfsConfigPath);
String tableName = "spark_demo";
String hbaseColumnMapping = "id STRING :key, " +
"name STRING cf1:name, " +
"gender STRING cf1:gender, " +
"age STRING cf1:age";
Dataset<Row> hbaseDF = spark.read()
.format("org.apache.hadoop.hbase.spark")
.option("hbase.columns.mapping", hbaseColumnMapping)
// Disable column pushdown optimization. OBKV-HBase does not support SparkSQLPushDownFilter yet.
.option("hbase.spark.pushdown.columnfilter", false)
.option("hbase.table", tableName)
.load();
hbaseDF.printSchema();
// Query all data.
hbaseDF.show();
// Run SQL queries on the DataFrame.
hbaseDF.createOrReplaceTempView(tableName);
// Query with a condition.
Dataset<Row> result = spark.sql("SELECT * FROM spark_demo WHERE age > 30");
result.show();
jsc.stop();
spark.stop();
}