This topic describes the server-side and client-side monitoring metrics of OBKV-HBase, including metric descriptions and usage.
Server-side monitoring metrics
You can query the following views for server-side monitoring metrics of OBKV-HBase:
View |
Description |
|---|---|
| GV$SYSSTAT | Query the QPS, RT, and ROWS metrics of OBKV-HBase. |
| GV$OB_SQL_AUDIT | Locate slow queries in OBKV-HBase. |
| GV$OB_QUERY_RESPONSE_TIME_HISTOGRAM | Query performance metrics such as P99 of OBKV-HBase. |
Specifically:
QPS/RT/ROWS-related metrics
You can query the QPS, RT, and ROWS metrics from the GV$SYSSTAT view.
Query all monitoring metrics
MySQL [oceanbase]> SELECT DISTINCT class,stat_id,name, value_type FROM GV$SYSSTAT WHERE name LIKE "hbase%";
A sample return result is as follows. The return result includes all HBase monitoring metrics. For the complete list, see the parameter description table below:
+-------+---------+------------------------------+------------+
| class | stat_id | name | value_type |
+-------+---------+------------------------------+------------+
| 4096 | 191001 | hbase scan count | ADD_VALUE |
+-------+---------+------------------------------+------------+
(Recommended) Query specific monitoring metrics by tenant
A sample query is as follows:
# Query the checkAndMutate metrics.
MySQL [oceanbase]> SELECT * FROM oceanbase.GV$SYSSTAT
WHERE name LIKE '%hbase check_mutate%'
AND con_id = (SELECT tenant_id FROM oceanbase.DBA_OB_TENANTS WHERE tenant_name = 'your_tenant_name'); -- Specify the tenant name.
The supported name parameters and their descriptions are as follows:
Name |
Description |
|---|---|
| hbase scan count | The number of scan operations. |
| hbase scan time | The total time spent on scan operations. |
| hbase scan row count | The number of cells scanned by scan operations. |
| hbase put count | The number of put operations. |
| hbase put time | The total time spent on put operations. |
| hbase put row count | The number of cells in put operations. |
| hbase delete count | The number of delete operations. |
| hbase delete time | The total time spent on delete operations. |
| hbase delete row count | The number of cells deleted. |
| hbase append count | The number of append operations. |
| hbase append time | The total time spent on append operations. |
| hbase append row count | The number of cells in append operations. |
| hbase increment count | The number of increment operations. |
| hbase increment time | The total time spent on increment operations. |
| hbase increment row count | The number of cells affected by increment operations. |
| hbase check_put count | The number of checkAndPut operations. |
| hbase check_put time | The total time spent on checkAndPut operations. |
| hbase check_put row count | The number of cells in checkAndPut operations. |
| hbase check_delete count | The number of checkAndDelete operations. |
| hbase check_delete time | The total time spent on checkAndDelete operations. |
| hbase check_delete row count | The number of cells in checkAndDelete operations. |
| hbase check_mutate | The number of checkAndMutate operations. This metric must be used with OBKV-Table V2.3.0 and OBKV-HBase V1.5.0/V2.5.0 or later clients. |
| hbase scan | The number of scan operations. This metric is required when the OBKV-Table V2.3.0 client is used with the OBKV-HBase V1.5.0 or V2.5.0 client. |
| hbase get | The number of get operations. This metric must be used with OBKV-Table V2.3.0 and OBKV-HBase V1.5.0/V2.5.0 or later clients. |
| hbase batch put | The number of batch put operations. This metric takes effect when the client is OBKV-Table V2.3.0 and OBKV-HBase V1.5.0 or later. |
| hbase batch delete | The number of batch delete operations. This metric takes effect only when the client is OceanBase Key-Value-Table V2.3.0 and OceanBase Key-Value-HBase V1.5.0 or V2.5.0 or later. |
| hbase batch get | The number of batch get operations. This metric must be used with OBKV-Table V2.3.0 and OBKV-HBase V1.5.0/V2.5.0 or later clients. |
| obkv failed operation | The number of failed OBKV operations. This metric is reported when the OBKV-Table V2.3.0 client is used with the OBKV-HBase V1.5.0 or V2.5.0 client. |
For more information about other monitoring metrics, see GV$SYSSTAT.
Locate slow HBase requests
SQL Audit is an important diagnostic tool in OceanBase AI Database. It records diagnostic information about each request, such as the cache hit rate, the number of rows scanned in the MemTable/SSTable, and the execution time. Based on this information, you can easily identify the bottleneck of a slow request. OBKV-HBase supports SQL Audit. You can execute the following statement to query diagnostic records related to OBKV-HBase:
MySQL [oceanbase]> SELECT * FROM v$OB_SQL_AUDIT WHERE query_sql LIKE "table api: hbase%" \G;
A sample return result is as follows:
*************************** 1. row ***************************
SVR_IP: 11.xxx.xxx.xxx
SVR_PORT: 54302
REQUEST_ID: 13841
SQL_EXEC_ID: 0
TRACE_ID: Y1EB2D1EB6D984-0000000000000002-0-0
SID: 0
CLIENT_IP: 30.xxx.xxx.xxx
CLIENT_PORT: 60205
TENANT_ID: 1
TENANT_NAME: sys
EFFECTIVE_TENANT_ID: 1
USER_ID: 200001
USER_NAME: root
USER_GROUP: 0
USER_CLIENT_IP: 30.xxx.xxx.xxx
DB_ID: 500001
DB_NAME: test
SQL_ID: TABLEAPI0x1103vv00000000CB87E8C9
QUERY_SQL: table api: hbase_put,
request: {credential:0193808080000101C19A0CA1C21E00FABAF9E6C0B589AEDB01, table_name:"htable1$family", table_id:500014, tablet_id:{id:200055}, entity_type:2, batch_operation:{is_readonly:false, is_same_type:true, is_same_properties_names:true, operatiton_count:1, table_operations:[{operation_type:4, entity:"{rowkey_names:[]rowkey:[{"VARCHAR":"testKey0", collation:"utf8mb4_general_ci", coercibility:"EXPLICIT"}, {"VARCHAR":"column", collation:"utf8mb4_general_ci", coercibility:"EXPLICIT"}, {"BIGINT":-9223372036854775807}], properties:{V:{"VARCHAR":"putValue", collation:"utf8mb4_general_ci", coercibility:"EXPLICIT"}}}"}]}, consistency_level:0, option_flag:0, returning_affected_entity:false, returning_affected_rows:true, batch_operation_as_atomic:false}
For more information about how to locate other slow requests, see GV$OB_SQL_AUDIT.
Performance metrics such as P99
OBKV-HBase collects execution time statistics for each operation and presents them in histograms. You can use the V$OB_QUERY_RESPONSE_TIME_HISTOGRAM view together with Prometheus to view performance metrics such as P95 and P99 for each operation.
A sample query is as follows:
# Query the checkAndMutate metrics.
MySQL [oceanbase]> SELECT * FROM oceanbase.GV$OB_QUERY_RESPONSE_TIME_HISTOGRAM
WHERE sql_type LIKE '%HBASE CHECK AND MUTATE%'; -- Specify the operation type.
For more information about the supported sql_type values and other details, see GV$OB_QUERY_RESPONSE_TIME_HISTOGRAM.
Client monitoring metrics
This section describes how to use MetricsExporter and jconsole of OBKV-HBase to view client operation monitoring metrics.
Obtain operation monitoring metrics by using MetricsExporter
MetricsExporter includes the following metrics:
Metric |
Description |
|---|---|
| FailRate | The average failure rate of the corresponding interface. |
| FailCount | The number of failures of the corresponding interface. |
| TotalRuntime | The total execution time of the corresponding API. |
| AverageSingleOpCount | The number of operations involved in the corresponding API. For batch operations, this is the length of the batch operation list. |
| Count | The number of API calls. |
| AverageOps | The average OPS of the corresponding interface. |
| OneMinuteAverageOps | The average OPS of the corresponding interface in 1 minute. |
| FiveMinuteAverageOps | The average OPS of the corresponding interface in 5 minutes. |
| FifteenMinuteAverageOps | The average OPS of the corresponding interface in 15 minutes. |
| AverageLatency | The average RT of the corresponding interface. |
| MaxLatency | The maximum RT of the corresponding interface. |
| MinLatency | The minimum RT of the corresponding interface. |
| MedianLatency | The median RT of the corresponding interface. |
| P75thPercentile | The P75 RT of the corresponding interface. |
| P95thPercentile | The P95 RT of the corresponding interface. |
| P98thPercentile | The P98 RT of the corresponding interface. |
| P99thPercentile | The P99 RT of the corresponding interface. |
| P999thPercentile | The P999 RT of the corresponding interface. |
Usage
// Sample code.
Configuration conf = ObHTableTestUtil.newConfiguration();
// Enable client-side metrics.
conf.set(CLIENT_SIDE_METRICS_ENABLED_KEY, "true");
Connection connection = ConnectionFactory.createConnection(conf);
Table hTable = connection.getTable(TableName.valueOf("test_metrics"));
OHMetrics metrics = ((OHTable) hTable).getMetrics();
// GET_LIST metrics.
MetricsExporter exporter = metrics.acquireMetrics(OHOperationType.GET_LIST);
System.out.println("AverageOps: " + exporter.getAverageOps());
System.out.println("99thPercentile: " + exporter.get99thPercentile());
System.out.println("AverageSingleOpCount: " + exporter.getAverageSingleOpCount());
View client monitoring metrics by using JMX
You can view client monitoring metrics in either of the following ways:
- View them by using MBeanServer.
- View them by using jconsole.
View by using MBeanServer
The following sample code shows how to view client monitoring metrics by using MBeanServer:
// Sample code.
import javax.management.*;
import java.lang.management.ManagementFactory;
MBeanServer mBeanServer = ManagementFactory.getPlatformMBeanServer();
/**
* Get the ObjectName of a specific attribute in JMX.
*
* @param opType The OHOperationType.
* @param metricsName The metrics name.
* @param attributeName The attribute name (latencyHistogram, failedOpCounter, totalSingleOpCount, or totalRuntime).
* @return The ObjectName.
*/
public ObjectName getObjectName(OHOperationType opType, String metricsName,
String attributeName) {
try {
// The JMX name format is domain:name=metricName.
// Example: com.alipay.oceanbase.hbase.util.OHMetrics.PUT.latencyHistogram.metricsName
String name = String.format("%s.%s.%s.%s",
OHMetrics.class.getName(),
opType.name(),
attributeName,
metricsName);
String objectNameStr = String.format("%s:name=%s",
JMX_DOMAIN, name);
return new ObjectName(objectNameStr);
} catch (MalformedObjectNameException e) {
throw new RuntimeException("Failed to create ObjectName", e);
}
}
/**
* Get the latency attribute.
* Timer attributes: Count, MeanRate, OneMinuteRate, FiveMinuteRate, FifteenMinuteRate,
* Min, Max, Mean, StdDev, 50thPercentile, 75thPercentile, 95thPercentile, 98thPercentile, 99thPercentile, 999thPercentile
*/
public Object getTimerAttribute(OHOperationType opType, String metricsName, String attributeName) {
try {
ObjectName objectName = getObjectName(opType, metricsName, "latencyHistogram");
return mBeanServer.getAttribute(objectName, attributeName);
} catch (Exception e) {
throw new RuntimeException("Failed to get Timer attribute: " + attributeName, e);
}
}
/**
* Get the failed operations attribute.
* Meter attributes: Count, MeanRate, OneMinuteRate, FiveMinuteRate, FifteenMinuteRate
*/
public Object getMeterAttribute(OHOperationType opType, String metricsName, String attributeName) {
try {
ObjectName objectName = getObjectName(opType, metricsName, "failedOpCounter");
return mBeanServer.getAttribute(objectName, attributeName);
} catch (Exception e) {
throw new RuntimeException("Failed to get Meter attribute: " + attributeName, e);
}
}
/**
* Get the Counter attribute.
* Counter attributes: Count
*/
public Object getCounterAttribute(OHOperationType opType, String metricsName,
String counterName, String attributeName) {
try {
ObjectName objectName = getObjectName(opType, metricsName, counterName);
return mBeanServer.getAttribute(objectName, attributeName);
} catch (Exception e) {
throw new RuntimeException("Failed to get Counter attribute: " + attributeName, e);
}
}
Configuration config = ObHTableTestUtil.newConfiguration();
// Enable client-side metrics.
config.set(CLIENT_SIDE_METRICS_ENABLED_KEY, "true");
Connection connection = ConnectionFactory.createConnection(config);
Table hTable = connection.getTable(TableName.valueOf("test_multi_cf"));
String metricsName;
if (hTable instanceof OHTable) {
OHMetrics metrics = ((OHTable) hTable).getMetrics();
if (metrics != null) {
metricsName = metrics.getMetricsName();
}
}
// PUT metrics
// Latency histogram.
Long totalOpCount = (Long) getTimerAttribute(OHOperationType.PUT, metricsName, "Count");
Double meanOps = (Double) getTimerAttribute(OHOperationType.PUT, metricsName, "MeanRate");
Double oneMinuteRateOps = (Double) getTimerAttribute(OHOperationType.PUT, metricsName, "OneMinuteRate");
Double meanLatency = (Double) getTimerAttribute(OHOperationType.PUT, metricsName, "Mean");
Double minLatency = (Double) getTimerAttribute(OHOperationType.PUT, metricsName, "Min");
Double maxLatency = (Double) getTimerAttribute(OHOperationType.PUT, metricsName, "Max");
Double P99thPercentile = (Double) getTimerAttribute(OHOperationType.PUT, metricsName, "99thPercentile");
// Failed operations.
Long failedOpCount = (Long) getMeterAttribute(OHOperationType.PUT, metricsName, "Count");
Double meanFailRate = (Double) getMeterAttribute(OHOperationType.PUT, metricsName, "MeanRate");
Double oneMinuteFailRate = (Double) getMeterAttribute(OHOperationType.PUT, metricsName, "OneMinuteRate");
// Counter
Double totalSingleOpCount = (Long) getCounterAttribute(OHOperationType.PUT, metricsName, "totalSingleOpCount", "Count");
Double totalRuntime = (Long) getCounterAttribute(OHOperationType.PUT, metricsName, "totalRuntime", "Count");
View by using jconsole
Set CLIENT_SIDE_METRICS_ENABLED_KEY to true in the Hadoop Configuration, and then start the OBKV-HBase client process. Run ps -ef | grep idea in the terminal to view the process PID. Run jconsole $PID in the terminal to open the JMX interface. Under MBeans, the com.oceanbase.hbase.metrics folder contains the monitoring items for all interface operations.