OceanBase Database Community Edition V4.0.0

OceanBase Database Community Edition V4.0.0 is a beta version that comprehensively upgrades the distributed database architecture. OceanBase pays close attention to user feedback and keeps refining this product. Before you put this product online for production, you must cautiously review and evaluate its performance.

In addition to all legacy features, OceanBase Database Community Edition V4.0 reviews fundamental designs of the database and distributed systems and provides the first integrated architecture in the industry, highlighting hybrid transaction/analytical processing (HTAP) and on-cloud deployment. Architecture bottlenecks of OceanBase Database Community Edition V3.2 are eliminated in this version. This allows OceanBase Database Community Edition to support more highly expected core capabilities for business processing, with a significant improvement in kernel functionality, compatibility, stability, and performance.

Overview

This version provides the following core capabilities:

• Architecture upgrade and benefits

  An integrated architecture that supports both standalone deployment and distributed deployment is adopted to provide essential capabilities, such as adaptive log streams, the processing of ultra-large transactions, a recovery time objective (RTO) of less than 8s, reduced dependency on the network time protocol (NTP) service, and a higher upper limit for the number of partitions.

• Enhanced kernel capabilities

  Online DDL operations are enhanced. More character sets are supported. The maximum allowable size of the LOB data type is increased. In addition, tenant-level backup, data encoding, IOPS isolation, table lock, and deadlock detection are supported.

• Enhanced compatibility

  Foreign key constraints of DDL statements are supported. The column information of a view can be displayed. More SQL modes and functions are supported. In addition, DML triggers, sequence objects, stored programs, and prepared SQL statements are supported. An auto-increment column can be used as a partitioning key.

• Significantly enhanced performance

  • Sysbench performance is improved. Compared with OceanBase Database Community Edition V3.1, the overall concurrent read/write performance of this version is doubled in running the Sysbench benchmark with 1,024 threads.

  • TPC-H query performance is improved. Compared with OceanBase Database Community Edition V3.1, the performance of this version is improved by five times in executing 22 sequential queries in the TPC-H benchmark with a 100 GB dataset.

• Enhanced O&M capabilities

  End-to-end tracing is supported. The session status can be monitored and diagnosed based on the Active Sessions History (ASH) report. Standard views are optimized. The recycling of schema history files can be configured. The recycle bin can be automatically purged.

New features

Architecture upgrade and benefits

• Adaptive log stream

  In the OceanBase Database architecture of an earlier version, a partition is the elementary unit of operation. When the number of partitions in the system reaches a specified value, the resource consumption of partition-based operations also increases, which causes some issues. For example, the number of partitions supported on a single node is limited. If multiple partitions are involved in the data modification on a single node, the two-phase commit protocol must be used to ensure the atomicity of the transaction.

  A log stream is an entity that is automatically created and managed by OceanBase Database. It is a collection of data and contains several partitions and ordered REDO logs. In the new system architecture, transaction modification logs of all partitions in a unit can be recorded in a log stream. The log stream then synchronizes the modifications to the corresponding units in other zones. Each unit of an OceanBase tenant has a corresponding log stream. The system fixes this relationship between a log stream and its corresponding partition. This relationship is changed only when the unit is migrated.

• Ultra large transactions

  Based on the new adaptive log stream feature, the transaction engine is redesigned to solve multiple transaction-related issues of most distributed databases, such as large transactions, a large number of participants, and lagging transaction commits. The new transaction engine stably handles various tasks such as online transactions, batch processing, and data correction. It ensures database reliability in complex business scenarios.

• RTO within 8s

  OceanBase Database Community Edition V4.0 uses a new automatic leader selection protocol and a comprehensive keepalive mechanism. This further reduces the RTO to less than 8s and minimizes the business impact in the event of a server failure, ensuring high business availability.

• NTP service optimization

  The new automatic leader selection protocol no longer depends on the NTP clock. In earlier versions, the clock offset among all OBServers has to be controlled within 100 ms. OceanBase Database Community Edition V4.0 allows a clock offset of up to 2s and supports dynamic clock modification without affecting data accuracy or cluster stability.

• Partition optimization

  The new adaptive log stream allows all partitions in a unit to share a resource group. A partition no longer needs to separately apply for resource reservation as in earlier versions, hence the higher utilization of system resources. You no longer need to specify the maximum number of partitions on an OBServer based on the configurations. However, the maximum number of partitions is still limited by the available physical resources of a server.

Enhanced kernel capabilities

• Enhanced online DDL operations

  The OceanBase Database architecture of an earlier version supports limited types of online DDL statements. For example, primary key modifications are not supported, which causes inconvenience in database use. Thanks to its integrated architecture, OceanBase Database Community Edition V4.0.0 supports more online DDL operations related to data migration, including:

  • Add, modify, and delete a primary key.

  • Modify the name and type of a column, including the conversion between the character, numeric, date, and other data types.

  • Convert a normal table to a partitioned table.

  • Modify the collation of a table or column.

• Backup of tenants

  Multitenancy is one of the core features of OceanBase Database. Most users choose to create multiple tenants in one cluster, with each tenant supporting a business unit. Therefore, fine-grained backup strategies are required for the backup of tenants at different frequencies based on the type and importance of the business unit. OceanBase Database Community Edition V4.0 supports backup at the tenant level, which is a finer granularity than backup at the cluster level and allows you to back up a tenant and recover the backup data in a new tenant. In addition, the backup snapshot retention strategies are optimized to reduce the impact of backup process on disk space. Data backups and log backups are stored in different directories, which allows you to separately store them on media that differ in performance.

  For more information about tenant backup, see Backup and restore.

• Character sets

  The following character sets are supported: UTF8, UTF16, GBK, GB18030, and BINARY. The following collations are supported: UTF8MB4_BIN, UTF16_BIN, GBK_BIN, GB18030_BIN, UTF8MB4_GENERAL_CI, UTF16_GENERAL_CI, GBK_CHINESE_CI, and GB18030_CHINESE_CI.

• Vectorized engine

  OceanBase Database Enterprise Edition V3.2.3 provides a vectorized engine with comprehensive features by transforming all operators and most execution expressions based on the architecture-aware design to make full use of the cache and optimization instructions of modern CPUs. The engine is applied in running the TPC-H benchmark. Vectorization makes it possible to optimize algorithms and data structures by using a plethora of methods. Compared with its non-vectorized cousins, the vectorized execution engine shows 4 to 5 times higher overall performance, and even 10 times higher performance for certain operators and in simple scenarios. This version opens the source code of the OceanBase vectorized engine to help you achieve a higher online analytical processing (OLAP) performance.

• Data encoding algorithms

  OceanBase Database achieves a high compression ratio by using data encoding-based compression, which is an important technology to help you reduce storage costs. This version opens the source code of a variety of data encoding algorithms, such as dictionary encoding, run-length encoding (RLE), constant encoding, delta encoding, prefix encoding, and span-column encoding. It can also automatically select the most appropriate encoding algorithm for each column. OceanBase Database uses the encoding (compression) technology, microblocks of the same size (16 KB), and the unified compression algorithm (LZ4) to process and store data. The same copy of data stored in OceanBase Database requires less than half of the space required for storage in a MySQL 5.7 database, without affecting the performance.

• IOPS isolation

  In OceanBase Database Community Edition of earlier versions, CPU and memory resources are isolated between tenants. OceanBase Database Community Edition V4.0 supports IOPS isolation between tenants. Disk and bandwidth for different tenants are independent of each other. This avoids I/O resource contention between tenants and implements complete tenant resource isolation.

  The available resources of a tenant are bound to units. You can configure resource specifications of a unit in unit configs. The following parameters are related to IOPS resource isolation: MIN_IOPS, MAX_IOPS, and IOPS_WEIGHT. Modifications to the IOPS parameters in unit configs take effect in real time. In addition, you can specify the percentages of I/O requests of various types, such as business and system log requests, by setting the io_category_config parameter. This allows you to perform fine-grained control over I/O resource allocation and scheduling.

• LOB specifications

  In OceanBase Database Community Edition of earlier versions, the storage size of LOB data types cannot exceed 48 MB. In OceanBase Database Community Edition V4.0, an LOB macroblock is split into multiple pieces of LOB metadata on the storage layer. When the data is retrieved, these segments of LOB metadata are aggregated into a continuous buffer and sent to the SQL layer for processing. This way, the maximum allowable storage size of the LOB data type is increased to 512 MB. OceanBase Database will support an LOB macroblock that is capable of storing terabytes of data in a later version.

• Table locks and deadlock detection

  A table lock allows you to lock a table or partition in a specified manner to avoid data corruption caused by concurrent operations on the table or partition. OceanBase Database Community Edition V4.0 supports more online DDL operations. Therefore, the table lock feature is provided to ensure data accuracy in concurrent DDL and DML operations. The LOCK TABLE syntax is supported in SHARE and EXCLUSIVE lock modes. Deadlocks due to lock contention can be detected.

Enhanced compatibility with MySQL

• Foreign key constraints on DDL statements

  By default, OceanBase Database checks foreign keys. To disable or enable foreign key check, modify the foreign_key_checks tenant variable. A foreign key constraint requires that the constrained column must contain only values from the primary key column of another table. In OceanBase Database Community Edition of earlier versions, the foreign_key_checks tenant variable takes effect only on DML operations. In OceanBase Database Community Edition V4.0, the foreign_key_checks tenant variable also takes effect on DDL operations in the same way as in a MySQL database.

• Column information of views

  In a MySQL database, the column information of tables and views is stored as metadata in a data dictionary. You can query the INFORMATION_SCHEMA.COLUMNS view to display the column information. OceanBase Database persistently stores only the column information of tables. To address this issue, OceanBase Database Community Edition V4.0 avoids parsing complex view dependencies by dynamically parsing view definitions and displays the column information of views when you query the INFORMATION_SCHEMA.COLUMNS view.

• SQL modes

  In MySQL databases, SQL modes are enabled by default. The following three SQL modes are supported in OceanBase Database Community Edition: NO_ZERO_DATE, ERROR_FOR_DIVISION_BY_ZERO, and NO_AUTO_CREATE_USER, in the same way as in a MySQL database. Only the syntax of NO_ENGINE_SUBSTITUTION is supported.

• Prepared statements

  Prepared statements are supported. The API for prepared statements uses a binary protocol and the execution efficiency of the prepared statements is higher than those that use an interactive SQL API. Examples:

  • Prepare a statement: PREPARE stmt1 FROM 'SELECT SQRT(POW(?,2) + POW(?,2))';

  • Execute the prepared statement: SET @a = 3; SET @b = 4; EXECUTE stmt1 USING @a, @b;

  • Release the prepared statement: DEALLOCATE PREPARE stmt1;

• An auto-increment column as a partitioning key

  Examples:

  create table t2(inv_id bigint not null auto_increment ,c1 bigint, primary key (inv_id) ) partition by hash(inv_id) partitions 8;
  

  Note that when you use an auto-increment column as a partition key in OceanBase Database, the values in the auto-increment column are globally unique, but they may not always be incremental in a partition. This is different from the behavior of a native MySQL database. Compared with partitioned tables that use other partitioning methods, the performance of insert operations on a table partitioned by using an auto-increment column can be lower.

• Sequence objects

  Sequence objects are supported to meet the dependency requirements of business systems and reduce the complexity of adaptation during business migration. The CREATE SEQUENCE, ALTER SEQUENCE, and DROP SEQUENCE statements are supported. You can call the CURRVAL and NEXTVAL functions to calculate the value of a sequence, and reset sequence values. Valid values range from INT64_MIN to INT64_MAX.

• Stored programs

  OceanBase Database Community Edition V4.0 supports stored programs that are compatible with the syntax of MySQL 5.7, including stored procedures and stored functions. Cursors, process-control statements, exception handling, DDL operations related to stored programs, and the view and status queries are supported. More system packages, such as DBMS_STATS and DBMS_SESSION, are supported.

• DML triggers

  A trigger can be created on a table and is compatible with the syntax of MySQL 5.6. When the specified DML operation is performed on the table, the specified actions are triggered.

• Functions

  • The ADDTIME() function is supported. You can call this function to add a time interval to the specified date and time. Example: SELECT ADDTIME('2007-12-31 23:59:59.999999', '1 1:1:1.000002');

  • The DAYNAME() function is supported. You can call this function to return the weekday of the specified date. Example: SELECT DAYNAME('2018-01-8');

  • The BIT_AND(), BIT_OR(), and BIT_XOR() aggregate functions are supported. You can call these functions to return the bitwise AND, OR, and XOR operation results of expressions.

  • The UUID_SHORT() function is supported. You can call the function to return a 64-bit unsigned integer. Example: SELECT UUID_SHORT();

Enhanced O&M capabilities

• End-to-end tracing

  OceanBase Database has been put into business operation for a long time. Its internal data access links are very complex. When a timeout error occurs, it is often impossible to quickly locate the root cause. Therefore, an experienced O&M engineer is required to troubleshoot each process, which is hardly responsive and affects the O&M efficiency. OceanBase Database Community Edition V4.0 provides the end-to-end tracing mechanism to help you locate errors in all processes of the whole link, from business applications to client drivers (JDBC or OCI), ODPs, and OBServers. You can specify the MODULE, ACTION, CLIENT_INFO, and CLIENT_IDENTIFIER parameters for applications by using PL statements or OBClient to identify the link in use. An O&M engineer can use the PL program package to control whether to enable end-to-end tracing and diagnosis of the specified identification metrics and set diagnosis information output strategies. Diagnosis logs are generated and stored in ODP and OBServer log files in the OpenTracing data model. The engineer can parse the diagnosis logs to obtain relevant diagnosis information such as the execution time of each SQL and transaction requests in the whole link.

• Session monitoring and diagnosis

  OceanBase Database Community Edition of an earlier version allows you to obtain the status information of an SQL statement that is being executed, such as the wait events. However, you can obtain only the last wait event in a session by querying the __all_virtual_session_wait table. OceanBase Database Community Edition V4.0 supports a comprehensive ASH report that describes the relationship between sessions and wait events. The report contains not only the status of the SQL statements being executed, but also the status history of multiple metrics, such as SESSION, USER, SQL, and WaitEvent. The ASH report collects the status of all active sessions in the system once every 1s. The collection process does not involve row locking or affect the execution of SQL statements. OceanBase Cloud Platform (OCP) analyzes the collected status information to help you learn about the system load and wait events in the past period of time, and identify exceptions in time.

• Real-time execution plan monitoring

  OceanBase Database provides SQL plan monitoring as a part of the SQL execution plan diagnosis feature. After an SQL statement is executed, you can query the GV$SQL_PLAN_MONITOR tenant view for information such as the logical execution plan, physical execution plan, the number of rows returned by an operator, and start and end time of the operator. OceanBase Database Community Edition V4.0 supports real-time SQL plan monitoring, which allows you to view the status of each SQL execution plan of the current tenant in real time. When the SQL execution is stuck, the real-time SQL plan monitoring feature also allows you to query the operator execution status of each execution thread.

• Recycling of schema history files

  The schema history files can be recycled. This feature is provided because too many schema history files can slow down the start of OBServers. The recycle interval of schema history files is specified by the hidden parameter _schema_history_recycle_interval. The default value is 0, which specifies to disable the schema history recycle feature.

• Automatic recycle bin purging

  OceanBase Database provides a recycle bin for each tenant. When the disk space available is insufficient, the recycle bin is automatically purged based on a first-in-first-out (FIFO) strategy. The recyclebin_object_expire_time parameter is added to specify the expiration time of objects in the recycle bin.

Performance report

Test environment specifications

Version for test

Sysbench OLTP benchmark

Test plan:

1. Use OceanBase Deployer (obd) to deploy an OceanBase cluster. Deploy OceanBase Database Proxy (ODP) and Sysbench on a separate server for stress testing.

2. Deploy the cluster in the 1-1-1 architecture, where the cluster has three zones and each zone has one OBServer. After successful deployment, create the tenant and user required for running the Sysbench benchmark. The sys tenant is a built-in system tenant used for managing the cluster. Do not use the sys tenant to run the benchmark. Set primary_zone to RANDOM for the tenant, which indicates that the leader of the new table partitions is randomly assigned to one of the three servers.

3. Launch the Sysbench client and run the point_select, read_write, read_only, and write_only tests.

4. Set the --time parameter to 60s for each round of test. The number of threads can be 32, 64, 128, 256, 512, or 1,024.

5. A total of 30 non-partitioned tables are used. Each table contains 1,000,000 rows of data. This means that the --tables parameter is set to 30 and --table_size is set to 1000000.

Tenant specifications:

• MAX_CPU = 26

• MEMORY_SIZE = 70 GB

Results:

Point select performance

Write-only performance

Write Only performance

Read/Write performance

BenchmarkSQL TPC-C benchmark

Test plan:

1. Use obd to deploy an OceanBase cluster. Deploy ODP and the TPC-C tools on a separate server to avoid inadequate stress on the same server.

2. For the three-node OceanBase cluster, deploy the cluster in the 1-1-1 architecture, where the cluster has three zones and each zone has one OBServer. After successful deployment, create the tenant and users required for running the TPC-C benchmark. The sys tenant is a built-in system tenant for managing the cluster. Do not use the sys tenant to run the benchmark. Set primary_zone to RANDOM for the tenant.

Tenant specifications:

• MAX_CPU = 26

• MEMORY_SIZE = 70 GB

Software version:

• mysql-connector-java-5.1.47

• Benchmark SQL V5.0

Test configuration:

• warehouse = 1000

• loadWorder=40

• terminals=800

• runMins=5

• newOrderWeight=45

• paymentWeight=43

• orderStatusWeight=4

• deliveryWeight=4

• stockLevelWeight=4

Results:

TPC-H benchmark

Test plan:

• Use obd to deploy OceanBase clusters. Deploy the TPC-H client on a server for stress testing. You do not need to deploy an ODP. Directly connect to a machine during the test.

• For the three-node OceanBase cluster, deploy the cluster in the 1-1-1 architecture, where the cluster has three zones and each zone has one OBServer. After successful deployment, create the tenant and users required for running the TPC-H benchmark. The sys tenant is a built-in system tenant for managing the cluster. Do not use the sys tenant to run the benchmark. Set primary_zone to RANDOM for the tenant.

• Size of the tested dataset: 100 GB.

Tenant specifications:

• MAX_CPU = 26

• MEMORY_SIZE = 70 GB

Results (in seconds):

Compatibility changes

Product behavioral changes

Views, entity tables, and virtual tables

• OceanBase Database Community Edition V4.0 provides many self-managed data dictionary views and dynamic performance views. A data dictionary view displays the data or combined data in an entity table, such as an internal table. Instead of directly querying an internal table, you can query the data dictionary view. A dynamic performance view provides information about the memory status during system operation. Instead of directly querying a virtual table, you can query the dynamic performance view. Examples:

  • The DBA_OB_SERVERS view provides information about servers of a cluster. After you add or delete servers by running the ADD SERVER or DELETE SERVER command, the number of servers displayed in the view will change. This view also displays information in fields such as WITH_ROOTSERVER, STATUS, and BUILD_VERSION.

  • The GV$OB_UNITS view displays dynamic performance information about units of tenants on each server. You can query this view for the latest information about the units of the Meta tenant and the units of the hidden sys tenant on each server. The units of the Meta tenant are created by the system.

• Entity tables and virtual tables are redesigned in OceanBase Database Community Edition V4.0. Examples:

  • The __all_tenant_partition_meta_table table is an entity table. It records metadata reported from partitions in partition groups. OceanBase Database Community Edition V4.0 no longer supports partition groups. Therefore, the table is deleted.

  • The __all_table_v2 table is an entity table. It records multi-version schemas of tables. In OceanBase Database Community Edition V4.0, the table name is changed to __all_table.

  • The __all_virtual_core_meta_table table is a virtual table. It displays the location of the replicas of the __all_core_table table of the sys tenant. The table_id, partition_id, and partition_cnt fields are deleted, and the ls_id field is added in OceanBase Database Community Edition V4.0.

For more information about the views, see System views.

Parameter changes

In this upgraded architecture, many parameters no longer have effect. Invalid parameters are deleted and changed in OceanBase Database Community Edition V4.0. For example, the minor_freeze_times parameter that specifies the threshold for triggering a major compaction is deleted. The default values of some parameters are adjusted. For example, the freeze_trigger_percentage parameter specifies the maximum memory usage for triggering a global freeze. OceanBase Database Community Edition V4.0 better supports large transaction processing, and transactions are no longer affected by the freeze operation. Therefore, the default value of this parameter is set to 20%.

Note

• Online upgrade from OceanBase Database Community Edition V3.x to V4.0 is not supported.

• You can upgrade OceanBase Database Community Edition V3.x to V4.0 through logical data migration by using OceanBase Migration Service (OMS).

Recommended versions of tools

The following table describes the recommended versions of components supported in OceanBase Database V4.0.0.

Limits

• OceanBase Database Community Edition V4.0.0 does not support data load balancing after the replica is scaled out. This means that the existing data cannot be distributed to the added nodes. We plan to support the load balancing feature in OceanBase Database Community Edition V4.1.

• OceanBase Database Community Edition V4.0.0 does not support the primary/secondary database deployment mode. We plan to support this mode in OceanBase Database Community Edition V4.1.

• OceanBase Database Community Edition V4.0.0 does not support table replication. This feature will be supported in later versions.

• OceanBase Database Community Edition V4.0.0 does not support converting data from the INT type to the BIGINT type. This feature will be supported in later versions.

• OceanBase Database Community Edition V4.0.0 supports data backup only from the primary cluster. Data backup from a standby cluster will be supported in later versions.

• OceanBase Database Community Edition V4.0.0 requires a minimum specification of 4 CPU cores and 16 GB of memory for an OBServer in the production environment. We recommend that you allocate at least 4 GB of memory and 2 CPU cores for a tenant.

• OceanBase Database Community Edition V4.0.0 supports only full-featured replicas. Other types of replicas such as read-only replicas will be supported in later versions.

• OceanBase Database Community Edition V4.0.0 allows you to increase the number of units by setting the UNIT_NUM parameter. The decrease in the number of units will be supported in later versions.