The advanced compression feature of OceanBase AI Database uses a newly designed hybrid row-column storage architecture, efficient data encoding techniques, and a comprehensive set of data compression algorithms. It significantly reduces the storage space required for data compressed with the same backend.
General compression
General compression refers to the process where the compression algorithm compresses data blocks directly without understanding the internal structure of the data. This type of compression typically encodes binary data based on its characteristics to reduce redundancy. Compressed data cannot be accessed randomly; both compression and decompression must be performed in whole data blocks. For block-level compression, OceanBase AI Database supports four compression algorithms: zlib, snappy, lz4, and zstd. zstd and lz4 use a compression level of 1, zlib uses a compression level of 6, and snappy uses its default compression level. In tests conducted within OceanBase AI Database, where microblocks of a default size of 16 KB were compressed, snappy and lz4 showed relatively fast compression speeds but lower compression ratios. zlib and zstd achieved higher compression ratios but slower speeds. lz4 and snappy have similar compression ratios, but lz4 is faster for both compression and decompression. Similarly, zstd has a compression ratio comparable to zlib's, but zstd is faster in both compression and decompression.
OceanBase AI Database allows you to specify either zlib, snappy, lz4, or zstd as the compression algorithm.
Data encoding
Building upon general compression, OceanBase AI Database has developed its own encoding method for hybrid row-column storage. Unlike general compression, encoding compresses data blocks based on the format and semantics of the data in the blocks. OceanBase AI Database is a relational database in which data is organized by table. Each column of a table has a fixed type, which ensures that data in the same column is logically similar. In addition, in some scenarios, data in adjacent rows of a business table is also more similar. Therefore, compressing and storing data by column achieves better compression results. For this reason, OceanBase AI Database introduces microblocks in the encoding format. Unlike microblocks in the flat format, which serialize all data row by row into a block, microblocks in the encoding format use hybrid row-column storage. Logically, a microblock still stores a group of rows, but the data is encoded by column. The encoded fixed-length data is stored in the columnar storage area of the microblock, while some variable-length data is still stored row by row in the variable-length area. In addition, data in microblocks in the encoding format can be accessed randomly. To read a specific row from a microblock, you need to decode only that row, which avoids the computational overhead of decompressing an entire data block just to read part of the data. During vectorized execution, you can also decode only the specified columns, which reduces projection overhead.
During data analysis, OceanBase AI Database selects an appropriate encoding algorithm. Currently, OceanBase AI Database supports multiple data encoding techniques. Some common and effective methods are as follows:
Dictionary encoding
Deduplicate data with a small cardinality, build a dictionary from the deduplicated data, and store references to specific dictionary subscripts in the locations that originally store the data. In addition, the data in the dictionary is sorted by type, which facilitates data compression and allows predicates to be pushed down directly to the dictionary during computing for fast iteration through binary logic.
Run-length encoding (RLE)
For consecutive identical data, such as 100, 100, 100, 120, 120, 120, 150, 150, ..., deduplicate the consecutive data and retain only its starting row number and value.
RLE encoding is commonly used in databases to handle low-cardinality, continuous data, such as index prefixes and suffixes.
Delta encoding
Delta encoding is a numerical encoding method applicable to integer data distributed within a small value range. It calculates the minimum and maximum values in the range, subtracts the minimum value from each data record, and encodes the result with a smaller bit width.
Constant encoding
Constant encoding encodes the most common data value as a constant and records only the values that are not equal to this constant and their row numbers.
In addition, OceanBase AI Database provides a variety of other encoding methods, including prefix encoding, hexadecimal encoding, column equal encoding, and column substring encoding. During major compaction, OceanBase AI Database selects an appropriate encoding method based on data characteristics and calculates the data compression ratio. If a low compression ratio is detected, it quickly rolls back the encoding and selects another encoding method, which ensures that the data encoding process does not affect normal data write performance. If you have a deep understanding of your data characteristics, you can also manually specify the encoding method when creating a table.