Silent data corruption means that the storage system provides corrupted data to applications, but no alerts are triggered. For example, bad blocks appear in disks due to medium damage. In this case, when an application reads a bad block, the application reads the wrong data.
The probability of silent data corruption is low. The point is that no alert is generated when an error occurs. If an application does not verify the correctness of data, serious exceptions can occur, such as process terminations and data loss. For databases, system stability is essential, and data loss is unacceptable. Data correctness and integrity are the lifeline of database systems.
Mechanisms to avoid silent data corruption
The data stored on disks of OceanBase Database has two parts: REDO logs (transaction logs) and SSTables. REDO logs are replayed to build MemTables in the memory. SSTables store the static data in the log-structured merge-tree (LSM tree). A MemTable and an SSTable form a replica of user data.
Multi-replica mechanism
OceanBase Database is a distributed database service and adopts the multi-replica disaster recovery mechanism. Due to the low probability of silent data corruption, the probability of concurrent silent data corruption of the same data block on multiple replicas is minimal. Silent data corruption detected on a replica can be fixed by copying data from the remaining normal replicas. OceanBase Database allows you to fix silent data corruption at the replica level. If silent data corruption is detected on a replica, O&M engineers can delete this replica and then build a correct replica from other OBServer nodes.
REDO log verification mechanism
The checksum of a REDO log is recorded at its header. The checksum of each log is forcibly verified during network transmission and log replay. This ensures that the logs synchronized to the three replicas are correct and consistent. If the data in a log encounters a silent corruption error, this log is not synchronized to other replicas.
SSTable verification mechanism
SSTable data is stored in a macroblock. A macroblock has a fixed length of 2 MB. The checksum of a macroblock is recorded at its header. A macroblock is split into multiple microblocks with unfixed lengths. The length of a microblock is usually 16 KB. The checksum of a microblock is also recorded at its header. The read I/O operations of SSTables are performed based on microblocks, and the write I/O operations are performed based on macroblocks. When a microblock is read, the checksum stored at its header is forcibly verified to ensure that the data read from the microblock is correct. In macroblock replication scenarios such as migration and backup, the destination also forcibly verifies the checksum of a macroblock before data is written. This ensures that the written data is correct and avoids the spread of silent data corruption.
In addition to checks on the correctness of data blocks during reads and writes, OceanBase Database provides another feature to help users detect silent data corruption at the earliest opportunity. OceanBase Database can enable an inspection task at the backend to periodically scan all macroblocks, verify their checksums, and generate an alert when silent data corruption is detected.
Cold standby mechanism
You may not adopt a deployment mode that involves multiple replicas, and silent data corruption may occur on multiple replicas at a time. To fix data issues in such scenarios, OceanBase Database provides a backup and restore mechanism. Data can be backed up to external media such as Network File System (NFS) and Object Storage Service (OSS). If silent data corruption occurs, correct data can be restored from the external media to your database.