System logs (SysLog) refer to the program logs of OceanBase AI Database. These are logs automatically printed during the operation of OceanBase AI Database for monitoring, alerting, and diagnostics. System logs are a powerful complement to monitoring metrics, together supporting the observability capabilities of OceanBase AI Database.
In some scenarios, both system logs and transaction logs are simply referred to as "logs," but they are entirely different entities:
System logs (SysLog): Logs printed during the operation of the OceanBase AI Database process for monitoring, alerting, and diagnostics. This falls under the domain of observability.
Transaction logs (commit log, abbreviated as clog): The write-ahead log (WAL) of OceanBase AI Database that precedes data persistence, used to ensure transactional consistency. This falls under the domain of data consistency.
Log modules
Program logs are divided into different modules, with a hierarchy of parent and child modules.
The parent modules of OceanBase AI Database include
CLIENT,CLOG,COMMON,ELECT,LIB,PROXY,RPC,RS,SERVER,SHARE,SQL,STORAGE, andTLOG.If a parent module contains child modules, it is represented as
parent_module.child_module. For example,SQL.PARSERindicates the PARSER sub-module under the SQL module;SQL.*indicates all sub-modules under the SQL module.
Log files
The log files belonging to the log modules of OceanBase AI Database are divided into five types: observer.log, election.log, rootservice.log, trace.log, and alert.log. By default, logs at or above the INFO level are printed.
Except for the trace.log and alert.log files, each type of log file automatically generates a WARNING log file with the .wf suffix (observer.log.wf, election.log.wf, rootservice.log.wf), which prints only logs at or above the WARN level. Whether to generate WARNING log files is controlled by the cluster-level configuration item enable_syslog_wf.
Log name |
Log path |
|---|---|
Startup and runtime logs (observer.log, observer.log.wf) |
In the $work_dir/log directory of the OBServer node. |
Election logs (election.log, election.log.wf) |
In the $work_dir/log directory of the OBServer node. |
Root Service logs (rootservice.log, rootservice.log.wf) |
In the $work_dir/log directory of the OBServer node. |
End-to-end tracing log (trace.log) |
In the $work_dir/log directory of the OBServer node. |
Alert log (alert.log) |
In the $work_dir/log/alert directory of the OBServer node. |
The size of a single log file in OceanBase AI Database does not exceed 256 MB. When a log file reaches 256 MB, log rotation occurs. The original log file is appended with a timestamp suffix (in the format yyyyMMddHHmmss), which is the generation time of the last log in the file, and a new log file is created. During log rotation, .wf log files are rotated even if they have not reached 256 MB. This means the xxx.log.wf file and the xxx.log file always correspond one-to-one, and the .wf file is generally much smaller than 256 MB.
A typical log file directory is as follows:
log
├── election.log
├── election.log.wf
├── observer.log
├── observer.log.20220427154619
├── observer.log.wf
├── observer.log.wf.20220427154619
├── rootservice.log
├── rootservice.log.20220427165438
├── rootservice.log.wf
├── rootservice.log.wf.20220427165438
└── log
├── alert.log
└── alert.log.20220304102928236
OceanBase AI Database natively supports log archiving (controlled by the cluster-level configuration item enable_syslog_recycle). When the total number of log files reaches max_syslog_file_count, the oldest log file is deleted during log rotation. The maximum space usage for log files can be estimated as follows: max_syslog_file_count * 256M * 3 * (enable_syslog_wf ? 2 : 1).
The native log archiving feature of OceanBase AI Database archives logs based on the number of files. When OceanBase AI Database is deployed together with an external management system (such as OCP), the log archiving feature provided by the external management system, which is based on disk usage, is typically used.
Log format
A typical log line and its format are as follows.
A general log line:
[2022-04-29 16:39:55.186527] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:127) [103594] [0] [Y0-0000000000000000-0-0] [lt=17] [dc=0] max file does not exist(max_file_id=4, b_exist=false)
The fields in the log are described as follows:
- time:
[2022-04-29 16:39:55.186527] - log_level:
WARN - module:
[COMMON] - function:
get_file_id_range - file_name:line_number:
(ob_log_file_group.cpp:127) - thread_id:
[103594] - coroutine_id:
[0] - trace_id:
[Y0-0000000000000000-0-0] - log_used_time:
[lt=17] - dropped_msg_count:
[dc=0] - info:
max file does not exist - parameter:
(max_file_id=4, b_exist=false)
The log format consists of two parts:
- header:
[2022-04-29 16:39:55.186527] WARN [COMMON] get_file_id_range (ob_log_file_group.cpp:127) [103594] [0] [Y0-0000000000000000-0-0] [lt=17] [dc=0] - message:
max file does not exist(max_file_id=4, b_exist=false)
As shown in the preceding format, a log consists of a header and a message body:
Header section:
- time: The time when this log was printed.
- log_level: The level of this log. Currently, the following levels are supported (from high to low): ERROR, USER_ERR, WARN, INFO, TRACE, DEBUG.
- module: The name of the module that prints the log. It consists of a parent module and child modules. OceanBase AI Database supports specifying log levels by module.
- function: The name of the function that prints the log.
- file_name:line_number: The name of the source code file where the log is printed and the line number.
- thread_id: The thread ID that prints the log.
- coroutine_id: The coroutine ID that prints the log.
- trace_id: The trace ID used to track a task, which is unique at the task level. This trace ID is passed between OBServer nodes via RPCs, allowing all log data for a task to be retrieved based on the trace ID. This corresponds to the
TRACE_IDfield in theGV$OB_SQL_AUDITview. - log_used_time: The processing time of the previous log. In asynchronous logging scenarios, this includes the time taken to write the log to a file.
- dropped_msg_count:
dc=xx, wherexxindicates the number of logs discarded between the previous log and the current log. If no logs are discarded, this value is 0.
Message section: The message body generally consists of information (info) and parameters (parameter).
- info: The specific content of the log.
- parameter: A KV list in
name=valueformat. If the value is a primitive type, it is displayed directly. If it is a complex type, it is displayed as text similar to JSON format (to improve readability, the double quotes around keys are removed).