obdiag gather log|V2.5.0|OceanBase Diagnostic Tool |docs|OceanBase Documentations

If OceanBase Diagnostic Tool (obdiag) is independently deployed, you can run the obdiag gather log command to collect logs of the specified OceanBase cluster.

If your OceanBase cluster is deployed by using OceanBase Deployer (obd), you can run obdiag commands on obd to collect diagnostic information of the cluster. For more information about the commands, see obdiag commands.

Syntax

obdiag gather log [options]

The following table describes the options.

Option	Required?	Data type	Default value	Description
--from	No	string	Empty	The start time of log collection in the `yyyy-mm-dd hh:mm:ss` format.
--to	No	string	Empty	The end time of log collection in the `yyyy-mm-dd hh:mm:ss` format.
--since	No	string	Empty	The most recent period of time for which logs are to be collected, in the `<n> <m\|h\|d>` format, where `n` is a number, `m` indicates minutes, `h` indicates hours, and `d` indicates days. For example, `30m` specifies to collect logs of the last 30 minutes.
--scope	No	string	all	The type of logs of the OceanBase cluster to be collected. Valid values: `observer`, `election`, `rootservice`, and `all`.
--grep	No	string	Empty	The search keyword.
--encrypt	No	string	false	Specifies whether to encrypt the returned files. Valid values: `true` and `false`.
--store_dir	No	string	The current path where the command is executed	The local path where the results are stored.
--temp_dir	No	string	`/tmp`	The directory where temporary files generated by the remote node during log collection are stored.
--redact	No	string	Empty	The data masking rule, for example, `all_sql`. If multiple rules are specified, separate them with commas (`,`). For more information about how to define a data masking rule, see the Define a data masking rule section in this topic. Note The data masking results are saved in a .zip file, which is not encrypted. You can manually encrypt it.
-c	No	string	`~/.obdiag/config.yml`	The path of the configuration file.
--inner_config	No	string	Empty	The configurations of obdiag.
--config	No	string	Empty	The configurations of the cluster diagnosed by obdiag, in the format of `--config key1=value1 --config key2=value2`.

Examples

Method 1: Use obdiag out-of-the-box without a configuration file

Note

For more information about the parameters used in the command in this section, see Configure obdiag.

Collect logs of the specified period

obdiag gather log --scope observer --from "2022-06-30 16:25:00" --to "2022-06-30 18:30:00" --grep STORAGE --encrypt true \
    --config obcluster.servers.nodes[0].ip=xx.xx.xx.1 \
    --config obcluster.servers.nodes[1].ip=xx.xx.xx.xx.2 \
    --config obcluster.servers.global.ssh_username=test \
    --config obcluster.servers.global.ssh_password=****** \
    --config obcluster.servers.global.home_path=/home/admin/oceanbase

The output is as follows:

...
ZipFileInfo:
+-------------------+-----------+
| Node              | LogSize   |
+===================+===========+
| xxx.xxx.xxx.xxx   | 36.184M   |
+-------------------+-----------+
...

ZipFileInfo:
+-------------------+-----------+
| Node              | LogSize   |
+===================+===========+
| xxx.xxx.xxx.xxx   | 44.176M   |
+-------------------+-----------+
...

Summary:
+-------------------+-----------+----------+------------------+--------+-------------------------------------------------------------------------------+
| Node              | Status    | Size     | Password         | Time   | PackPath                                                                      |
+===================+===========+==========+==================+========+===============================================================================+
| xxx.xxx.xxx.xxx   | Completed | 36.762M  | HYmVourcUyRNP8Om | 19 s   | obdiag_gather_pack_20220701183246/ob_log_xxx.xxx.xxx.xxx_20220701183247.zip   |
+-------------------+-----------+----------+------------------+--------+-------------------------------------------------------------------------------+
| xxx.xxx.xxx.xxx   | Completed | 638.200M | 1RicMaiLUUNfemnj | 718 s  | obdiag_gather_pack_20220701183246/ob_log_xxx.xxx.xxx.xxx_20220701183918.zip   |
+-------------------+-----------+----------+------------------+--------+-------------------------------------------------------------------------------+

Notice

If encryption is enabled, collected logs are encrypted when being compressed. The value in the Password column indicates the password for decrypting the .zip package. Encryption is disabled by default.

Collect logs of a recent period

# Collect logs of the last hour with the target nodes accessible to obdiag without a password.
obdiag gather log --since 1h \
  --config obcluster.servers.nodes[0].ip=xx.xx.xx.1 \
  --config obcluster.servers.nodes[1].ip=xx.xx.xx.xx.2 \
  --config obcluster.servers.nodes[2].ip=xx.xx.xx.xx.3 \
  --config obcluster.servers.global.home_path=/home/admin/oceanbase

# Collect logs of the last 30 minutes with the target nodes accessible to obdiag without a password.
obdiag gather log --since 30m \
  --config obcluster.servers.nodes[0].ip=xx.xx.xx.1 \
  --config obcluster.servers.nodes[1].ip=xx.xx.xx.xx.2 \
  --config obcluster.servers.nodes[2].ip=xx.xx.xx.xx.3 \
  --config obcluster.servers.global.home_path=/home/admin/oceanbase

# Collect logs of the last 30 minutes with the information on different OBServer nodes varying.
obdiag gather log --since 30m \
  --config obcluster.servers.nodes[0].ip=xx.xx.xx.1 \
  --config obcluster.servers.nodes[0].ssh_username=test1 \
  --config obcluster.servers.nodes[0].ssh_password=****** \
  --config obcluster.servers.nodes[0].home_path=/home/test1/oceanbase \
  --config obcluster.servers.nodes[1].ip=xx.xx.xx.xx.2
  --config obcluster.servers.nodes[1].ssh_username=test2 \
  --config obcluster.servers.nodes[1].ssh_password=****** \
  --config obcluster.servers.nodes[1].home_path=/home/test2/oceanbase

# Collect logs of the last hour with the connection information of the sys tenant specified. In this case, obdiag automatically obtains the IP addresses of all nodes in the cluster.
obdiag gather log --since 1h \
  --config db_host=xx.xx.xx.xx \
  --config db_port=xxxx \
  --config tenant_sys.user=root@sys \
  --config tenant_sys.password=*** \
  --config obcluster.servers.global.ssh_username=test \
  --config obcluster.servers.global.ssh_password=****** \
  --config obcluster.servers.global.home_path=/home/admin/oceanbase

Collect logs filtered by keyword

# Collect logs of the last 30 minutes with logs filtered by `TRACE_ID`.
obdiag gather log --grep "TRACE_ID"  \
  --config db_host=xx.xx.xx.xx \
  --config db_port=xxxx \
  --config tenant_sys.user=root@sys \
  --config tenant_sys.password=*** \
  --config obcluster.servers.global.ssh_username=test \
  --config obcluster.servers.global.ssh_password=****** \
  --config obcluster.servers.global.home_path=/home/admin/oceanbase

# Collect logs of the last 30 minutes with logs filtered by multiple keywords, such as `AAAAA` and `BBBBB`.
obdiag gather log --grep "AAAAA" --grep "BBBBB" \
  --config db_host=xx.xx.xx.xx \
  --config db_port=xxxx \
  --config tenant_sys.user=root@sys \
  --config tenant_sys.password=*** \
  --config obcluster.servers.global.ssh_username=test \
  --config obcluster.servers.global.ssh_password=****** \
  --config obcluster.servers.global.home_path=/home/admin/oceanbase

# Collect logs of the specified period with logs filtered by multiple keywords, such as `AAAAA` and `BBBBB`.
obdiag gather log --from "2022-06-30 16:25:00" --to "2022-06-30 18:30:00" --grep "AAAAA" --grep "BBBBB" \
  --config db_host=xx.xx.xx.xx \
  --config db_port=xxxx \
  --config tenant_sys.user=root@sys \
  --config tenant_sys.password=*** \
  --config obcluster.servers.global.ssh_username=test \
  --config obcluster.servers.global.ssh_password=****** \
  --config obcluster.servers.global.home_path=/home/admin/oceanbase

Notice

If logs are filtered by multiple keywords, the keywords are evaluated by using the AND operator. In other words, only logs that meet all the specified keywords are collected.

Method 2: Use obdiag with a configuration file

Before you use obdiag, make sure that you have configured the login information of the target nodes in the config.yml configuration file of obdiag in the ~/.obdiag/ directory. For more information, see Configure obdiag.

Collect logs of the specified period

obdiag gather log --scope observer --from "2022-06-30 16:25:00" --to "2022-06-30 18:30:00" --grep STORAGE --encrypt true

The output is as follows:

...
ZipFileInfo:
+-------------------+-----------+
| Node              | LogSize   |
+===================+===========+
| xxx.xxx.xxx.xxx   | 36.184M   |
+-------------------+-----------+
...

ZipFileInfo:
+-------------------+-----------+
| Node              | LogSize   |
+===================+===========+
| xxx.xxx.xxx.xxx   | 44.176M   |
+-------------------+-----------+
...

Summary:
+-------------------+-----------+----------+------------------+--------+-------------------------------------------------------------------------------+
| Node              | Status    | Size     | Password         | Time   | PackPath                                                                      |
+===================+===========+==========+==================+========+===============================================================================+
| xxx.xxx.xxx.xxx   | Completed | 36.762M  | HYmVourcUyRNP8Om | 19 s   | obdiag_gather_pack_20220701183246/ob_log_xxx.xxx.xxx.xxx_20220701183247.zip   |
+-------------------+-----------+----------+------------------+--------+-------------------------------------------------------------------------------+
| xxx.xxx.xxx.xxx   | Completed | 638.200M | 1RicMaiLUUNfemnj | 718 s  | obdiag_gather_pack_20220701183246/ob_log_xxx.xxx.xxx.xxx_20220701183918.zip   |
+-------------------+-----------+----------+------------------+--------+-------------------------------------------------------------------------------+

Notice

Collect logs of a recent period

# Collect logs of the last hour.
obdiag gather log --since 1h

# Collect logs of the last 30 minutes.
obdiag gather log --since 30m

# Collect logs of the last 30 minutes with the configuration file of the target cluster specified.
obdiag gather log --since 30m -c /root/config.yml

Collect logs filtered by keyword

# Collect logs of the last 30 minutes with logs filtered by `TRACE_ID`.
obdiag gather log --grep "TRACE_ID"

# Collect logs of the last 30 minutes with logs filtered by multiple keywords, such as `AAAAA` and `BBBBB`.
obdiag gather log --grep "AAAAA" --grep "BBBBB"

# Collect logs of the specified period with logs filtered by multiple keywords, such as `AAAAA` and `BBBBB`.
obdiag gather log --from "2022-06-30 16:25:00" --to "2022-06-30 18:30:00" --grep "AAAAA" --grep "BBBBB"

Notice

If logs are filtered by multiple keywords, the keywords are evaluated by using the AND operator. In other words, only logs that meet all the specified keywords are collected.

Results

The command generates a folder in the specified directory. A sample folder name is obdiag_gather_pack_20240808105846.

Define a data masking rule

In most cases, data masking rules are defined on the business side. In other words, sensitive data is defined by users. obdiag supports custom data masking rules in V2.5.0 and later. Data masking rule files are stored in the ~/.obdiag/gather/redact/ directory. Each .py file corresponds to a data masking rule. The file name is the rule name.

This section describes how to define a data masking rule named a_test.

Create a rule file.

In the ~/.obdiag/gather/redact/ directory, create a rule file named a_test.py, which is also the rule name.
Create a rule class.

To facilitate management, you need to create a rule class to inherit the RedactBase class. You can define the class name, which is usually the same as the file name. Here is an example:
```
from handler.gather.plugins.redact import RedactBase
class a_test(RedactBase):
```
Create a rule method.

The rule class needs to implement the redact method. text is the input file content, which is not split into lines. You can split the content as needed.
```
text_lines = text.split('\n')
```
Return the processed information as a string.

If the file content is split into lines, the processing time increases greatly. We recommend that you do not split the file content unless necessary.
Create a rule instance.

You need to set a variable named after the rule name for being called by the obdiag kernel. In this example, you need to set a a_test variable as follows:
```
a_test = a_test()
```

Complete sample code

from handler.gather.plugins.redact import RedactBase
import re
class a_test(RedactBase):
    def redact(self, text):
        text_lines = text.split('\n')
        redacted_log = ''
        for line in text_lines:
            if 'test' in line:
                redacted_log = redacted_log + re.sub('test', '******', line)
        return redacted_log
a_test = a_test()