Troubleshooting

Troubleshooting procedure

Before troubleshooting, ensure that the versions of OceanBase Database, OceanBase Database Proxy (ODP), and obbinlog are compatible. For information about obbinlog version compatibility, see Release notes.

The following figure illustrates the troubleshooting procedure for common issues.

If no abnormal information is found, or if you cannot resolve the issue based on the FAQ section, check for similar questions in the Q&A forum of the OceanBase community. If no similar issue exists, submit a new question. When submitting your question, provide the following information to help OceanBase technical support quickly locate and resolve the issue:

1. Version numbers of OceanBase Database, ODP, and obbinlog

2. Relevant status information obtained during your troubleshooting

3. Original log files containing the exception information

FAQ

The Node resources are insufficient error is returned when CREATE BINLOG is executed

Check the resource usage of the Binlog cluster in the log/logproxy.log file. For example, the following log entry may appear:

[error] selection_strategy.cpp(519): [ResourcesFilter] The resource threshold of node [89842bbc393b3d1afea947f3267895d(10.10.10.1:2983)] does not meet requirements, used cpu: 6.7903543 >= 0.8 || used memory: 0.6238438 >= 0.85 || used disk: 0.12204241 >= 0.7000000000000001 

In the preceding example, the CPU check condition used cpu: 6.7903543 >= 0.8 is true, meaning the system considers the CPU utilization to exceed the threshold of 0.8. As a result, the binlog task creation fails.

Specifically, in this example, the binlog server is deployed in a ccontainer, and the CPU information obtained is inaccurate, causing the reported CPU usage to be higher than the actual value.

If you encounter a similar situation, you can either adjust the threshold values in the metadata table config_template as needed, or directly disable the resource usage check. For more information about the metadata table, see Metadata information.

-- Example: Change the CPU utilization threshold to 0.9.
UPDATE config_template SET value='90' WHERE key_name='node_cpu_limit_threshold_percent';

-- Example: Disable resource usage check.
UPDATE config_template SET value='false' WHERE key_name='enable_resource_check';

After the modification, you need to restart the binlog server and then execute the CREATE BINLOG statement again.

# Restart without using supervisord
./run.sh stop
./run.sh start

# Restart with supervisord
supervisorctl restart binlog

The error start lsn from all server fail is returned in libobcdc while no new binlogs are generated by the binlog instance

This error usually occurs because the data dictionary is reclaimed. In this case, you can decrease the interval for generating a data dictionary. For example, set it to 1 hour.

Adjust the data dictionary generation interval for the current tenant (non-sys tenant):

ALTER SYSTEM SET dump_data_dictionary_to_log_interval = '1h';

Or, adjust the data dictionary generation interval for all tenants by connecting to the sys tenant:

ALTER SYSTEM SET dump_data_dictionary_to_log_interval = '1h' TENANT all_user; 

The Failed to match gtid ... error is returned after the binlog instance is recovered or restarted

This error usually occurs when binlog files are inconsistent with the binlog index file. Binlog files and the binlog index file are stored in the run/${instance_name}/data/ directory. The binglog index file mysql-bin.index contains the offset information of all binlog files.

/home/ds/oblogproxy/run/ob4g1x9cpve0qo/data/mysql-bin.001352      1352    {hash:11302655175424709514, inc:938515546, addr:"10.10.10.10:2882", t:1690537006395697}=68058433       {hash:11268779827260349032, inc:938515541, addr:"10.10.10.10:2882", t:1690537006386235}=68058432      1690537005389969        414233659

In the example above, this line of index record indicates that the maximum offset for the Binlog log file mysql-bin.001352 is 414233659 (second to last column), and the corresponding timestamp is 1690537005389969 (third to last column).

To fix inconsistencies between the binlog log files and the binlog index files, navigate to the run/${instance_name}/data/ directory and follow these steps:

1. Check whether the mysql-bin.index.tmp file exists in the directory. If it does and its last record is later than the one in the mysql-bin.index file, overwrite mysql-bin.index with mysql-bin.index.tmp. Back up the existing mysql-bin.index file before this operation.

2. Obtain the latest binlog file and offset based on the last record in the mysql-bin.index file.

3. Split the latest binlog file. Back up the binlog file before this operation.

   cp mysql-bin.001352 mysql-bin.001352-bak
   split -b 414233659 mysql-bin.001352
   # By default, the generated files are named `xaa` and `xab`.
   mv xaa mysql-bin.001352
   

4. Try to recover or restart the binlog instance again.