Application exception -- error message contains an OceanBase error code

During database operation, various exceptions may occur, such as application errors, database connection errors, database permission issues, database resource constraints, and network problems. One type of exception is an application exception, where the error message contains an OceanBase error code.

To help you quickly identify the root cause of such exceptions and efficiently resolve them, this topic provides a clear and practical troubleshooting process for application exceptions with error messages containing OceanBase error codes. The process offers detailed steps to enhance troubleshooting efficiency and minimize business impact, thereby supporting your daily O&M work.

The following figure shows the troubleshooting process for an application exception where the error message contains an OceanBase error code.

Process

When an application execution exception is reported with OceanBase error codes, you can troubleshoot the issue by following this process.

First, determine whether you can use OBClient to connect to the database and manually reproduce the issue:

• If yes, connect to the database after the issue is reproduced for further troubleshooting.

• If no, troubleshoot the program configuration's data source information.

Troubleshoot the issue by reproducing it

1. Use OBClient to connect to the OceanBase cluster at port 2881 (the default SQL port). If the port is customized, replace 2881 with the actual port number.

2. Manually execute the SQL statement that caused the error to reproduce the issue.

3. Execute the following statement to obtain the trace_id.

   Notice

   You must execute the following statement immediately after the error occurs; otherwise, the trace_id of the erroneous SQL statement will not be returned.

   MySQL mode

   Oracle mode

   The following example shows how to obtain the trace_id in MySQL mode:

   obclient> SELECT last_trace_id();
   

   The following example shows how to obtain the trace_id in Oracle mode:

   obclient> SELECT last_trace_id() FROM DUAL;
   

4. Obtain the host information where the SQL statement is executed.

   Generally, an OceanBase cluster is deployed with multiple nodes. You can execute the following SQL statement to obtain the node where the SQL statement is executed and then filter the logs.

   MySQL mode

   Oracle mode

   Execute the following statement in MySQL mode:

   obclient> SELECT * FROM oceanbase.GV$OB_SQL_AUDIT WHERE trace_id=last_trace_id;
   

   Replace last_trace_id with the trace_id obtained in the previous step.

   Execute the following statement in Oracle mode:

   obclient> SELECT * FROM SYS.GV$OB_SQL_AUDIT WHERE trace_id=last_trace_id;
   

   Replace last_trace_id with the trace_id obtained in the previous step.

   In the GV$OB_SQL_AUDIT view, the host corresponding to svr_ip is the one where the SQL statement is executed.

5. Use the ssh command to log in to the host based on the obtained host information.

6. Go to the log directory.

   Assume that the installation directory of OceanBase Database is /home/admin/oceanbase. The following example shows the specific path of logs, which may vary depending on your actual environment.

   cd /home/admin/oceanbase/log
   

7. Execute the following command to filter the required information from the logs.

   grep "${trace_id}" observer.log
   

   grep "${trace_id}" observer.log.xxx
   

   Replace ${trace_id} with the trace_id obtained earlier. observer.log.xxx is a log file with a timestamp. Replace xxx with the actual timestamp.

8. Based on the information provided in the logs and the error message, analyze the issue.

   For more information about logs, see Overview. If the information in the logs is unclear, contact technical support for assistance.

Troubleshoot the data source configuration

1. Based on the data source, identify the access link.

2. The access link contains the following information:

   • The number and addresses of OBProxies

   • The number and addresses of OBServer hosts

3. Determine whether OBProxy is involved.

   • If yes, follow these steps:

     1. Run the ssh command to log in to the corresponding OBProxy node.

     2. Run the cd command to go to the log directory of the OBProxy. The OBProxy log is stored in the /log directory under its installation directory.

     3. Run the following command to filter out the error logs:

        grep "Error text" obproxy_error.log obproxy_error.log.xxx | grep "Error time point"
        

     4. Identify the OBServer node that the SQL statement is routed to based on the filtered log entries.

     5. Run the ssh command to log in to the OBServer node for further troubleshooting.

   • If no, follow these steps:

     1. Run the ssh command to log in to the corresponding OBProxy node.

     2. Go to the log directory.

        Assume that the installation directory of OceanBase Database is /home/admin/oceanbase. For the specific path where logs are stored, refer to the actual environment.

        cd /home/admin/oceanbase/log
        

     3. Run the following command to filter out the error logs:

        grep "sending error" observer.log observer.log.xxx | grep "Error code"
        

     4. Obtain the trace_id from the filtered logs, and then filter the logs again based on the trace_id.

        grep "trace id" observer.log observer.log.xxx
        

     5. Analyze the problem based on the information provided in the logs and the error messages.

        For more information about logs, see Overview. If the information in the logs is unclear, contact technical support for assistance.

Case studies

• A regular user of a general business tenant encountered the error ERROR 1040 (08004): Too many connections when directly connecting to port 2881 of OceanBase Database. For more information, see Troubleshoot the error ERROR 1040 (08004): Too many connections when a regular user connects.

References

• Find logs of a specific SQL request

• Find logs of the last SQL request based on the Trace feature

• Log overview