OAgent supports scoring models to help administrators objectively select models and continuously collect usage feedback to optimize default models and prompt strategies.
When to use
After integrating a new model, compare its capability level and score with existing models to decide whether to replace the default model.
Quantify the capability tiers of different models in OceanBase O&M scenarios (whether they meet the OAgent baseline, or possess complex O&M and long-context capabilities), serving as a basis for selection.
Comprehensive capability test
The comprehensive capability test evaluates a model's O&M capabilities level by level from L1 to L4:
Level |
Capability |
|---|---|
| L1 | Basic commands and structured expressions |
| L2 | Read-only Tool Usage (OAgent Baseline) |
| L3 | Advanced O&M Capabilities |
| L4 | Long-context O&M capability |
Test execution rules:
Execute the test level by level from L1 to L4. All capability items within a level are tested. If the current level is not passed, subsequent levels are skipped. Unexecuted items are scored 0 points and included in the total score.
Each item in L1 and L2 is tested three rounds, with passing determined by majority of rounds passed; L3 is scored based on the proportion of passing rounds. L4 progressively probes synthetic long contexts ranging from about 16K to 256K.
Procedure
In the left navigation bar, click Model Configuration to go to the model configuration page. For details, see Manage model services.
Click Test on the model service card of the target model to open the Model Capability Test page.
After reading the level descriptions, click Start Comprehensive Test and wait for the test to complete.
View the execution progress and historical reports in Evaluation Records. Each record includes the model name and its model service, target level, score, capability conclusion, status (Passed / Failed), and creation time. Click View Report to open the detailed report for that test.
View the report
The report displays pass/fail status, individual scores, and capability conclusions for each level. For example, the L4 test report for qwen3.5-plus might look like this:
Overall score: 98 points. Items such as "All Passed" and "Parallel Capability Meets Expectations" all score 100, indicating the L4 effective context reaches about 40K baseline.
Summary: "Passed the L4 Long-Context O&M Capability Test. The model can accurately generate inspection reports in OCP tool contexts above about 40K; any change operations still require manual confirmation."
Recommended Scope: Recommended for use.
Interpretation of results
The status is determined by the target level, and the capability conclusion indicates the actual achieved level. For example, a model with a score of 75 may state "Meet OAgent Baseline (L2), Possess L3 Complex O&M Enhanced Capability," but if the target level is L4, the status will still show Failed; a model with a score of 98 that meets L4 shows Passed.
When the score is 0 or abnormally low, the capability conclusion provides the reason, such as "Model name or API endpoint does not exist (HTTP 404)," which is often due to incorrect model name spelling or failed API address configuration. You can retest after verifying the model service configuration.
Note
A full capability test will actually call the model and consume tokens. For specific usage, refer to the evaluation report. L4 tests involve hundreds of kilobytes of synthetic long contexts, resulting in relatively high consumption. It is recommended to initiate a full test only after confirming model connectivity. For more information, see Manage model services.