OceanBase
English日本語简体中文
Docs Home
Products
Integrations
Release Notes

Databases

OceanBase DatabaseOceanBase AI DatabaseOceanBase Cloud OceanBase Best Practices

Tools

OceanBase Cloud PlatformOceanBase Migration ServiceOceanBase Developer CenterOceanBase Migration AssessmentOceanBase Admin ToolOceanBase Loader and DumperOceanBase DeployerKubernetes operator for OceanBaseOceanBase Diagnostic Tool OceanBase Binlog ServiceOceanBase DataPilotOceanBase OAgent

Connectors and Middleware

OceanBase Database Proxy OceanBase Connector/C OceanBase Connector/J OceanBase Connector/ODBC OceanBase Connector/NETOceanBase Command-Line Client

Databases

OceanBase DatabaseOceanBase AI DatabaseOceanBase Cloud OceanBase Best Practices

Connectors and Middleware

OceanBase Database Proxy OceanBase Connector/C OceanBase Connector/J OceanBase Connector/ODBC OceanBase Connector/NETOceanBase Command-Line Client

Tools

OceanBase Cloud PlatformOceanBase Migration ServiceOceanBase Developer CenterOceanBase Migration AssessmentOceanBase Admin ToolOceanBase Loader and DumperOceanBase DeployerKubernetes operator for OceanBaseOceanBase Diagnostic Tool OceanBase Binlog ServiceOceanBase DataPilotOceanBase OAgent
OceanBase DatabaseOceanBase Cloud ToolsConnectors and MiddlewareLifecycle Support Policy

OceanBase Loader and Dumper

V5.0.0
  • Document Overview
  • Introduction
  • Technical Mechanism
  • Preparations
    • Prepare the environment
    • Prepare data
    • Download OBLOADER & OBDUMPER
    • Startup parameters
  • User Guide (OBLOADER)
    • Quick start
    • Command-line options
    • Multi-database import
    • Direct load
    • Data processing
      • Define a control file
      • Preprocessing functions
      • Case expressions
    • Use cases of command-line options
    • Performance tuning
    • Error handling
    • FAQ
  • User Guide (OBDUMPER)
    • Quick start
    • Command-line options
    • Multi-database export
    • Data processing
      • Define a control file
      • Preprocessing functions
      • Case expressions
    • Performance tuning
    • FAQ
  • Control file guide
  • Security Features
  • Connection configuration
  • Self-service Troubleshooting
  • Release Note
    • Release Note
      • 5.x
        • OBLOADER & OBDUMPER V5.0.0
      • 4.x
        • OBLOADER & OBDUMPER V4.3.7
        • OBLOADER & OBDUMPER V4.3.6
        • OBLOADER & OBDUMPER V4.3.5
        • OBLOADER & OBDUMPER V4.3.4.1
        • OBLOADER & OBDUMPER V4.3.4
        • OBLOADER & OBDUMPER V4.3.3.1
        • OBLOADER & OBDUMPER V4.3.3
        • OBLOADER & OBDUMPER V4.3.2.1
        • OBLOADER & OBDUMPER V4.3.2
        • OBLOADER & OBDUMPER V4.3.1.1
        • OBLOADER & OBDUMPER V4.3.1
        • OBLOADER & OBDUMPER V4.3.0
        • OBLOADER & OBDUMPER V4.2.8.2
        • OBLOADER & OBDUMPER V4.2.8.1
        • OBLOADER & OBDUMPER V4.2.8
        • OBLOADER & OBDUMPER V4.2.7
        • OBLOADER & OBDUMPER V4.2.6
        • OBLOADER & OBDUMPER V4.2.5
        • OBLOADER & OBDUMPER V4.2.4
        • OBLOADER & OBDUMPER V4.2.1
        • OBLOADER & OBDUMPER V4.2.0
        • OBLOADER & OBDUMPER V4.1.0
        • OBLOADER & OBDUMPER V4.0.0
      • 3.x
        • OBLOADER & OBDUMPER V3.3.2
        • OBLOADER \& OBDUMPER V3.1.0
        • OBLOADER \& OBDUMPER V3.0.0
    • Versioning rules

Download PDF

Document Feedback
  1. Documentation Center
  2. OceanBase Loader and Dumper
  3. V5.0.0

OBLOADER & OBDUMPER V5.0.0

Last Updated:2026-11-10 11:21:27

Performance improvements
New features
Bug fixes
File and script processing
Data type and object compatibility
Error handling and runtime diagnostics
Security fixes
Dependency upgrades

OBLOADER & OBDUMPER V5.0.0 adds support for skipping records at the beginning and end of files. It improves parallel export, credential management for multiple accounts, and object definition export in OceanBase Database Oracle mode. This release also fixes issues with file processing, data compatibility, and runtime diagnostics, and upgrades dependencies.

Performance improvements

OBLOADER & OBDUMPER V5.0.0 improves import and export performance across different scenarios. The following table summarizes the improvements.

The results below use V4.3.7 as the baseline.

Category
Metric
Improvement
Test conditions
INSERT Standard import throughput Up to 3.75 times the baseline ClickBench, 1 million rows, 4 GB heap
INSERT Import throughput with a small heap Up to 38.73 times the baseline ClickBench, 4 million rows, 1 GB heap; resolves the bottleneck caused by sustained memory throttling
INSERT Cumulative CPU consumption on the OceanBase Database side Reduced by 27.16% ClickBench, 1 million rows, 4 GB heap, for the same import workload
Load Direct Load throughput 1.47 times the baseline ClickBench, 20 GB, 4 GB heap
Load Cumulative CPU consumption on the OceanBase Database side Reduced by 7.84% ClickBench, 20 GB, 4 GB heap, for the same import workload
Dump CSV export throughput Up to 2.18 times the baseline Approximately 1 GB of data, 4 GB heap

Import and export throughput in V5.0.0 compared with V4.3.7

Standard import
Direct Load
CSV export
Small-heap performance
Cumulative CPU consumption
Adaptive scheduling
File parsing

INSERT: Standard import throughput reaches up to 3.75 times the baseline

Improved read/write scheduling, default batching strategies, and reuse of binding plans within batches. These changes reduce read waits, repeated construction work, and write overhead, improving standard import performance with the default configuration.

Test conditions
V4.3.7 elapsed time
V5.0.0 elapsed time
Speedup
1 million rows, 4 GB heap 73.13 seconds 19.48 seconds 3.75 times

Load: Throughput for a 20 GB Direct Load reaches 1.47 times the baseline

Improved batching and budgeting for in-flight data in Direct Load, reducing data backlog and client memory usage when importing large files.

Test conditions
V4.3.7 elapsed time
V5.0.0 elapsed time
Speedup
20 GB, 4 GB heap 493.73 seconds 336.89 seconds 1.47 times
Resource metric
V4.3.7
V5.0.0
Change
Peak client RSS 20.21 GiB 3.47 GiB Approximately one-sixth of the baseline (17.17%)

RSS includes resident memory-mapped pages and is not equivalent to Java heap memory. A reduction in cumulative CPU consumption does not imply a lower instantaneous CPU load during import.

Dump: CSV export throughput reaches up to 2.18 times the baseline

Reusing row buffers and reducing initialization and object allocation overhead improves export efficiency for CSV and other formats.

Source CSV size
V4.3.7 elapsed time
V5.0.0 elapsed time
Speedup
Approximately 1 GB (measured: 0.99 GB) 64.07 seconds 29.41 seconds 2.18 times

Resolving memory throttling bottlenecks: Throughput reaches up to 38.73 times the baseline

In earlier versions, limited memory and a backlog of in-flight data in small-heap environments could repeatedly trigger throttling for memory protection. V5.0.0 redesigns read/write scheduling and task-level memory budgets to address this issue. It limits in-flight data based on available memory, reduces the rate of reading as memory pressure rises, and prioritizes draining existing batches. This prevents a persistent backlog from compounding repeated throttling. In tests with a 1 GB heap, these changes resolved the throughput bottleneck caused by sustained memory throttling in the earlier version, allowing concurrent writes to be effective even with a small heap.

Performance shows little regression with a 1 GB heap compared with a 4 GB heap. In the same-version test with 4 million rows, the 1 GB configuration used one-quarter of the heap memory while retaining 97.46% of the throughput of the 4 GB configuration, substantially reducing the dependence on heap size.

The following chart compares elapsed times for a 1-million-row INSERT test. With a 1 GB heap, V5.0.0 achieved 15.83 times the throughput of V4.3.7 in this test. The 38.73-fold result above applies to the 4-million-row test.

Elapsed time for a 1-million-row INSERT with 1 GB and 4 GB heaps

Cumulative CPU consumption on the OceanBase Database side

In addition to improved client processing performance, OceanBase Database consumes less cumulative CPU time to complete the same import workload: 27.16% less for INSERT with 1 million rows, and 7.84% less for Direct Load with 20 GB of data.

Test conditions
V4.3.7 cumulative CPU consumption
V5.0.0 cumulative CPU consumption
Reduction
INSERT, 1 million rows, 4 GB heap 228.79 CPU-seconds 166.65 CPU-seconds 27.16%
Direct Load, 20 GB, 4 GB heap 2471.13 CPU-seconds 2277.48 CPU-seconds 7.84%

Cumulative CPU consumption in V5.0.0 compared with V4.3.7

Adaptive Reader/Writer scaling and dynamic read/write ratios

V5.0.0 enables adaptive Reader/Writer scheduling and task-level budgets for in-flight data by default. The numbers of Readers and Writers can increase or decrease automatically as runtime conditions change. The read/write ratio adjusts accordingly to match the file read rate to the database's write capacity.

  • Dynamic read and write concurrency: Reader and Writer counts are adjusted independently based on the read rate, write throughput, database response, CPU usage, and heap memory pressure. The read/write ratio is then rebalanced to reduce waits and backlogs caused by insufficient reads or slow consumption.

  • Prioritizing draining under memory pressure: Task-level budgets limit the memory occupied by in-flight batches. As memory pressure rises, the scheduler reduces the read rate, adjusts concurrency, and prioritizes writing batches that have already been generated. This reduces backlogs and repeated throttling for memory protection.

  • Less manual tuning: Standard INSERT workloads no longer require repeated experimentation with thread counts, fixed read/write ratios, buffer capacity, or batch sizes. Tasks adapt to the current workload across different resource configurations.

File parsing improvements

Improved parsing and read paths for multiple file formats, reducing parsing and object allocation overhead and enhancing parallel reads of a single file.

Single-threaded parsing throughput increased by 71.33%. In the dedicated single-threaded CUT test, throughput increased from 64.56 MiB/s to 110.61 MiB/s. These are measured single-threaded results using the same test baseline as the table below.

Format
Concurrency
Parsing throughput improvement
CSV 16 threads → 16 threads 155.77%
CUT Single thread → single thread 71.33%
CUT 32 threads → 32 threads 56.76%
SQL 16 threads → 16 threads 74.02%
POS 16 concurrent readers → 16 concurrent readers 56.88%
Avro Actual read concurrency: 1 → 16 Up to 15.43 times the baseline
Parquet Actual read concurrency: 1 → 16 Up to 13.40 times the baseline

New features

  • Added the --skip-header N and --skip-footer N options to skip the specified number of records at the beginning and end of each source file, respectively, when importing CSV, CUT, or POS files.

  • Improved parallel export. SAMPLE is now the default split strategy. Use --split-type LEGACY to select the previous strategy.

  • Enhanced credential management in secure-gen to support multiple accounts. A single encrypted credential file can now store credentials for multiple application and sys accounts. Credentials are matched by user, tenant, cluster, and address, simplifying connections across environments without manual password entry.

  • Improved object definition export in OceanBase Database Oracle mode. Added support for exporting comments on views and view columns, and improved the preservation of metadata for partitioned unique indexes.

Bug fixes

File and script processing

  • Fixed issues with explicit column mappings for CSV files, MySQL column name matching, and control file lookup that could cause values to be imported into the wrong columns.

  • Fixed a split processing performance regression when filtering records at the beginning and end of files.

  • Fixed an issue where valid U+FEFF characters at the beginning of CUT file splits were lost.

  • Fixed issues with short reads, final lines without line breaks, record lengths, and multibyte boundary handling in POS files. Data with invalid encoding is now handled as bad records.

  • Fixed issues with changes to column lists and mixed use of explicit and implicit column lists in SQL files. Improved batch error handling to avoid affecting subsequent valid records.

  • Improved statement boundary detection and JDBC escape rewriting for DDL/MIX files containing comments, DELIMITER directives, Oracle blocks, or q-quote syntax. Session state and transaction order are preserved within MIX files.

Data type and object compatibility

  • Fixed NLS conversion errors when importing native ORC timestamps into OceanBase Database in Oracle mode, and issues with preserving DECIMAL precision.

  • Improved handling of time, Decimal, NULL, and nested types in Parquet/Avro files. Fixed an issue where batches containing valid Avro union data were rejected in their entirety.

  • Fixed issues with exporting MySQL TIME values, including loss of microsecond precision and failure to preserve values exceeding 24 hours.

  • Fixed an issue where records with parameter binding failures were still included in the successful row count.

  • Fixed index and constraint name conflicts and incorrect rewriting of special identifiers during imports with remapping.

  • Fixed foreign key dependency ordering when importing parent and child tables through standard JDBC into OceanBase Database 4.x in Oracle mode.

  • Fixed an issue where hidden primary keys were exported for tables that already had explicitly defined primary keys.

  • Disabled LOB Locator V2 by default to improve LOB import and export compatibility.

Error handling and runtime diagnostics

  • Fixed issues where export tasks hung or were incorrectly marked as complete when the disk was full, file rollover failed, or finalization failed. Improved error propagation when merging parallel export output into a single file and initializing CSV headers.

  • Fixed incorrect parsing of local paths containing special characters such as #, ?, and %, as well as issues with output directory validation and inaccurate error messages.

  • Restored periodic progress logging and corrected the reporting of throughput, buffer capacity, and wait time.

  • Fixed an issue where a connection was returned even after its initialization SQL failed to execute.

  • Fixed an issue where trustStore configuration was still required for SSL connections configured with trustServerCertificate=true.

Security fixes

  • Upgraded Guava and addressed the related deserialization risks in Hadoop's relocated namespaces.

  • Updated dependencies, including Log4j, Netty, and the Jackson library embedded in Parquet, to address related security risks.

  • Removed the legacy SOFA Common and Commons Lang 2 runtime dependencies.

Dependency upgrades

  • Upgraded Guava from 32.0.1-jre to 33.7.2-jre.

  • Upgraded Log4j from 2.18.0 to 2.25.5.

  • Upgraded Netty from 4.1.136.Final to 4.1.137.Final.

  • Upgraded AWS SDK v2 from 2.35.4 to 2.54.14.

  • Upgraded obkv-table-client from 2.4.0 to 2.4.1.

  • Upgraded parquet-jackson from 1.15.1 to 1.18.1.

Previous topic

Self-service Troubleshooting
Last

Next topic

OBLOADER & OBDUMPER V4.3.7
Next
on this page
Performance improvements
New features
Bug fixes
File and script processing
Data type and object compatibility
Error handling and runtime diagnostics
Security fixes
Dependency upgrades