Direct load

Background information

OceanBase Database allows you to insert data into a database by using direct load. In other words, OceanBase Database can directly write data into data files. With the direct load feature, you can bypass SQL layer interfaces, directly allocate space and insert data into data files, thereby improving the data import efficiency.

Scenarios

The direct load feature applies to the following scenarios:

• Data migration and synchronization. In data migration and synchronization, a large amount of data of different types need to be migrated from different data sources to OceanBase Database. Conventional SQL interfaces cannot meet the requirement on timeliness.

• Conventional ETL. After data is extracted and transformed in the source, a large amount of data often needs to be loaded to the destination within a short time. The direct load technology can improve the import performance.

When you load data from text files or other data sources to OceanBase Database by using the direct load technology, the data loading efficiency can be improved.

Considerations

• Direct write requires data transmission to be performed through the RPC port, which is not the SQL protocol port.

• Direct write is performed on a table basis, and is not session-based or transaction-based.

• Retries and resumption of interrupted transfers are not supported.

• Data of the bit type is not supported.

• Virtual generated columns are not supported.

• The direct load feature is not recommended for small-scale import tasks.

• The direct load feature does not support importing multiple tables at the same time.

• If you specify the --replace-data option, unique index conflicts still cannot be resolved.

• The --thread and --parallel options have different meanings:

  • The --thread option specifies the size of the connection pool from the client to the server. The connection pool is maintained by the client.

  • The --parallel option specifies the number of worker threads that an OBServer node can use to write data and perform sorting.

  • We recommend that you set the values of the --thread and --parallel options to the same value.

Command-line options for the direct load mode

Parameters of the direct load mode

You can configure the parameters of direct load in the session.config.json file in the {ob-loader-dumper}/conf directory.

Example:

"direct_path_load": {
    "task_timeout_ms": "25920000", 
    "heartbeat_timeout_ms": "60000000",
    "heartbeat_interval_ms": "30000000" 
}

• task_timeout_ms: the timeout period for the task. If the task is not completed within the specified period, it is considered a timeout. The value is in milliseconds.

• heartbeat_timeout_ms: specifies the heartbeat timeout period in milliseconds. This parameter is used to detect the activity of the import operation.

• heartbeat_interval_ms: specifies the interval between two heartbeats.

--rpc-port

Private deployment

In private deployment scenarios, OBLOADER in direct load mode supports connecting to OBServer and ODP.

• Connect to OBServer

  When OBLOADER is in direct load mode and connects to OBServer, the default port is 2882. You can use the sys tenant user to connect to the database and query the system view DBA_OB_SERVERS to obtain the RPC port SVR_PORT.

  obclient [information_schema]> select * from oceanbase.DBA_OB_SERVERS \G;
  *************************** 1. row ***************************
                SVR_IP: 127.0.0.1
              SVR_PORT: 2882
                    ID: 1
                  ZONE: zone1
              SQL_PORT: 2881
        WITH_ROOTSERVER: YES
                STATUS: ACTIVE
    START_SERVICE_TIME: 2025-05-16 16:39:12.412141
              STOP_TIME: NULL
  BLOCK_MIGRATE_IN_TIME: NULL
            CREATE_TIME: 2025-05-16 16:35:52.066722
            MODIFY_TIME: 2025-05-16 16:39:14.436893
          BUILD_VERSION: 4.3.5.2_102000162025051417-b3faexxxx(May 14 2025 18:01:25)
      LAST_OFFLINE_TIME: NULL
  1 row in set (0.001 sec)
  

• Connect to ODP

  When OBLOADER is in direct load mode and connects to ODP, the default port is 2885. You can use the sys tenant user to connect to the database and view the RPC port value.

  obclient [information_schema]> show proxyconfig like "%rpc_listen_port%";
  +-----------------+-------+---------------------------------+-------------+---------------+--------------+--------------+
  | name            | value | info                            | need_reboot | visible_level | range        | config_level |
  +-----------------+-------+---------------------------------+-------------+---------------+--------------+--------------+
  | rpc_listen_port | 2885  | obproxy rpc service listen port | true        | SYS           | (1024,65536) | LEVEL_GLOBAL |
  +-----------------+-------+---------------------------------+-------------+---------------+--------------+--------------+
  1 row in set (0.003 sec)
  

ApsaraDB for OceanBase/OceanBase Cloud

If you are using ApsaraDB for OceanBase or OceanBase Cloud, you need to enable Direct Load Port on the Database Proxy page. The default port is 3307.

• For ApsaraDB for OceanBase, see Direct load for more information.

• For OceanBase Cloud, see Direct load for more information.