When the environment where the data files are located cannot be directly accessed by OBServer, that is, when the network is disconnected from OBServer or it is inconvenient to copy the files to the local path of OBServer, you can use obloader to write the data to OceanBase AI Database.
Unlike executing statements such as LOAD DATA and Import data by using the LOAD DATA statement on the OBServer side, obloader reads files in the client process and interacts with the storage layer through RPC, making it suitable for scenarios such as offline package delivery and cross-subnet delivery.
Application scenarios
- Files are inaccessible to OBServer: The exported files are located on a business server, jump server, or in an isolated subnet. The network between OBServer and the data file location is disconnected, making it inconvenient to use the import method in which OBServer reads the files directly.
If the files are already placed in a path accessible to OBServer or in object storage, you should first evaluate the path selection options described in Import data by using the LOAD DATA statement, Import data by using the LOAD DATA statement, and Data import overview.
Command-line options
Refer to the OceanBase Loader and Dumper and configure according to your actual environment.
Command-line options |
Description |
OceanBase Cloud & ODP |
OceanBase AI Database & ODP |
OceanBase AI Database & OBServer |
|---|---|---|---|---|
| --direct | Indicates that direct load is used. | Required | Required | Required |
| --parallel | Server-side concurrency. The default value is 1, which is recommended to match the tenant's CPU specification. It is recommended to specify this option to ensure stable performance. |
Optional | Optional | Optional |
| --rpc-port | Server-side inner RPC port. To obtain the port:
|
Required | Required | Required |
| -u(--user) | The database username. | Required | Required | Required |
| -P(--port) | The SQL port number. | Required | Required | Required |
| -t(--tenant) | The tenant name of the cluster. | Optional If left empty, partition calculation may be skipped. |
Required | Required |
| -c(--cluster) | The cluster name of the database. | Optional | Required | - |
| --public-cloud | Specifies the OceanBase Cloud runtime environment. | Required | - | - |
| --no-sys | Used to identify tenants that do not depend on the sys tenant. | Optional | Optional | Optional |
| --sys-user | Identifies the user that depends on the sys tenant. If not specified, the default value is root. | Optional Exclusive with --no-sys |
Optional Exclusive with --no-sys |
Optional Exclusive with --no-sys |
| --sys-password | The password used to identify the dependency on the sys tenant. | Optional Exclusive with --no-sys |
Optional Exclusive with --no-sys |
Optional Exclusive with --no-sys |
Procedure
Execute the following command on the machine where the CSV files are stored (replace the placeholders with the actual address, port, user, and path):
bin/obloader -h <ODP or OBServer address> -P 2883 -u <user> -t <tenant name> -c <cluster name> -p \
-D <database name> --table LINEITEM --external-data --csv \
-f /data/tpch/bak/ \
--truncate-table --column-separator='|' \
--thread 16 --rpc-port 2885 --direct --parallel=16
Direct load accesses the RPC port, not the SQL port. By default, the SQL port of OBServer is 2881 and the RPC port is 2882; for ODP, the RPC port defaults to 2885. When obloader bypasses ODP to connect directly to OBServer and explicitly specifies the RPC port, it can use the direct load path.
Note that in a production environment, it is generally not recommended to have clients directly connect to OBServer bypassing ODP. When accessing through ODP, the location of the data's primary replica is transparent to the client, and routing is handled uniformly by ODP. However, when connecting directly to a specific OBServer, if the target data's primary replica is not on the current node, the write process may involve cross-node transaction coordination, increasing network and transaction overhead.
-fspecifies a readable data directory on the local machine. The tool reads files on the client side and writes them through the RPC direct load path. It is not required that OBServer can access this path.--rpc-portmust match the RPC listening port of the current connection target (ODP or OBServer). Do not confuse it with the SQL port.- If connecting directly to a specific OBServer, you need to check whether the data's primary replica resides on that node. Cross-server direct load may incur additional costs. If reachable, prefer ODP to align with the production business access method. Specific routing and network policies shall comply with operational specifications.
When data volume reaches the TB level, direct load is more conducive to throughput. You can plan the window by combining tenant resources with the considerations (such as table locks and merges) outlined in Data import overview and Data import overview.
Direct load mode configuration file (optional)
You can configure the direct_path_load section in {ob-loader-dumper}/conf/session.config.json for settings such as RPC timeout, retry, and task timeout. For example:
"direct_path_load": {
"rpc_connect_timeout": "15000",
"rpc_execute_timeout": "20000",
"runtime_retry_times": "5",
"runtime_retry_intervals": "50",
"task_timeout": "2592000000000",
"heartbeat_timeout": "60000000"
}
Field |
Description |
|---|---|
rpc_connect_timeout |
RPC connection timeout (in milliseconds). |
rpc_execute_timeout |
RPC execution timeout (in milliseconds). |
runtime_retry_times |
The maximum number of retries on a failed operation. If an operation fails for some reason, it will be retried according to this parameter. |
runtime_retry_intervals |
Retry interval (in milliseconds). The length of time to wait before retrying an operation. |
task_timeout |
Task timeout (in microseconds). 0 indicates no limit. If the operation is not completed within the configured time limit, it is considered a timeout. |
heartbeat_timeout |
Heartbeat timeout (in microseconds), used to detect the activity status of import operations. 0 indicates that this feature is disabled. |
Considerations
Direct load uses the RPC port for data transmission, not the SQL protocol port.
Data is committed on a per-table basis, not at the session or transaction level.
Retry or resumable upload from a breakpoint are not currently supported.
Bit-type data is not currently supported.
Virtual generated columns are not currently supported.
For import tasks involving a small amount of data, direct load is not recommended.
Specifying the
--replace-datacommand-line option does not handle unique index conflicts.The difference between the
--threadand--parallelcommand-line options:--threadspecifies the connection pool from the client to the server, which is maintained by the client.--parallelspecifies the number of worker threads that the OBServer can call for data writing and sorting.When using these options, it is recommended to keep
--threadand--parallelconsistent.