Complex stream processing, cross-system ETL, or cloud vendor-managed pipelines are typically performed in computing engines external to OceanBase AI Database (such as Flink, DataWorks, dbt, AWS Glue, etc.), and then integrated with OceanBase AI Database through synchronization or loading. This topic provides a brief introduction to mainstream tools.
OceanBase Developer Center (ODC)
OceanBase Developer Center (ODC) is a database graphical development tool and a collaborative platform for data development and production change management. ODC is available in two editions: desktop and web. The desktop edition focuses on database development capabilities and supports Windows, macOS, and Linux operating systems. It is lightweight and easy to deploy. The web edition provides the same tool capabilities as the desktop edition but also includes management and collaboration features. It focuses on ensuring the security, compliance, and efficiency of database changes.
For more information about how to use ODC, see OceanBase Developer Center (ODC).
ETL tools in the ecosystem
Flink
Flink is an open-source framework for large-scale data processing and analysis. Flink CDC is a component that captures database change events on the Flink platform. Together, they provide a powerful real-time data processing and analysis solution.
For more information about how to use Flink, see Synchronize data from a MySQL database by using Flink CDC.
dbt (Data Build Tool)
dbt (data build tool) is an open-source data transformation tool that uses SQL to convert commands into tables or views. This topic describes how to use dbt-oceanbase to analyze data in OceanBase AI Database.
For more information about how to use dbt, see Analyze data in OceanBase by using dbt.
DataWorks
DataWorks is a data development and governance platform on Alibaba Cloud. It provides unified end-to-end processing capabilities for data warehouses, data lakes, and hybrid data lake and warehouse solutions based on big data engines such as MaxCompute, Hologres, EMR, AnalyticDB, and CDP. DataWorks offers various features, including data integration, data development, data modeling, and data analysis.
AWS Glue
AWS Glue is a serverless data integration service that helps you easily discover, prepare, move, and integrate data from multiple sources to support analysis, machine learning, and application development. It provides comprehensive data integration capabilities, including data discovery, modern ETL, data cleaning and transformation, and centralized cataloging, all integrated into a single service. AWS Glue does not require you to manage infrastructure. It supports various workloads such as ETL, ELT, and streaming, and can scale on demand to handle any data size or type.
For more information about how to use AWS Glue, see Migrate OceanBase data by using AWS Glue.
Tool selection recommendations
Tool |
Core advantages |
Scenarios |
|---|---|---|
| ODC | A dedicated management tool for OceanBase databases, deeply integrated with the ecosystem | Development and O&M of OceanBase AI Database |
| Flink | Real-time stream processing and high-throughput batch processing | Stream data monitoring, real-time report generation |
| dbt | Standardized data modeling, traceable analysis processes | Modeling of analytical data warehouses (such as OceanBase AI Database) |
| DataWorks | Full-stack support for the Alibaba Cloud ecosystem, visual development | Cloud-native data lakes and offline analysis |
| AWS Glue | Serverless ETL, seamless integration with the AWS ecosystem | Building data lakes and batch processing on AWS Cloud |