This topic provides an overview of the concepts and usage of vector embedding technology.
What is vector embedding?
Vector embedding is a technique that converts unstructured data into numerical vectors. These vectors capture the semantic information of unstructured data, enabling computers to "understand" and process the meaning of unstructured data. Specifically:
- Vector embedding maps unstructured data such as text, images, or audio/video into points in a high-dimensional vector space.
- In this vector space, semantically similar unstructured data is mapped to nearby positions.
- Vectors typically consist of hundreds of numbers (for example, 512 dimensions, 1024 dimensions).
- Mathematical methods (such as cosine similarity) can be used to calculate the similarity between vectors.
- Common vector embedding models include Word2Vec, BERT, and BGE. For example, when developing an RAG application, you usually need to embed text data into vector format before storing it in a vector database, while other structured data is stored in a relational database.
OceanBase AI Database supports storing vector data as a data type in relational tables, allowing vectors and traditional scalar data to be stored in OceanBase AI Database in an orderly and efficient manner.
How to generate vector embeddings
In OceanBase AI Database, you can generate vector embeddings using either of the following two methods:
Method |
Note |
Advantages |
Differences |
|---|---|---|---|
| Use the AI_EMBED function | Use the built-in AI function service of the database without installing additional dependencies. | Simple and convenient, can be called directly in SQL. | Supports embedding of text, image URLs, or image binary data.
NoteImage embedding is supported starting from V4.6.2.1. |
| Use external embedding models | Use external models such as Sentence Transformers, Ollama, and online APIs. | Flexible and supports more model choices. | Supports generating vector embeddings from data sources in various formats, including text and images. |