Purpose
This function takes text and a specified tokenizer or analyzer, along with parameters in JSON format, and outputs the tokenization result.
Syntax
TOKENIZE('text' [, 'parser'] [, 'behavior_ctrl'])
Parameters
Parameters |
Type |
Required |
Note |
|---|---|---|---|
text |
VARCHAR / TEXT / CHAR |
Yes | The text content to be segmented. |
parser |
string | No | Tokenizer name. Default: space You can specify the original parser name (space(spacer), ngram(N-gram tokenizer), beng(English basic word tokenizer), ik(Chinese word tokenizer), ngram2(N-gram tokenizer), etc.) or analyzer(Analyzer text analyzer). |
behavior_ctrl |
JSON Array String | No | An array of parameters that control the output format and word segmentation. Valid options include output(output format), stopwords(stopword list), case(capitalization), additional_args(Additional parameters of the original parser), analysis(The JSON definition of Analyzer). |
The behavior_ctrl parameter is described in detail as follows:
Parameter |
Note |
|---|---|
output |
Output format. default(Default): Returns a token array, for example, ["token1", "token2"]; all: Returns a JSON object containing the word frequency and document length, for example, {"tokens": [{"token1": 1}, {"token2": 2}], "doc_len": 5}. |
stopwords |
Stopword list. If not specified, the global default configuration is used. If set to empty, stopwords are disabled. |
case |
Case sensitivity. If not specified, the system default behavior is used. You can specify upper / lower. |
additional_args |
Additional parameters of the original parser, such as ngram of ngram_token_size, space/beng of min_token_size / max_token_size. |
analysis |
Only parser = 'analyzer' Valid for a period of time. The JSON definition of Analyzer supports two formats: JSON object and JSON string. It is recommended to use the object Syntax: {"analysis": {"analyzer": "$analyzer_type"}}; It is also compatible with string Syntax: {"analysis": "{\"analyzer\": \"standard\"}"}. The return result is in JSON format, and the token order is not guaranteed to be consistent with the original text.
NoticeThis feature is supported starting from V4.6.2.1. |
The additional_args parameter is described in detail as follows:
Parser |
Parameters |
Value range |
Note |
|---|---|---|---|
ngram |
ngram_token_size |
[1, 10] | The length of the N-gram token. |
space / beng |
min_token_size |
[1, 16] | The minimum length of a token. |
space / beng |
max_token_size |
[10, 84] | Maximum Length of Token (must be ≥min_token_size). |
The following values are supported for $analyzer_type:
Analyzer type |
Language |
|---|---|
standard |
Multilingual |
english |
English |
thai |
Thai |
vietnamese |
Tiếng Việt |
indonesian |
Indonesian |
malay |
Malay |
Examples
Use the original parser
TOKENIZE supports all original built-in parsers, and their usage is the same as that of the parsers used when creating indexes.
SELECT TOKENIZE('hello world');
-- Result: ["world", "hello"]
SELECT TOKENIZE('hello world', 'ngram');
-- Result: ["lo", "he", "ll", "or", "rl", "wo", "ld", "el"]
SELECT TOKENIZE('hello world', 'beng');
-- Result: ["world", "hello"]
SELECT TOKENIZE('hello world', 'space', '[{"output": "all"}]');
-- Result: {"tokens": [{"world": 1}, {"hello": 1}], "doc_len": 2}
additional_args parameter
SELECT TOKENIZE('hello', 'ngram', '[{"additional_args": [{"ngram_token_size": 3}]}]');
-- Result: ["hel", "ell", "llo"]
SELECT TOKENIZE('hello world abcdefg', 'space',
'[{"output": "all"}, {"additional_args": [{"min_token_size": 6}]}]');
-- Result: {"tokens": [{"abcdefg": 1}], "doc_len": 1}
Use the text analyzer (Analyzer)
Notice
Once the behavior_ctrl parameter is specified, it must contain at least one valid configuration item. Passing an empty object [{}] will result in an error. To use the default standard analyzer, either omit the third parameter entirely or explicitly pass [{"analysis": {"analyzer": "standard"}}].
-- Recommended: Write the analysis result as a JSON object.
SELECT TOKENIZE('Hello World! OceanBase is a distributed database.', 'analyzer',
'[{"analysis": {"analyzer": "standard"}}]');
-- Compatible: analysis written as a JSON string
SELECT TOKENIZE('text', 'analyzer',
'[{"analysis": "{\\"analyzer\\": \\"standard\\"}"}]');
-- If the third parameter is omitted, the standard analyzer is used by default.
SELECT TOKENIZE('John', 'analyzer');
-- Result: ["john"]
SELECT TOKENIZE('The quick brown fox jumps.', 'analyzer',
'[{"analysis": {"analyzer": "english"}}, {"output": "all"}]');
-- Result: {"tokens": [{"quick": 1}, {"brown": 1}, {"jump": 1}, {"fox": 1}], "doc_len": 4}
The following table describes the syntax of the TOKENIZE function.
TOKENIZE returns a JSON array, which can be used with the MEMBER OF syntax to filter text:
SELECT * FROM articles
WHERE 'database' MEMBER OF (TOKENIZE(body, 'analyzer',
'[{"analysis": {"analyzer": "standard"}}]'));
SELECT
TOKENIZE(content, 'space') AS space_tokens,
TOKENIZE(content, 'ngram') AS ngram_tokens,
TOKENIZE(content, 'analyzer',
'[{"analysis": {"analyzer": "standard"}}]') AS std_tokens
FROM docs;