This topic describes the preparations before you archive logs.
Configure the concurrency of log archiving (optional)
Before you enable the archiving mode, you can configure the concurrency of log archiving to improve the log archiving speed of the tenant.
Log in to the database as the
systenant user or the tenant administrator of the user tenant.Run the appropriate statement to set the
log_archive_concurrencyconfiguration item.The
log_archive_concurrencyconfiguration item of a tenant specifies the total number of worker threads for log archiving. This configuration item takes effect immediately and does not require a restart of the OBServer node. The value range is [0, 100]. The default value is0, which indicates that OceanBase AI Database adapts the log archiving concurrency. We recommend that you use the default value.Notice
For small tenants (with 4 CPU cores or fewer), we recommend that you use the default value and do not modify the value of this configuration item.
The following methods are supported for setting the configuration item:
Adjust the log archiving concurrency for a specified tenant as the
systenant user.ALTER SYSTEM SET log_archive_concurrency = 10 TENANT = mysql_tenant;Adjust the log archiving concurrency for all tenants as the
systenant user.ALTER SYSTEM SET log_archive_concurrency = 10 TENANT = all_user;or
ALTER SYSTEM SET log_archive_concurrency = 10 TENANT = all;Note
TENANT = all_userhas the same meaning asTENANT = all. When you need the setting to take effect for all user tenants, we recommend that you useTENANT = all_user.TENANT = allwill be deprecated in the future.Adjust the log archiving concurrency for the current tenant as a user tenant.
ALTER SYSTEM SET log_archive_concurrency = 10;
For more information about the
log_archive_concurrencyconfiguration item, see log_archive_concurrency.
Configure the archiving destination
Before you start a log archiving task, you must run the ALTER SYSTEM command to set the LOG_ARCHIVE_DEST parameter. The system tenant does not contain user data and is a tenant used for cluster management. Backup and restore are not supported for the sys tenant. Therefore, you do not need to configure an archive destination for this tenant.
Configuring an archiving destination mainly involves setting the LOCATION, BINDING, and PIECE_SWITCH_INTERVAL attributes.
Considerations
When you configure the archiving destination, make sure that the archiving path for each tenant is an independent empty directory, and that the archiving paths for different tenants are different.
Procedure
Log in to the database as the tenant administrator of the
systenant or a user tenant.Note
The administrator user of a user tenant is the
rootuser.Configure the archive destination.
Configure an archive destination for a specified tenant as the system tenant.
ALTER SYSTEM SET LOG_ARCHIVE_DEST='LOCATION=archive_path [BINDING=archive_mode] [PIECE_SWITCH_INTERVAL=piece_switch_interval]' TENANT = tenant_name;Configure the archive destination for the current tenant as a user tenant.
ALTER SYSTEM SET LOG_ARCHIVE_DEST='LOCATION=archive_path [BINDING=archive_mode] [PIECE_SWITCH_INTERVAL=piece_switch_interval]';
The following table describes the attributes.
Configure
LOCATION(required)The
LOCATIONattribute specifies the archiving path. Currently, OceanBase AI Database supports the following archive media: Azure Blob, AWS S3, and S3-compatible object storage such as Alibaba Cloud OSS, Huawei OBS, Google GCS, and Tencent Cloud COS. Some archive media may need to meet certain basic requirements before they can be used. For more information about the requirements for each archive medium, see the Backup media requirements section in Overview of physical backup and restore.AWS S3Object storage compatible with the S3 protocolAzure BlobYou can specify the
delete_modeparameter to configure the mode for deleting archive files when you use AWS S3 as the archive medium. You can also specify thechecksum_typeparameter to indicate the checksum algorithm for verifying the integrity of archive files. The supported values aremd5andcrc32. If you do not explicitly specify thechecksum_typeparameter, the default value ismd5.For more information about the
delete_modeandchecksum_typeparameters, see SET LOG_ARCHIVE_DEST.Notice
When an object storage service is used as the archive medium, separate the parameters for the object storage path with the
&character. Make sure that the parameter values contain only uppercase and lowercase letters, numbers,/-_$+=, and wildcard characters. If you enter other characters, the setting may fail.Here are some examples of using S3 as the archive medium:
When using S3 as the archive medium, set the archiving path for the tenant
mysql_tenantin the system tenant, configure the cleanup mode asdelete, and specify the checksum algorithm ascrc32.obclient> ALTER SYSTEM SET LOG_ARCHIVE_DEST='LOCATION=s3://oceanbase-test-bucket/backup/archive?host=s3.<region>.amazonaws.com&access_id=******&access_key=******&s3_region=******&delete_mode=delete&checksum_type=crc32' TENANT = mysql_tenant;When using S3 as the archive medium, set the archiving path for the current tenant in a user tenant, configure the cleanup mode as
delete, and specify the checksum algorithm ascrc32.obclient> ALTER SYSTEM SET LOG_ARCHIVE_DEST='LOCATION=s3://oceanbase-test-bucket/backup/archive?host=s3.<region>.amazonaws.com&access_id=******&access_key=******&s3_region=******&delete_mode=delete&checksum_type=crc32';
In the preceding examples,
s3://indicates that AWS S3 is used as the archive medium. The bucket name isoceanbase-test-bucket, and the path in the bucket is/backup/archive. The parameters are separated from the path by using?, and the parameters are separated by using&.hostspecifies the domain name of the Amazon S3 service.access_idandaccess_keyspecify the access keys of the AWS service.s3_regionis required when S3 is used as the archive medium and specifies the region where the S3 bucket is located. The cleanup mode isdelete, and the CRC32 algorithm is used to verify the integrity of archive files.For more information about the AWS S3 path format, see Appendix: S3 path format.
Most object storage services are compatible with the S3 protocol. Object storage services that are compatible with the S3 protocol and meet the requirements of OceanBase AI Database (the object storage must be compatible with the behaviors of the S3 APIs that implement OceanBase system calls) can serve as the archive media for OceanBase AI Database. You can access compatible object storage services such as OBS, GCS, and COS through the S3 protocol.
For object storage services such as OBS, GCS, and COS accessed through the S3 protocol, you can also specify the
checksum_typeparameter to indicate the checksum algorithm for verifying the integrity of archive files. Onlymd5is supported. If you do not explicitly specify thechecksum_typeparameter, the default value ismd5. For more information about thechecksum_typeparameter, see SET LOG_ARCHIVE_DEST.Notice
- When an object storage service is used as the archive medium, separate the parameters for the object storage path with the
&character. Make sure that the parameter values contain only uppercase and lowercase letters, numbers,/-_$+=, and wildcard characters. If you enter other characters, the setting may fail. - Some S3-compatible object storage services may not support the default URL encoding method of the AWS S3 SDK. When you access them through the
s3://protocol, if object storage-related error codes such as 9129 or 9026 occur, we recommend that you set the cluster-level configuration item ob_storage_s3_url_encode_type to specify that AWS S3 uses the RFC 3986 standard for URL encoding when sending requests. For detailed configuration methods, see the Procedure and example for using COS as the archive medium section below.
The following example shows how to configure OBS as the archive medium:
When using OBS as the archive medium, set the archiving path for the tenant
mysql_tenantin the system tenant.obclient> ALTER SYSTEM SET LOG_ARCHIVE_DEST='LOCATION=s3://oceanbase-test-bucket/backup/archive?host=obs.****.myhuaweicloud.com&access_id=****&access_key=****' TENANT = mysql_tenant;When using OBS as the archive medium, set the archiving path for the current tenant in a user tenant.
obclient> ALTER SYSTEM SET LOG_ARCHIVE_DEST='LOCATION=s3://oceanbase-test-bucket/backup/archive?host=obs.****.myhuaweicloud.com&access_id=****&access_key=****';
In the preceding examples,
s3://indicates that the object storage path is accessed through the S3 protocol. The bucket name isoceanbase-test-bucket, and the path in the bucket is/backup/archive. The parameters are separated from the path by using?, and the parameters are separated by using&.hostspecifies the domain name of the OBS service.access_idandaccess_keyspecify the access keys of the OBS service.The following example shows how to configure GCS as the archive medium:
When using GCS as the archive medium, set the archiving path for the tenant
mysql_tenantin the system tenant.obclient> ALTER SYSTEM SET LOG_ARCHIVE_DEST='LOCATION=s3://oceanbase-test-bucket/backup/archive?host=https://storage.googleapis.com&access_id=****&access_key=****' TENANT = mysql_tenant;When using GCS as the archive medium, set the archiving path for the current tenant in a user tenant.
obclient> ALTER SYSTEM SET LOG_ARCHIVE_DEST='LOCATION=s3://oceanbase-test-bucket/backup/archive?host=https://storage.googleapis.com&access_id=****&access_key=****';
In the preceding examples,
s3://indicates that the object storage path is accessed through the S3 protocol. The bucket name isoceanbase-test-bucket, and the path in the bucket is/backup/archive. The parameters are separated from the path by using?, and the parameters are separated by using&.hostspecifies the domain name of the GCS service.access_idandaccess_keyspecify the access keys of the GCS service.The procedure and example for using COS as the archive medium are as follows:
View the value of the cluster-level configuration item
ob_storage_s3_url_encode_type.obclient> SHOW PARAMETERS LIKE '%ob_storage_s3_url_encode_type%';The result is as follows:
+-------+----------+----------------+----------+-------------------------------+-----------+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+---------+---------+-------------------+ | zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level | +-------+----------+----------------+----------+-------------------------------+-----------+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+---------+---------+-------------------+ | zone1 | observer | 172.xx.xxx.xxx | 2882 | ob_storage_s3_url_encode_type | NULL | default | Determines the URL encoding method for S3 requests."default": Uses the S3 standard URL encoding method."compliantRfc3986Encoding": Uses URL encoding that adheres to the RFC 3986 standard. | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | +-------+----------+----------------+----------+-------------------------------+-----------+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+---------+---------+-------------------+ 1 row in setIf the value is
default, change it tocompliantRfc3986Encoding.obclient> ALTER SYSTEM SET ob_storage_s3_url_encode_type='compliantRfc3986Encoding';For more information about this configuration item, see ob_storage_s3_url_encode_type.
Set the archiving path and specify the mode for deleting archive files by using the
delete_modeparameter.Use COS as the archive medium. Set the archiving path for the tenant
mysql_tenantin the sys tenant and configure thedeletemode for deleting archive files:obclient> ALTER SYSTEM SET LOG_ARCHIVE_DEST='LOCATION=s3://oceanbase-test/backup/archive?host=cos.ap-xxxx.myqcloud.com&access_id=***&access_key=***&delete_mode=delete' TENANT = mysql_tenant;Use COS as the archive medium. Set the archiving path for the current tenant and configure the
deletemode for deleting archive files:obclient> ALTER SYSTEM SET LOG_ARCHIVE_DEST='LOCATION=s3://oceanbase-test/backup/archive?host=cos.ap-xxxx.myqcloud.com&access_id=***&access_key=***&delete_mode=delete';In these examples,
s3://indicates that the object storage path is accessed through the S3 protocol. The bucket name isoceanbase-testand the path in the bucket is/backup/archive. The parameters are separated from the path by using?, and the parameters are separated by using&. Thehostparameter specifies the endpoint of COS without the bucket name. Theaccess_idandaccess_keyparameters specify the access keys of the COS service. Thedelete_modeparameter is set todelete.Notice
The archive access protocol cannot be changed during archiving.
Azure Blob also supports the
delete_modeandchecksum_typeparameters. Thedelete_modeparameter specifies the cleanup mode for the archive files. If you do not explicitly configure thedelete_modeparameter, thedeletemode is used by default. Thechecksum_typeparameter specifies the verification algorithm for verifying the integrity of the archive files. The supported values aremd5andno_checksum. If you do not explicitly configure thechecksum_typeparameter, the default value ismd5.For more information about the parameters, see SET LOG_ARCHIVE_DEST.
Here are some examples:
Notice
- When you use an object storage service as the archive medium, the parameters of the object storage path are separated by
&. Make sure that the parameter values contain only uppercase and lowercase English letters, numbers,/-_$+=, and wildcard characters. If you enter other characters, the configuration may fail. - When you use Azure Blob as the archive medium, the prefix of the
hostparameter must be specified ashttp://orhttps://. By default, Azure Blob enables secure transfer. Therefore, it is accessed by usinghttps://. If you want to access it by usinghttp://, you must disable secure transfer for Azure Blob. For more information, see Azure Blob official documentation.
When you use Azure Blob as the archive medium, set the archive path for the
mysql_tenanttenant in the system tenant. The cleanup mode isdelete, and the verification algorithm ismd5.obclient> ALTER SYSTEM SET LOG_ARCHIVE_DEST='LOCATION=azblob://oceanbase-test-bucket/backup/archive?host=http://****.blob.core.windows.net&access_id=****&access_key=****&delete_mode=delete&checksum_type=md5' TENANT = mysql_tenant;When you use Azure Blob as the archive medium, set the archive path for the current tenant in the user tenant. The cleanup mode is
delete, and the verification algorithm ismd5.obclient> ALTER SYSTEM SET LOG_ARCHIVE_DEST='LOCATION=azblob://oceanbase-test-bucket/backup/archive?host=http://****.blob.core.windows.net&access_id=****&access_key=****&delete_mode=delete&checksum_type=md5';
In the preceding examples,
azblob://indicates that Azure Blob is used as the archive medium. The bucket name isoceanbase-test-bucket, and the path in the bucket is/backup/archive. The path and parameters are separated by?, and the parameters are separated by&.hostspecifies the host address of the bucket, which can be obtained by concatenating the container attribute or connection string of the storage account.access_idspecifies the name of the storage account.access_keyspecifies the access key of Azure Blob. The cleanup mode isdelete, and the integrity of the archive files is verified by using the MD5 algorithm.Configure the
BINDINGattribute (optional)The
BINDINGattribute specifies the priority mode between archiving and business. The attribute supports theOptionalandMandatorymodes. If you do not configure the attribute, the default valueOptionaltakes effect.The
Optionalmode indicates that business takes priority. In this mode, if the archiving speed falls behind the log generation speed, logs may be recycled before they are archived, causing a log stream interruption.The
Mandatorymode indicates that archiving takes priority. In this mode, if archiving falls behind the data write speed, data write may fail.
Here is an example of setting the archiving path and the
BINDINGattribute for themysql_tenanttenant in the sys tenant when you use S3 as the archive medium:obclient> ALTER SYSTEM SET LOG_ARCHIVE_DEST='LOCATION=s3://oceanbase-test-bucket/backup/archive?host=s3.<region>.amazonaws.com&access_id=******&access_key=****** BINDING=Optional' TENANT = mysql_tenant;Here is an example of setting the archiving path and the
BINDINGattribute for the current tenant when you use S3 as the archive medium:obclient> ALTER SYSTEM SET LOG_ARCHIVE_DEST='LOCATION=s3://oceanbase-test-bucket/backup/archive?host=s3.<region>.amazonaws.com&access_id=******&access_key=****** BINDING=Optional';Configure the
PIECE_SWITCH_INTERVALattribute (optional)The
PIECE_SWITCH_INTERVALattribute specifies the interval for switchingpiece. The value range is[1d, 7d]. If you do not configure the attribute, the default value1dtakes effect.Here is an example of setting the archiving path and the
PIECE_SWITCH_INTERVALattribute to switch a logpieceevery other day for themysql_tenanttenant in the sys tenant when you use S3 as the archive medium:obclient> ALTER SYSTEM SET LOG_ARCHIVE_DEST='LOCATION=s3://oceanbase-test-bucket/backup/archive?host=s3.<region>.amazonaws.com&access_id=******&access_key=****** BINDING=Optional PIECE_SWITCH_INTERVAL=1d' TENANT = mysql_tenant;Here is an example of setting the archiving path and the
PIECE_SWITCH_INTERVALattribute to switch a logpieceevery other day for the current tenant when you use S3 as the archive medium:obclient> ALTER SYSTEM SET LOG_ARCHIVE_DEST='LOCATION=s3://oceanbase-test-bucket/backup/archive?host=s3.<region>.amazonaws.com&access_id=******&access_key=****** BINDING=Optional PIECE_SWITCH_INTERVAL=1d';
After the configuration is completed, you can query the parameter settings in views.
The
systenant can query the parameter settings in theCDB_OB_ARCHIVE_DESTview, and a user tenant can query the parameter settings in theDBA_OB_ARCHIVE_DESTview. For more information about how to query the parameter settings, see View archive parameters.
Considerations and descriptions after the configuration is completed
After the
LOG_ARCHIVE_DESTparameter is successfully configured, the system creates aformatfile in the directory where the specified backup destination is located. Theformatfile is used to verify the validity of the backup destination and ensure the integrity of data in the destination. Therefore, note the following considerations when you configure the archiving destination:If the
formatfile does not exist, the specified directory must be empty. Otherwise, an error-9080is returned, indicating that the file does not exist.If the
formatfile already exists, its content must pass the verification for the specified parameter to take effect. Otherwise, an error-9081is returned, indicating that the file does not match. The content verification of theformatfile mainly checks whether the cluster, tenant, and backup destination type specified in the file are consistent with those in the current operation.If the
formatfile does not exist or its content fails the verification during backup task execution, the task will fail.
After the archiving destination is successfully configured, incremental configuration is not supported. For example, assume the archive path
/oceanbase-test-bucket/backup/archiveunder S3 has theBINDINGattribute set toMandatoryand thePIECE_SWITCH_INTERVALattribute set to1d. If you need to updatePIECE_SWITCH_INTERVALto2dwhile retaining theMANDATORYconfiguration forBINDING, you still need to specify the values for other attributes in the command. Otherwise, unspecified attributes will use their default values.You need to execute the following statements again for the modification.
Modify a specified tenant in the sys tenant
obclient> ALTER SYSTEM SET LOG_ARCHIVE_DEST = 'LOCATION=s3://oceanbase-test-bucket/backup/archive?host=s3.<region>.amazonaws.com&access_id=******&access_key=****** BINDING=Mandatory PIECE_SWITCH_INTERVAL=2d' TENANT = mysql_tenant;Modify the current tenant in a user tenant
obclient> ALTER SYSTEM SET LOG_ARCHIVE_DEST = 's3://oceanbase-test-bucket/backup/archive?host=s3.<region>.amazonaws.com&access_id=******&access_key=****** BINDING=Mandatory PIECE_SWITCH_INTERVAL=2d';
After the archiving destination is set successfully, if the user changes the key information of the object storage due to security or other reasons, you need to update the
access_idandaccess_keyinformation for the configured archiving path. For more information, see Update object storage key information.
(Optional) Configure archive lag
OceanBase AI Database uses the tenant-level configuration item archive_lag_target to control the latency of log archiving for the tenant. This configuration item supports units of ms, s, m, and h. The default value is 2 minutes, which specifies the maximum interval between two consecutive archive I/O operations and ensures that online logs can be archived in a timely manner, thereby reducing the risk of data loss.
OceanBase AI Database archives logs by log stream. If a log stream has logs written to it and the time since its last archive exceeds the interval specified by the archive_lag_target configuration item, the system triggers an archive action for that log stream to archive all unarchived logs in the log stream.
For example, if the configuration item is set to archive_lag_target='120s' (2 minutes), each log stream attempts to trigger an archive action every 2 minutes (unless other conditions are met, such as the archive buffer being full), ensuring that the interval between two consecutive archive actions of the log stream is less than 2 minutes. If the archive_lag_target configuration item is set to 0, real-time archiving is achieved.
For more information about the archive_lag_target configuration item, see archive_lag_target.
Prerequisites
Before you modify the value of the archive_lag_target configuration item, make sure that the archive destination has been configured. If the archive destination is not configured, the system will return an error message indicating that the archive destination has not been set when you try to modify the value of the archive_lag_target configuration item.
Considerations
For archive media such as S3 or other object storage services compatible with the S3 protocol, the value of the
archive_lag_targetconfiguration item cannot be less than60s. If you set the value to less than60s, the system will return an error.We recommend that you set the
archive_lag_targetconfiguration item to a reasonable value to avoid frequent I/O operations that may affect system performance, especially when object storage is used. A high value may fail to meet the timeliness requirement for data recovery. Therefore, you need to strike a balance between the RPO and the performance of the archive media based on your business needs.
Procedure
Log in to the database as the tenant administrator of the
systenant or a user tenant.Based on your needs, execute the appropriate statement to configure the archive lag.
Here are some examples:
Configure the archive lag for a specified tenant when you are in the
systenant.obclient> ALTER SYSTEM SET archive_lag_target = '120s' TENANT = mysql_tenant;Notice
When you configure the
archive_lag_targetparameter, you cannot useTENANT = all_userto specify all user tenants in thesystenant.Configure the archive lag for the current tenant when you are in a user tenant.
obclient> ALTER SYSTEM SET archive_lag_target = '120s';