In a shared storage architecture, the observer process uses object storage to persist business data, and LogService uses object storage to persist log data. When you need to update the object storage access credentials (for example, due to periodic rotation of AK/SK or credential leakage), you can use the ALTER SYSTEM statement to complete the credential update. This topic describes how to update the observer process's access credentials for object storage.
The AK/SK rotation for LogService is independent of the observer process credential update. For specific operations, see Rotate LogService access credentials.
Prerequisites
Before updating the access credentials for data storage and log storage, you must create new AK/SK on the object storage service and ensure the new credentials have read/write permissions for the target bucket. For permission requirements, see Object storage overview.
Update data storage access credentials
Data storage access credentials are directly held by the compute node OBServer and point to the object storage bucket where the data storage is located. They are used to read from and write baseline data and minor compaction data. These credentials are initially configured during cluster creation using the SHARED_STORAGE_INFO parameter.
If you later need to change the access key for an object storage bucket, you can update it online using the following statement without restarting the cluster: ALTER SYSTEM CHANGE SHARED_STORAGE_DEST.
Notice
The ALTER SYSTEM CHANGE SHARED_STORAGE_DEST statement can only be executed by the system tenant (i.e., the sys tenant).
Procedure
Log in to the
systenant of the cluster as therootuser.Note that you must specify the corresponding options in the following sample code based on your actual database configurations.
obclient -h10.xx.xx.xx -P2883 -uroot@sys#obdemo -p***** -AQuery the
oceanbase.DBA_OB_ZONE_STORAGEview to view the object storage information configured for the current cluster.obclient [(none)]> SELECT * FROM oceanbase.DBA_OB_ZONE_STORAGE;Update the object storage access credentials:
ALTER SYSTEM {ALTER|CHANGE|MODIFY} SHARED_STORAGE_DEST PATH [=] 'path' SET ACCESS_INFO [=] 'access_info';The parameters are described as follows:
PATH: the path of the object storage. For S3-compatible object storage, the format iss3://<bucket>/<prefix>?host=<endpoint>. For Azure Blob, the format isazblob://<container>/<prefix>?host=<endpoint>. It must be consistent with the object storage path configured for the current cluster.ACCESS_INFO: the new access credentials, in the format ofaccess_id=<new_access_id>&access_key=<new_access_key>.
Here is an example:
obclient [(none)]> ALTER SYSTEM CHANGE SHARED_STORAGE_DEST PATH='s3://xxx?host=cn-hangzhou-alipay-b.oss-cdn.aliyun-inc.com' SET ACCESS_INFO='access_id=xxx&access_key=xxx';After the update is successful, query the
oceanbase.DBA_OB_ZONE_STORAGEview again to confirm the update result. ASTATEfield value ofCHANGINGindicates the rotation is in progress, and a value ofCHANGEDindicates the rotation is complete.
Update log service access credentials
The log service access credentials are held by the compute node OBServer through the LibPalf client library and point to the object storage bucket used by the log service. The OBServer uses these credentials to perform service discovery (read AccessPoint) and communicate with the log service.
These credentials are configured and managed through the cluster parameter logservice_access_point. You can update the credential information by modifying this parameter.
Notice
- The
logservice_access_pointparameter only allows modification of theaccess_idandaccess_keyfields. Other fields (such ashost,s3_region, etc.) cannot be modified; otherwise, the statement execution will report an error. - The
logservice_access_pointparameter cannot be set using startup parameters (-o). It can only be modified using theALTER SYSTEM SETstatement, or automatically set by the system during cluster initialization (bootstrap).
Procedure
Log in to the
systenant of the cluster as therootuser.Note that you must specify the corresponding options in the following sample code based on your actual database configurations.
obclient -h10.xx.xx.xx -P2883 -uroot@sys#obdemo -p***** -AExecute the following statement to view the current value of the
logservice_access_pointparameter.obclient [(none)]> SHOW PARAMETERS LIKE 'logservice_access_point';Note
The value of this parameter is stored in encrypted form. Therefore, the
SHOW PARAMETERSstatement displays the encrypted value.Execute the following statement to update the access credentials in the
logservice_access_pointparameter (only modify theaccess_idandaccess_keyfields, and keep other fields unchanged).The statement is as follows:
ALTER SYSTEM SET logservice_access_point = 's3://<bucket>/<prefix>?host=<endpoint>&logservice_cluster_id=<cluster_id>&s3_region=<region>&max_iops=<iops>&max_bandwidth=<bandwidth>&access_id=<new_access_id>&access_key=<new_access_key>';Here is an example:
obclient [(none)]> ALTER SYSTEM SET logservice_access_point = 's3://test/logservice?host=http://10.0.0.1:9000&logservice_cluster_id=20260617&s3_region=cn-north-1&max_iops=10000&max_bandwidth=1GB&access_id=<NEW_AK>&access_key=<NEW_SK>';Notice
When modifying this parameter, the system uses the new AK/SK to access the object storage to verify its validity. If the new credentials do not exist or lack permissions, the statement execution will be denied.
After the update is successful, confirm the result.
Re-execute the
SHOW PARAMETERS LIKE 'logservice_access_point';statement to confirm that the configuration has been updated.Confirm in the observer process log that the log storage credentials have been refreshed.
grep "rotate_object_store_ak" observer.log | tail -5
(Optional) Clean up old access credentials.
After the new credentials take full effect, you can delete the old credentials from the object storage service. Before deleting, confirm the following:
- All OBServer nodes have completed credential updates and the cluster is running normally.
- You have continuously observed for at least one config reload cycle (about 10 seconds), and success logs for the update appear in the logs of all nodes.
Notice
If you delete the old credentials before the update is complete, OBServer nodes still using the old credentials will be unable to access the object store, which may cause node exceptions or even prevent the cluster from starting (because the
observer.config.binfile still contains the old credentials, causing load failure upon restart).
Common errors and troubleshooting
Errors that may occur when you update the logservice_access_point parameter and their handling methods are as follows:
Error message |
Cause |
Solution |
|---|---|---|
logservice_access_point: libpalf env not initialized |
The libpalf environment is not initialized (the observer is starting or the cluster runs in non-shared storage mode). | Retry after the observer is started. |
only access_id and access_key can be changed |
A field other than AK or SK is modified (for example, host, path, or region). |
Check the new value and make sure that only access_id and access_key are modified. |
new access_id/access_key validation failed |
The new AK/SK cannot access the object storage. | Make sure that the new credentials are created on the object storage service and have read permissions. |
logservice_access_point cannot be setted by -o |
An attempt is made to set the parameter by using the startup parameter -o. |
You can set the parameter only by using the ALTER SYSTEM SET statement or during bootstrap. |