In the shared storage architecture, the OBServer process uses object storage to persist business data, and LogService uses object storage to persist log data. The AccessKeys (AKs) required by LogService to access object storage are centrally managed by the cluster management node (Log Manager, LM). This topic describes how to use the ls_ctrl command-line tool to rotate the AK used by LogService to access object storage online.
The update of object storage access credentials on the OBServer side is independent of the rotation of the LogService AK/SK. For more information, see Update object storage access credentials.
Prerequisites
Before performing the rotation, ensure that a new AK has been created in object storage and that you have the necessary permissions to access object storage. For permission requirements, see Object storage overview.
Procedure
Run the following command to trigger the rotation:
ls_ctrl -H <HOST_ADDRESS> rotate-ak \ -i <ACCESS_KEY_ID> \ -k <ACCESS_KEY_SECRET>Here is an example of the output:
{"msg":"ok","data":{"new_param_version":43}}The
data.new_param_versionin the output represents the parameter version after this rotation, which is used to confirm that the entire cluster has switched to the new AK in the next step.Confirm that the entire cluster has switched to the new AK:
The rotation takes effect asynchronously. Run the following command repeatedly to confirm that all nodes have switched to the new AK:
ls_ctrl -H <HOST_ADDRESS> param-version-statusCheck the parameter version for each node in the output. When the version number of all nodes reaches the
new_param_versionreturned in Step 1, the entire cluster has switched to the new AK.Note
If you accidentally lose the
new_param_version(for example, due to a terminal closure or machine change), you can re-executerotate-akwith exactly the same AK: the server will recognize it as an idempotent retransmission, return the current parameter version, and continue the process.(Optional) Revoke the old AK:
After confirming that the entire cluster has switched to the new AK, you can revoke the old AK in the object storage service management console or by using its CLI tool to complete the rotation. For more information, see the official documentation of your object storage service provider.
Note
In the preceding command, HOST_ADDRESS is the HTTP address (including the port number) of any LogService node, for example, 127.0.0.1:8080. For more information about LS-CTRL commands, see LS-CTRL overview.
Common errors and troubleshooting
Symptom |
Cause |
Solution |
|---|---|---|
The rotate-ak command returns 403 |
The new AK lacks the object storage permissions required by LogService (including batch deletion). | The response body contains failed_step, which identifies the failed steps (for example, put, head, get, list, and delete_batch). Complete the permission granting on the object storage side and retry. |
| LM is down or restarted during rotation | LM needs to load the checkpoint from object storage during startup. In this case, the old AK has been disabled or deleted. | Re-enable the old AK on the object storage side. After LM is started and parameter synchronization is complete, confirm again that the entire cluster has switched to the new AK. |
| Authentication error after the old AK is disabled | Some nodes (such as newly added nodes) have not switched to the new AK. | Re-enable the old AK immediately. After confirming that the entire cluster has switched to the new AK, disable the old AK again. |
Considerations
- This feature is supported only for clusters whose
min_oblogservice_versionis 1.3.1 or later. Clusters with an earlier version rejectrotate-akandparam-version-statusrequests. - A single
rotate-akoperation replaces both the AccessKey ID and AccessKey Secret. It does not support replacing only one of them. - The new AK Secret is passed in by using command-line parameters. Make sure that the Secret does not enter the shell history or log collection. You are advised to temporarily store the Secret by using methods such as
read -sor environment variables, and do not post commands containing the Secret in shared chat tools, wikis, or ticket systems.