When a LogService node (LN) fails or requires vertical scaling, you can replace it. LogService performs node replacement by using LS-CTRL management commands. The basic procedure is as follows: start a new node and add it to the target Zone by using the add ln command, and then delete the node to be replaced by using the delete ln command.
Procedure
Start an available node.
Use the
ls_ctrltool to execute theadd lncommand to add the new node to the Zone where the node to be replaced is located.ls_ctrl -H <HOST_ADDRESS> add ln <host:port> <az>Here,
<host:port>is the address of the node to be added, and<az>is the name of the target Zone.For more information about how to add a node, see Add a LogService node.
Use the
ls_ctrltool to execute thedelete lncommand to delete the node to be replaced.ls_ctrl -H <HOST_ADDRESS> delete ln <ln>Here,
<ln>is the ID or address (host:port) of the node to be deleted.Use the
ls_ctrltool to execute theshow lncommand and theshow meta --table-name t_ln_info_historycommand to check whether the node to be replaced can be safely deleted:ls_ctrl -H <HOST_ADDRESS> show ln ls_ctrl -H <HOST_ADDRESS> show meta --table-name t_ln_info_historyIf the information about the node to be replaced cannot be found by using the
show lncommand but can be found by using theshow meta --table-name t_ln_info_historycommand, the node can be safely deleted.
Note
In the preceding commands, HOST_ADDRESS is the HTTP address (including the port number) of any LogService node, for example, 127.0.0.1:8080. For more information about LS-CTRL commands, see Overview of LS-CTRL.
Deletion mechanism
The
delete lncommand marks the node asDELETINGand drives LogServiceManager (LM) to initiate a migration task to move the stream replicas on the node to other available nodes.If a log stream's member list no longer contains an LN in the
DELETINGstate, the LN is not the destination of migration for any log stream, and no log streams are being created, LM considers that the LN can be deleted safely.If LM detects that an LN has been in the
Inactivestate for a long time, that is, the current time is later than the time obtained by adding theoffline_timeint_ln_infoand the value of thenode_permanent_offline_time_secparameter, the system considers that the node cannot recover and generates a migration task to move the log stream replicas from this LN to another available LN in the same Zone.
Monitor deletion progress
You can monitor the deletion progress of a node by using the following methods:
Use the
show lncommand to check whether the node is in theDELETINGstate.Use the
show metacommand to view the internal tablet_stream_replica_task, which records migration tasks. After a node is marked asDELETING, LM generates a migration task for the node, and the sourcesrc_lnis the node itself.ls_ctrl -H <HOST_ADDRESS> show meta --table-name t_stream_replica_taskAfter the migration task is completed, you can use the
show metacommand to view thet_stream_replica_task_historytable of migration and replication tasks. Each row in this table records a completed migration and replication task. Thestatuscolumn indicates whether the migration task is successfully executed, and thecommentcolumn records the reason for a task failure.ls_ctrl -H <HOST_ADDRESS> show meta --table-name t_stream_replica_task_historyAfter all replicas on the LN are migrated out, you can use the
show metacommand to view thet_ln_infotable and its history tablet_ln_info_history. If the record of the node does not exist int_ln_infobut exists int_ln_info_history, the LN is deleted from the cluster.ls_ctrl -H <HOST_ADDRESS> show meta --table-name t_ln_info ls_ctrl -H <HOST_ADDRESS> show meta --table-name t_ln_info_history
Considerations
- If a log stream remains in the
CREATINGstate and theinitial_member_listcontains the node to be deleted, the node cannot be removed.
References
For more information about node-related operations, see the following topics: