Description
In a Linux system, an inode stores the metadata of a file, including the file size, document owner ID, read/write permissions, modification timestamp, and block location.
An inode is an index of a file, and it also occupies disk space. If too many small files are created, it may occupy excessive inodes. This alert is triggered when the inode usage exceeds 80%.
Principle
Parameter |
Value |
|---|---|
| Monitoring metric | file_inode_usage |
| Source of the metric | Host basic monitoring, collected by node_exporter. Inode usage is similar to IUsed / Inodes in df -i. |
| Metrics to be collected | node_filesystem_files_free, node_filesystem_files |
| Monitoring expression | round(100 - 100 * sum(node_filesystem_files_free{@LABELS}) by (@GBLABELS) / sum(node_filesystem_files{@LABELS}) by (@GBLABELS)) |
| Collection interval | 1 second |
Rule information
Monitoring expression |
Meaning of the monitoring metric |
Default threshold (percent) |
Detection interval |
Elimination interval |
|---|---|---|---|---|
| file_inode_usage > 80 | Inode usage | 80% | 10 seconds | 5 minutes |
Alert information
Alert trigger method |
Alert level |
Scope |
|---|---|---|
| Monitoring expression | Severe | Server |
Alert template
Alert overview
- Template: ${alarm_target} ${alarm_name}
- Example: svr_ip=xxx.xxx.xxx.xxx High inode usage on the server
Alert details
- Template: Cluster: ${ob_cluster_name}, Host: ${host}, Alert: ${alarm_name}. Inode usage ${value_shown} exceeds ${alarm_threshold} %.
- Example: Cluster: obcluster-1, Host: xxx.xxx.xxx.xxx, Alert: High inode usage on the server. Inode usage 95 % exceeds 80 %.
Alert recovery
- Template: Alert: ${alarm_name}, Server inode usage: ${value_shown}
- Example: Alert: High inode usage on the server, Server inode usage: 60 %
Impact on the system
After all inodes are used, the system cannot create files. This is fatal for database writes, leading to transaction commit failures, log synchronization issues, and OBServer unavailability.
Possible causes
Too many small files occupy a large number of inodes.
Procedure
View the directory that uses the most inodes.
df -iView the directory that contains the most files.
for i in /target-path/* ; do echo $i; find $i | wc -l | sort -nr; doneDelete garbage files.
Notice
- The system may contain hidden files, soft links, and hard links, which all occupy inodes. After you confirm that the files are garbage, delete them.
- If there are too many files, an error may be returned when you run the
rm -rf *command. In this case, you can run the batch deletion command:ls -a | xargs -n 1000 rm -rf.