データ型の優先順位

Binlog 服务集成了 Prometheus 指标采集，默认端口为 2984，您可以通过 http://{ip}:2984/metrics 获取当前的指标情况，也可以通过 Prometheus Alertmanager 对特定的指标进行监控和告警。

宿主机状态

监控指标

告警规则参考

• CPU 告警

  - alert: HighCpuUsage
    expr: binlog_cpu_used_ratio > 0.9
    for: 10s
    labels:
      severity: critical
    annotations:
      summary: "CPU 使用率超过阈值"
      description: "节点 {{ $labels.ip }} 的 CPU 使用率超过 90%。"
  

• 内存告警

  - alert: HighMemUsage
    expr: binlog_mem_used_ratio > 0.8
    for: 10s
    labels:
      severity: critical
    annotations:
      summary: "内存使用率超过阈值"
      description: "节点 {{ $labels.ip }} 的内存使用率超过 80%。"
  

• 集群平均内存水位告警

  - alert: HighMemUsage
    expr: avg(binlog_mem_used_ratio) > 0.65
    for: 10s
    labels:
      severity: critical
    annotations:
      summary: "内存使用率超过阈值"
      description: "集群内存平均使用率超过 65%。"
  

• 机器负载告警

  - alert: HighLoad1
    expr: binlog_load1 > 16
    for: 10s
    labels:
      severity: critical
    annotations:
      summary: "机器负载 load1 超过阈值"
      description: "节点 {{ $labels.ip }} 的机器负载超过 16。"
  

• 磁盘告警

  - alert: HighDiskUsage
    expr: binlog_disk_used_ratio > 0.8
    for: 10s
    labels:
      severity: critical
    annotations:
      summary: "磁盘使用率超过阈值"
      description: "节点 {{ $labels.ip }} 的磁盘使用率超过 80%。"
  

OBM 状态

监控指标

告警规则参考

• Binlog 任务创建

  rules:
    - alert: BinlogCreateAlert
      expr: increase(binlog_create[1m]) > 0
      for: 1m
      labels:
        severity: info
      annotations:
        summary: "Binlog 开通通知"
        description: "租户 {{ $labels.ob_cluster_name }}.{{ $labels.tenant_name }} 开通了 Binlog 服务。"
  

• Binlog 任务释放

  rules:
  - alert: BinlogReleaseAlert
    expr: increase(binlog_release[1m]) > 0
    for: 1m
    labels:
      severity: info
    annotations:
      summary: "Binlog 释放通知"
      description: "租户 {{ $labels.ob_cluster_name }}.{{ $labels.tenant_name }} 释放了 Binlog 服务。"
  

• OBM 宕机

  - alert: OBMDownAlert
    expr: increase(binlog_manager_down_count[1m]) > 0
    for: 10s
    labels:
      severity: critical
    annotations:
      summary: "OBM 宕机告警"
      description: "节点 {{ $labels.host_name }} 宕机。"
  

OBI 状态

监控指标

告警规则参考

• 服务节点分配失败

  - alert: BinlogAllocateFailedAlert
    expr: increase(binlog_allocate_node_fail_count[1m]) > 0
    for: 10s
    labels:
      severity: critical
    annotations:
      summary: "分配节点失败告警"
      description: "集群 {{ $labels.ob_cluster_name }} 下的租户 {{ $labels.tenant_name }} 分配节点失败，需要立即处理。"
  

• OBI GTID 不一致

  - alert: GtidInconsistentFailedAlert
    expr: increase(binlog_instance_gtid_inconsistent_count[1m]) > 0
    for: 10s
    labels:
      severity: critical
    annotations:
      summary: "gtid 巡检不一致告警"
      description: "集群 {{ $labels.ob_cluster_name }} 下的租户 {{ $labels.tenant_name }} gtid 巡检不一致。"
  

• OBI 切主

  rules:
  - alert: MasterSwitchAlert
    expr: increase(binlog_instance_master_switch_count[1m]) > 0 
    for: 1m
    labels:
      severity: info
    annotations:
      summary: "切主通知"
      description: "集群 {{ $labels.ob_cluster_name }} 下的租户 {{ $labels.tenant_name }} 发生了切主事件。"
  

• OBI 频繁切主

  rules:
  - alert: MasterSwitchAlert
    expr: increase(binlog_instance_master_switch_count[1m]) > 2
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "频繁切主告警"
      description: "集群 {{ $labels.ob_cluster_name }} 下的租户 {{ $labels.tenant_name }} 发生了大量切主事件。"
  

• OBI 切主失败

  - alert: MasterSwitchFailedAlert
    expr: increase(binlog_instance_master_switch_failed_count[1m]) > 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "切主失败告警"
      description: "集群 {{ $labels.ob_cluster_name }} 下的租户 {{ $labels.tenant_name }} 发生了切主失败事件，需要立即处理。"
  

• OBI 无主

  - alert: NoMasterAlert
    expr: increase(binlog_instance_no_master_count[1m]) > 0
    for: 10s
    labels:
      severity: critical
    annotations:
      summary: "无主告警"
      description: "集群 {{ $labels.ob_cluster_name }} 下的租户 {{ $labels.tenant_name }} 无主，需要立即处理。"
  

• OBI 宕机

  - alert: InstanceDownAlert
    expr: changes(binlog_instance_down[15m]) > 0 or (binlog_instance_convert_delay==0)
    for: 10s
    labels:
      severity: critical
    annotations:
      summary: "instance 发生宕机"
      description: "实例 {{ $labels.instance_id }} 宕机。"
  

• OBI 自动拉起失败

  - alert: FailoverFailedAlert
    expr: increase(binlog_instance_failover_fail_count[1m]) > 0
    for: 10s
    labels:
      severity: critical
    annotations:
      summary: "自动拉起失败告警"
      description: "集群 {{ $labels.ob_cluster_name }} 下租户 {{ $labels.tenant_name }} 的 binlog 实例 { {$labels.instance_id} } 自动拉起失败，需要立即处理。"
  

OBI 性能

监控指标

告警规则参考

• 转换延迟

  - alert: ConversionDelayAlert
    expr: |
      (binlog_instance_convert_delay > 120000) and
      (binlog_instance_convert_fetch_rps < 5) and
      (binlog_instance_convert_storage_rps < 5) or
      (time() - binlog_instance_convert_checkpoint - binlog_instance_convert_delay) > 120000
    for: 10s
    labels:
      severity: critical
    annotations:
      summary: "转换延迟超过阈值"
      description: "集群 {{ $labels.ob_cluster_name }} 下租户 {{ $labels.tenant_name }} 的 binlog 实例 { {$labels.instance_id} } 转换延迟:{{ $binlog_instance_convert_delay }}s。"
  

• 订阅数超限

  - alert: InstanceDumpAlert
    expr: binlog_instance_dump_count > 100
    for: 1m
    labels:
      severity: warning
    annotations:
      summary: "Binlog 订阅数告警"
      description: "实例 {{ $labels.instance_id }} 的订阅数 {{$binlog_instance_dump_count}}。"
  
  - alert: InstanceDumpResolved
    expr: binlog_instance_dump_count <= 100
    for: 1m
    labels:
      severity: normal
    annotations:
      summary: "Binlog 订阅数告警恢复"
      description: "实例 {{ $labels.instance_id }} 的订阅数已下降至 100 以下。"