Starting from V1.6.0, obd supports the deployment of Prometheus and Grafana. If you want to add a GUI monitoring system to an existing cluster, you can follow the steps in this topic.
This topic describes three scenarios. You can choose the scenario that applies to your cluster.
Note
The configurations in this topic are for reference only. For more information about the configurations, go to the /usr/obd/example directory and view the sample configurations of the corresponding components.
Scenario 1: The existing cluster does not have OBAgent deployed
If the existing cluster does not have OBAgent deployed, to add a GUI monitoring, you need to create a new cluster that contains the obagent, prometheus, and grafana components.
In this scenario, OBAgent is configured separately to collect monitoring information from OceanBase Database. In the configuration file, you need to declare that Prometheus depends on the obagent component and Grafana depends on the prometheus component.
Here is an example of the configuration file:
# user:
# username: your username
# password: your password if need
# key_file: your ssh-key file path if need
# port: your ssh port, default 22
# timeout: ssh connection timeout (second), default 30
obagent:
servers:
# Please don't use hostname, only IP can be supported
- 10.10.10.1
- 10.10.10.2
- 10.10.10.3
global:
# The working directory for obagent. obagent is started under this directory. This is a required field.
home_path: /home/admin/obagent
# The port of monitor agent. The default port number is 8088.
monagent_http_port: 8088
# The port of manager agent. The default port number is 8089.
mgragent_http_port: 8089
# Log path. The default value is log/monagent.log.
log_path: log/monagent.log
# The log level of manager agent.
mgragent_log_level: info
# The total size of manager agent.Log size is measured in Megabytes. The default value is 30M.
mgragent_log_max_size: 30
# Expiration time for manager agent logs. The default value is 30 days.
mgragent_log_max_days: 30
# The maximum number for manager agent log files. The default value is 15.
mgragent_log_max_backups: 15
# The log level of monitor agent.
monagent_log_level: info
# The total size of monitor agent.Log size is measured in Megabytes. The default value is 200M.
monagent_log_max_size: 200
# Expiration time for monitor agent logs. The default value is 30 days.
monagent_log_max_days: 30
# The maximum number for monitor agent log files. The default value is 15.
monagent_log_max_backups: 15
# Username for HTTP authentication. The default value is admin.
http_basic_auth_user: admin
# Password for HTTP authentication. The default value is a random password.
# http_basic_auth_password: ******
# Monitor password for OceanBase Database. The default value is empty. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the ocp_agent_monitor_password in oceanbase-ce.
monitor_password: ******
# The SQL port for observer. The default value is 2881. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the mysql_port in oceanbase-ce.
sql_port: 2881
# The RPC port for observer. The default value is 2882. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the rpc_port in oceanbase-ce.
rpc_port: 2882
# Cluster name for OceanBase Database. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the appname in oceanbase-ce.
cluster_name: obcluster
# Cluster ID for OceanBase Database. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the cluster_id in oceanbase-ce.
cluster_id: 1
# The redo dir for Oceanbase Database. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the redo_dir in oceanbase-ce.
ob_log_path: /home/admin/observer/store
# The data dir for Oceanbase Database. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the data_dir in oceanbase-ce.
ob_data_path: /home/admin/observer/store
# The work directory for Oceanbase Database. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the home_path in oceanbase-ce.
ob_install_path: /home/admin/observer
# The log path for Oceanbase Database. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the {home_path}/log in oceanbase-ce.
observer_log_path: /home/admin/observer/log
# Monitor status for OceanBase Database. Active is to enable. Inactive is to disable. The default value is active. When you deploy an cluster automatically, obd decides whether to enable this parameter based on depends.
ob_monitor_status: active
10.10.10.1:
# Zone name for your observer. The default value is zone1. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the zone name in oceanbase-ce.
zone_name: zone1
10.10.10.2:
# Zone name for your observer. The default value is zone1. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the zone name in oceanbase-ce.
zone_name: zone2
10.10.10.3:
# Zone name for your observer. The default value is zone1. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the zone name in oceanbase-ce.
zone_name: zone3
prometheus:
depends:
- obagent
servers:
- 10.10.10.4
global:
home_path: /home/admin/prometheus
grafana:
depends:
- prometheus
servers:
- 10.10.10.4
global:
home_path: /home/admin/grafana
login_password: *********
For more information about the configuration items in the configuration file, see Configuration file description. After you modify the configuration file, run the following command to deploy and start the new cluster:
obd cluster deploy <new deploy name> -c new_config.yaml
obd cluster start <new deploy name>
After the cluster is started, access the Grafana page based on the displayed information. You can view the monitoring information of the existing cluster.
Scenario 2: The existing cluster has OBAgent deployed
If the existing cluster has OBAgent deployed, to add a GUI monitoring, you need to create a new cluster that contains the prometheus and grafana components.
In this scenario, you cannot declare that the prometheus component depends on the obagent component. Therefore, you need to manually associate the components. View the conf/prometheus_config/prometheus.yaml file in the installation directory of the obagent component in the existing cluster. Copy the corresponding configuration to the global -> conifg section of the prometheus component in the new cluster. Here is an example of the configuration:
# user:
# username: your username
# password: your password if need
# key_file: your ssh-key file path if need
# port: your ssh port, default 22
# timeout: ssh connection timeout (second), default 30
prometheus:
servers:
- 10.10.10.4
global:
# The working directory for prometheus. prometheus is started under this directory. This is a required field.
home_path: /home/admin/prometheus
config: # Configuration of the Prometheus service. The format is consistent with the Prometheus config file. Corresponds to the `config.file` parameter.
global:
scrape_interval: 1s
evaluation_interval: 10s
rule_files:
- "rules/*rules.yaml"
scrape_configs:
- job_name: prometheus
metrics_path: /metrics
scheme: http
static_configs:
- targets:
- 'localhost:9090'
- job_name: node
basic_auth:
username: ******
password: ******
metrics_path: /metrics/node/host
scheme: http
static_configs:
- targets:
- 10.10.10.1:8088
- job_name: ob_basic
basic_auth:
username: ******
password: ******
metrics_path: /metrics/ob/basic
scheme: http
static_configs:
- targets:
- 10.10.10.1:8088
- job_name: ob_extra
basic_auth:
username: ******
password: ******
metrics_path: /metrics/ob/extra
scheme: http
static_configs:
- targets:
- 10.10.10.1:8088
- job_name: agent
basic_auth:
username: ******
password: ******
metrics_path: /metrics/stat
scheme: http
static_configs:
- targets:
- 10.10.10.1:8088
grafana:
servers:
- 10.10.10.4
depends:
- prometheus
global:
home_path: /home/admin/grafana
login_password: ********* # Grafana login password. The default value is 'oceanbase'.
For more information about the configuration items in the configuration file, see Configuration file description. In the preceding example, the username and password in the basic_auth configuration item must correspond to those in the http_basic_auth_xxx configuration item in the OBAgent configuration file.
After you modify the configuration file, run the following command to deploy the new cluster:
obd cluster deploy <new deploy name> -c new_config.yaml
After the deployment is completed, copy the conf/prometheus_config/rules directory in the OBAgent installation directory to the installation directory of Prometheus.
Run the following command to start the new cluster:
obd cluster start <new deploy name>
After the cluster is started, access the Grafana page based on the displayed information. You can view the monitoring information of the existing cluster.
Notice
In the
scrape_configssection, the'localhost:9090'value must be modified to the listening address of the current Prometheus server. If the current Prometheus server is configured with authentication, thebasic_authconfiguration must also be configured. The listening address mentioned here is the address where Prometheus is deployed, which is specified by the address and port configuration items in the Prometheus configuration file.If the obagent nodes in the existing cluster have changed, run the
obd cluster edit-configcommand to synchronize the content of theconf/prometheus_config/prometheus.yamlfile in the OBAgent installation directory.
Scenario 3: Monitor multiple clusters and dynamically synchronize OBAgent changes
If you want Prometheus to collect monitoring information from multiple clusters or dynamically synchronize OBAgent changes, you can make slight modifications to the configuration in Scenario 2.
Replace the static_configs section in the Prometheus configuration with the file_sd_config section to obtain and synchronize OBAgent node information. In the following example, all YAML files in the targets directory under the Prometheus installation directory (home_path) are collected.
Note
The targets directory under the Prometheus installation directory is created only when the obagent component in the existing cluster configuration file is configured to create it. For more information, see Modify the monitored cluster configuration.
# user:
# username: your username
# password: your password if need
# key_file: your ssh-key file path if need
# port: your ssh port, default 22
# timeout: ssh connection timeout (second), default 30
prometheus:
servers:
- 10.10.10.4
global:
# The working directory for prometheus. prometheus is started under this directory. This is a required field.
home_path: /home/admin/prometheus
config: # Configuration of the Prometheus service. The format is consistent with the Prometheus config file. Corresponds to the `config.file` parameter.
global:
scrape_interval: 1s
evaluation_interval: 10s
rule_files:
- "rules/*rules.yaml"
scrape_configs:
- job_name: prometheus
metrics_path: /metrics
scheme: http
static_configs:
- targets:
- 'localhost:9090'
- job_name: node
basic_auth:
username: ******
password: ******
metrics_path: /metrics/node/host
scheme: http
file_sd_configs:
- files:
- targets/*.yaml
- job_name: ob_basic
basic_auth:
username: ******
password: ******
metrics_path: /metrics/ob/basic
scheme: http
file_sd_configs:
- files:
- targets/*.yaml
- job_name: ob_extra
basic_auth:
username: ******
password: ******
metrics_path: /metrics/ob/extra
scheme: http
file_sd_configs:
- files:
- targets/*.yaml
- job_name: agent
basic_auth:
username: ******
password: ******
metrics_path: /metrics/stat
scheme: http
file_sd_configs:
- files:
- targets/*.yaml
grafana:
servers:
- 10.10.10.4
depends:
- prometheus
global:
home_path: /home/admin/grafana
login_password: ********* # Grafana login password. The default value is 'oceanbase'.
For more information about the configuration items in the configuration file, see Configuration file description. In the preceding example, the username and password in the basic_auth configuration item must correspond to those in the http_basic_auth_xxx configuration item in the OBAgent configuration file.
After you modify the configuration file, run the following command to deploy the new cluster:
obd cluster deploy <new deploy name> -c new_config.yaml
After the deployment is completed, copy the conf/prometheus_config/rules directory in the OBAgent installation directory to the installation directory of Prometheus.
Run the following command to start the new cluster:
obd cluster start <new deploy name>
After the new cluster is deployed, access the Grafana page based on the displayed information. At this point, you cannot view the monitoring information of the monitored cluster. You need to modify the obagent configuration of the monitored cluster.
Modify the monitored cluster configuration
To create the targets directory under the Prometheus installation directory, run the obd cluster edit-config <deploy name> command to modify the configuration file and add the target_sync_configs configuration item to point to the targets directory under the Prometheus installation directory (by default, the user configuration of the current cluster is used. If it is different, you can set it as in the example).
obagent:
servers:
# Please don't use hostname, only IP can be supported
- 10.10.10.1
- 10.10.10.2
- 10.10.10.3
global:
....
target_sync_configs:
- host: 10.10.10.4
target_dir: /home/admin/prometheus/targets
# username: your username
# password: your password if need
# key_file: your ssh-key file path if need
# port: your ssh port, default 22
# timeout: ssh connection timeout (second), default 30
...
After you modify the configuration file, restart the cluster based on the prompt. After the restart, access the Grafana page to view the monitoring information of the existing cluster.
Notice
In the
scrape_configssection, the'localhost:9090'value must be modified to the listening address of the current Prometheus server. If the current Prometheus server is configured with authentication, thebasic_authconfiguration must also be configured. The listening address mentioned here is the address where Prometheus is deployed, which is specified by the address and port configuration items in the Prometheus configuration file.The HTTP username and password of the OBAgent component collected by Prometheus must be consistent. If they are inconsistent, split the collection items.