Deploy OBLogProxy by using the CLI

This topic describes how to deploy OBLogProxy by using the CLI in full deployment and minimal deployment scenarios.

Introduction to the components

• ODP

  ODP is a proxy server for OceanBase Database. For more information, see ODP Documentation.

• Config Server

  Config Server allows you to register, store, and query metadata of OceanBase Database. For more information, see ob-configserver.

• OBLogProxy

  OBLogProxy is a proxy service for managing incremental logs of OceanBase Database. It connects to OceanBase Database and reads incremental logs. It provides the change data capture (CDC) capability for downstream services. For more information, see OBLogProxy GitHub.

Configuration files

OceanBase Deployer (OBD) provides sample configuration files required for OBLogProxy deployment. You can modify them based on your server resources.

• If you have installed OBD by downloading the RPM package for OBD, you can view sample configuration files in the /usr/obd/example/oblogproxy directory.

• If you have installed OBD by using the all-in-one installation package, you can view sample configuration files in the ~/.oceanbase-all-in-one/obd/usr/obd/example/oblogproxy directory.

• You can also view sample configuration files in the OBD repository at GitHub.

OBD provides the following sample configuration files for OBLogProxy deployment:

• The distributed-with-obproxy-and-oblogproxy-example.yaml configuration file is for full deployment.

• The oblogproxy-only-example.yaml configuration file is for minimal deployment.

Full deployment

Scenario

You have not deployed any OceanBase Database component in the current environment and will deploy OceanBase Database Community Edition, ODP Community Edition, Config Server, and OBLogProxy in a cluster by using OBD.

Prerequisites

You have installed OBD V2.5.0 or later. If you use OBD of an earlier version, upgrade it to OBD V2.5.0. For more information, see How do I upgrade OBD? in FAQ.

Procedure

The following example describes how to deploy OBLogProxy by using the distributed-with-obproxy-and-oblogproxy-example.yaml configuration file.

1. Configure the user information.

   The following sample code shows user information parameters in the configuration file:

   ## Only need to configure when remote login is required
   user:
     username: admin
   #   password: your password if need
     key_file: /home/admin/.ssh/id_rsa
   #   port: your ssh port, default 22
   #   timeout: ssh connection timeout (second), default 30
   

   username specifies the username of the account used to log on to the target server. Make sure that this account has the write privilege on home_path. password and key_file are used for user verification. Generally, you only need to specify one of them.

   Notice

   After you specify the path of the key, comment out or delete the password field if your key does not require a password. Otherwise, the value of the password parameter will be taken as the password of the key and used for logon, leading to a logon verification failure.

2. Configure OceanBase Database Community Edition.

   You must create a user named cdcro@sys and grant read-only permissions for OBLogProxy to the user. The cdcro@sys user belongs to OceanBase Database. OBLogProxy only uses this user.

   The following sample code shows parameters of OceanBase Database Community Edition in the configuration file:

   oceanbase-ce:
     servers:
       - name: server1
         # Do not use the hostname. Only IP address is supported.
         ip: 10.10.10.1
       - name: server2
         ip: 10.10.10.2
       - name: server3
         ip: 10.10.10.3
     depends:
       - ob-configserver
     global:
       memory_limit: 64G # The maximum running memory for an observer
       # The reserved system memory. system_memory is reserved for general tenants. The default value is 30G.
       system_memory: 30G
       datafile_size: 192G # Size of the data file. 
       log_disk_size: 192G # The size of disk space used by the clog files.
       enable_syslog_wf: false # Print system logs whose levels are higher than WARNING to a separate log file. The default value is true.
       enable_syslog_recycle: true # Enable auto system log recycling or not. The default value is false.
       max_syslog_file_count: 4 # The maximum number of reserved log files before enabling auto recycling. The default value is 0.
       # observer cluster name, consistent with obproxy's cluster_name
       appname: obcluster
       root_password: ******** # root user password, can be empty
       proxyro_password: ******** # proxyro user password, consistent with obproxy's observer_sys_password, can be empty
       cdcro_password: ******** # cdcro user password, consistent with oblogproxy's observer_sys_password, can be empty
     server1:
       mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. DO NOT change this value after the cluster is started.
       rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. DO NOT change this value after the cluster is started.
       # Required. The working directory for OceanBase Database. OceanBase Database is started under this directory.
       home_path: /home/admin/observer
       # The directory for data storage. The default value is $home_path/store.
       # data_dir: /data
       # The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
       # redo_dir: /redo
       zone: zone1
     server2:
       mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. DO NOT change this value after the cluster is started.
       rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. DO NOT change this value after the cluster is started.
       # Required. The working directory for OceanBase Database. OceanBase Database is started under this directory.
       home_path: /home/admin/observer
       # The directory for data storage. The default value is $home_path/store.
       # data_dir: /data
       # The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
       # redo_dir: /redo
       zone: zone2
     server3:
       mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. DO NOT change this value after the cluster is started.
       rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. DO NOT change this value after the cluster is started.
       # Required. The working directory for OceanBase Database. OceanBase Database is started under this directory.
       home_path: /home/admin/observer
       # The directory for data storage. The default value is $home_path/store.
       # data_dir: /data
       # The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
       # redo_dir: /redo
       zone: zone3
   

   For more information about the parameters, see the OceanBase Database part of the Parameters section in the Configuration file topic. Compared with deploying only an OceanBase cluster, the configuration file provides the cdcro_password parameter. You can use this parameter to specify the password of the cdcro@sys user. If you do not specify the user password, OBD automatically generates a random password string.

3. Configure ODP Community Edition and Config Server.

   The following sample code shows parameters of ODP Community Edition and Config Server in the configuration file:

   obproxy-ce:
     depends:
       - oceanbase-ce
     servers:
       - 10.10.10.1
     global:
       listen_port: 2883 # External port. The default value is 2883.
       prometheus_listen_port: 2884 # The Prometheus port. The default value is 2884.
       home_path: /home/admin/obproxy
       enable_cluster_checkout: false
       # observer cluster name, consistent with oceanbase-ce's appname. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.
       # cluster_name: obcluster
       skip_proxy_sys_private_check: true
       enable_strict_kernel_release: false
       obproxy_sys_password: ******** # obproxy sys user password, can be empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.
       # observer_sys_password: # proxyro user password, consistent with oceanbase-ce's proxyro_password, can be empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.
   ob-configserver:
     servers:
       - 10.10.10.1
     global:
       listen_port: 8080 # The port of ob-configserver web
       # server_ip: 0.0.0.0 # Listen to the ob-configserver server IP. When you want to listen to the specified IP address, use it.
       home_path: /home/admin/ob-configserver  # The working directory for prometheus. ob-configserver is started under this directory. This is a required field.
       ## log config
       # log_level: info # Log printing level of ob-configserver. The default value is `info`
       # log_maxsize: 30 # The total size of manager ob-configserver.Log size is measured in Megabytes.The default value is 30
       # log_maxage: 7 # The days of manager expired ob-configserver.Log retention days. The default value is 7
       # log_maxbackups: 10  #The number of manager expired ob-configserver.Log. The default value is 10
       # log_localtime: true #  Switch of ob-configserver.Log naming with localtime. The default value is true
       # log_compress: true # Compress ob-configserver.Log switch. The default value is true
   
       ## vip config, configserver will generate url with vip_address and port and return it to the client
       ## do not use some random value that can't be connected
       # vip_address: "10.10.10.1"
       # vip_port: 8080
       ## storage config
       # storage:
       #   database_type: sqlite3 # sqlite3 or mysql. Default sqlite3
       #   connection_url: "" # When database_type is set to sqlite3, the connection_url parameter can be left empty. If it is empty, the default value $home_path/.data.db?cache=shared&_fk=1 will be used. When database_type is set to mysql, the connection_url parameter must be configured, with a sample value of user:password@tcp(10.10.10.1:2883)/test?parseTime=true.
   

   For more information about the parameters, see the ODP and Config Server parts of the Parameters section in the Configuration file topic.

4. Configure OBLogProxy.

   The following sample code shows parameters of OBLogProxy in the configuration file:

   oblogproxy:
     version: 2.0.0
     depends:
       - oceanbase-ce
       - obproxy-ce
     servers:
       - 10.10.10.1
     global:
       home_path: /home/admin/oblogproxy
       service_port: 2983
       # binlog_dir: /home/admin/oblogproxy/run   # The directory for binlog file. The default value is $home_path/run.
       # binlog_mode: true   # enable binlog mode, default true
   

   OBLogProxy depends on OceanBase Database Community Edition and ODP Community Edition. The following table describes parameters in the configuration file:

   Parameter	Required?	Default value	Description
   version	No	The latest version	The version of the component to deploy. Generally, you do not need to specify this parameter. The latest version is deployed by default.
   servers	Yes	N/A	You must specify the information of each server in the following format: -name: server name(line break)ip: IP address of the server or -<IP address of the server>.
   home_path	Yes	N/A	The installation path of OBLogProxy.
   service_port	Yes	2983	The listening port of OBLogProxy.
   binlog_dir	No	$home_path/run	The absolute path of the binlog service.
   binlog_mode	No	true	Specifies whether to enable the binlog service.

   Take note of the following considerations:

   • When you deploy OBLogProxy by using OBD, the binlog_dir parameter in the configuration profile of OBD is equivalent to the binlog_log_bin_basename parameter in the configuration profile of OBLogProxy.

   • When you deploy OBLogProxy by using OBD, the default value of the binlog_mode parameter is true.

5. Deploy the cluster.

   [admin@test001 ~]$ obd cluster deploy obtest -c distributed-with-obproxy-and-oblogproxy-example.yaml
   

   After you run the obd cluster deploy command, if your server is connected to the Internet, OBD checks whether the desired installation package exists on the target server. If no, OBD automatically obtains the installation package from the YUM source.

6. Start your cluster.

   [admin@test001 ~]$ obd cluster start obtest
   

7. View the cluster status.

   [admin@test001 ~]$ obd cluster display obtest
   

Minimal deployment

Scenario

If an OceanBase cluster already exists in the environment and you want to use OBLogProxy, you only need to deploy OBLogProxy.

Prerequisites

You have installed OBD V2.5.0 or later. If you use OBD of an earlier version, upgrade it to OBD V2.5.0. For more information, see How do I upgrade OBD? in FAQ.

Procedure

The following example describes how to deploy OBLogProxy by using the oblogproxy-only-example.yaml configuration file.

1. Configure the user information.

   The following sample code shows user information parameters in the configuration file:

   ## Only need to configure when remote login is required
   user:
     username: admin
   #   password: your password if need
     key_file: /home/admin/.ssh/id_rsa
   #   port: your ssh port, default 22
   #   timeout: ssh connection timeout (second), default 30
   

   username specifies the username of the account used to log on to the target server. Make sure that this account has the write privilege on home_path. password and key_file are used for user verification. Generally, you only need to specify one of them.

   Notice

   After you specify the path of the key, comment out or delete the password field if your key does not require a password. Otherwise, the value of the password parameter will be taken as the password of the key and used for logon, leading to a logon verification failure.

2. Configure OBLogProxy.

   The following sample code shows parameters of OBLogProxy in the configuration file:

   oblogproxy:
     version: 2.0.0
     servers:
       - 10.10.10.1
     global:
       home_path: /home/admin/oblogproxy
       service_port: 2983
       ob_sys_username: cdcro  # A user under the sys tenant of oceanbase-ce, oblogproxy communicates with oceanbase-ce using this user, default ''
       ob_sys_password: ******** # ob_sys_username`s password, default ''
       # binlog_dir: /home/admin/oblogproxy/run   # The directory for binlog file. The default value is $home_path/run.
       # binlog_mode: true   # enable binlog mode, default true
   

   The following table describes parameters in the configuration file:

   Parameter	Required?	Default value	Description
   version	No	The latest version	The version of the component to deploy. Generally, you do not need to specify this parameter. The latest version is deployed by default.
   servers	Yes	N/A	You must specify the information of each server in the following format: -name: server name(line break)ip: IP address of the server or -<IP address of the server>.
   home_path	Yes	N/A	The installation path of OBLogProxy.
   service_port	Yes	2983	The listening port of OBLogProxy.
   ob_sys_username	No	This parameter is left empty by default.	The username used for communication between OBLogProxy and OceanBase Database. You must create the user in the sys tenant before the deployment. You do not need to include the cluster name or tenant name in the username.
   ob_sys_password	No	This parameter is left empty by default.	The password of the user specified by the ob_sys_username parameter. The password must be consistent with that of the corresponding user of the OceanBase Database.
   binlog_dir	No	$home_path/run	The absolute path of the binlog service.
   binlog_mode	No	true	Specifies whether to enable the binlog service.

   Take note of the following considerations:

   • When you deploy OBLogProxy by using OBD, the binlog_dir parameter in the configuration profile of OBD is equivalent to the binlog_log_bin_basename parameter in the configuration profile of OBLogProxy.

   • When you deploy OBLogProxy by using OBD, the default value of the binlog_mode parameter is true.

   • When you deploy OBLogProxy by using OBD, you must specify the ob_sys_username and ob_sys_password parameters in plaintext.

3. Deploy OBLogProxy.

   [admin@test001 ~]$ obd cluster deploy obtest -c oblogproxy-only-example.yaml
   

   After you run the obd cluster deploy command, if your server is connected to the Internet, OBD checks whether the desired installation package exists on the target server. If no, OBD automatically obtains the installation package from the YUM source.

4. Start OBLogProxy.

   [admin@test001 ~]$ obd cluster start obtest
   

5. Check the status of OBLogProxy.

   [admin@test001 ~]$ obd cluster display obtest
   

Verify the deployment

Depending on whether the binlog service is enabled, you can use the following two methods to verify the deployment:

• Method 1: If you set the binlog_mode parameter to true, you can copy and execute the connection string exported by OBD after OBLogProxy is started to verify whether OBD can connect to OBLogProxy.

  obclient -h10.10.10.1 -P2983
  

• Method 2: If you set the binlog_mode parameter to false, you can log on to the server where OBLogProxy is located and run the following command to check whether the oblogproxy process exists:

  ps axu | grep logproxy
  

  Output:

  admin    10493  1.1  0.0 634812  2940 ?        Sl   16:21   0:07 /home/admin/oblogproxy/bin/logproxy -f /home/admin/oblogproxy/conf/conf.json
  admin    18071  0.0  0.0 112812   980 pts/1    S+   16:32   0:00 grep --color=auto logproxy
  

References

• You can use OBD to manage OBLogProxy. For example, you can start, stop, restart, and destroy OBLogProxy. For more information, see Cluster commands. Note that if you stop OBLogProxy by running the obd cluster stop command, OBD stops only the oblogproxy process. The binlog converter process of the binlog service is not stopped.