OceanBase Database uses Memleak to diagnose memory leaks. Memleak records the memory allocation stacks of the tracing module. If the number of times that a stack is recorded increases continuously within a period, the memory allocation context corresponding to this stack may have encountered a memory leak. However, Memleak is generally enabled after a memory leak. If no memory leak occurs after Memleak is enabled, it cannot track the module with the memory leak and has to wait for the occurrence of another memory leak. Therefore, we want a Memleak-based regular memory leak diagnostics mechanism that can sample memory allocation information in real time and analyze modules with possible memory leak issues based on the sampling result.
The _min_malloc_sample_interval and _max_malloc_sample_interval parameters are used to control the sampling rate for memory allocation information.
Parameter |
Description |
Value range |
Default value |
|---|---|---|---|
| _min_malloc_sample_interval | The minimum number of ob_malloc times between two adjacent sampling operations. |
[1,10000] | 16 |
| _max_malloc_sample_interval | The maximum number of ob_malloc times between two adjacent sampling operations. |
[1,10000] | 256 |
Three sampling modes are available.
Zero sampling mode: The system does not record memory allocation information.
obclient> ALTER SYSTEM SET _min_malloc_sample_interval=10000; Query OK, 0 rows affected obclient> ALTER SYSTEM SET _max_malloc_sample_interval=10000; Query OK, 0 rows affectedFull sampling mode: The system records all memory allocation information. In this mode, the performance of OceanBase Database significantly deteriorates.
obclient> ALTER SYSTEM SET _min_malloc_sample_interval=1; Query OK, 0 rows affected obclient> ALTER SYSTEM SET _max_malloc_sample_interval=1; Query OK, 0 rows affectedRegular mode: The system records memory allocation information based on the specified sampling interval.
obclient> ALTER SYSTEM SET _min_malloc_sample_interval=16; Query OK, 0 rows affected obclient> ALTER SYSTEM SET _max_malloc_sample_interval=256; Query OK, 0 rows affected
Notice
- In OceanBase Database, the name of a hidden parameter begins with an underscore (_). Hidden parameters are used by developers only for troubleshooting or emergency O&M.
- Make sure that the value of
_min_malloc_sample_intervalis smaller than or equal to the value of_max_malloc_sample_interval. - We recommend that you do not modify the
_min_malloc_sample_intervaland_max_malloc_sample_intervalparameters to avoid causing the cluster performance to deteriorate.
Use Memleak for memory leak diagnostics in regular mode
Log in to the
systenant of the cluster as therootuser.Query the
__all_virtual_malloc_sample_infovirtual table for memory allocation information about the module that is suspected of memory leaks.The following example shows how to query the memory allocation information about the
glibc_mallocmodule.obclient [oceanbase]> SELECT * FROM __all_virtual_malloc_sample_info WHERE mod_name='glibc_malloc' ORDER BY alloc_bytes DESC limit 10; +-------------+----------+-----------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+-------------+-------------+ | svr_ip | svr_port | tenant_id | ctx_id | mod_name | back_trace | ctx_name | alloc_count | alloc_bytes | +-------------+----------+-----------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+-------------+-------------+ | xx.xx.xx.xx | 25224 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x188a867a 0x188aa126 0x15c53d3f 0x16b06ae2 0x16b369ef 0x16b32d7e 0x15c8bcf8 0x15c8606a | GLIBC | 3 | 9437232 | | xx.xx.xx.xx | 25225 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x188a867a 0x188aa126 0x15c53d3f 0x16b06ae2 0x16b369ef 0x16b32d7e 0x15c8bcf8 0x15c8606a | GLIBC | 3 | 9437232 | | xx.xx.xx.xx | 25226 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x188a867a 0x188aa126 0x15c53d3f 0x16b06ae2 0x16b369ef 0x16b32d7e 0x15c8bcf8 0x15c8606a | GLIBC | 3 | 9437232 | | xx.xx.xx.xx | 25224 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x19dee05c 0x19e00077 0xb49139a 0x5f6d17f 0x7fc3a6c85445 0x5f6abe1 0x0 0x0 0x0 0x0 0x0 | GLIBC | 1 | 4194320 | | xx.xx.xx.xx | 25225 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x19dee05c 0x19e00077 0xb49139a 0x5f6d17f 0x7ff8c02b3445 0x5f6abe1 0x0 0x0 0x0 0x0 0x0 | GLIBC | 1 | 4194320 | | xx.xx.xx.xx | 25226 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x19dee05c 0x19e00077 0xb49139a 0x5f6d17f 0x7f06f1b63445 0x5f6abe1 0x0 0x0 0x0 0x0 0x0 | GLIBC | 1 | 4194320 | | xx.xx.xx.xx | 25224 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x15c76adf 0xb49e963 0xb4915b6 0x5f6d17f 0x7fc3a6c85445 0x5f6abe1 0x0 0x0 | GLIBC | 1 | 3145744 | | xx.xx.xx.xx | 25225 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x15c76adf 0xb49e963 0xb4915b6 0x5f6d17f 0x7ff8c02b3445 0x5f6abe1 0x0 0x0 | GLIBC | 1 | 3145744 | | xx.xx.xx.xx | 25226 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x15c76adf 0xb49e963 0xb4915b6 0x5f6d17f 0x7f06f1b63445 0x5f6abe1 0x0 0x0 | GLIBC | 1 | 3145744 | | xx.xx.xx.xx | 25224 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e28d9 0x161a7a6b 0x161a7148 0x161aa227 0x64f4472 0x64f003c 0x64b9e68 0x64b1d45 0x6520720 0x651e8db | GLIBC | 2 | 131120 | +-------------+----------+-----------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+-------------+-------------+ 10 rows in set (0.012 sec)The query result shows that the
0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x188a867a 0x188aa126 0x15c53d3f 0x16b06ae2 0x16b369ef 0x16b32d7e 0x15c8bcf8 0x15c8606astack has the largestalloc_bytesvalue and is suspected of memory leaks.Columns in the query result are described as follows:
ColumnTypeDescriptionsvr_ip varchar:MAX_IP_ADDR_LENGTH The IP address. svr_port bigint(20) The port number of the server. tenant_id bigint(20) The ID of the tenant. ctx_id bigint(20) The CTX ID. mod_name varchar:OB_MAX_CHAR_LENGTH The name of the module. back_trace varchar:DEFAULT_BUF_LENGTH The stack for memory allocation. ctx_name varchar:OB_MAX_CHAR_LENGTH The CTX name. alloc_count bigint(20) The number of memory allocation times. alloc_bytes bigint(20) The total size of allocated memory. Based on the virtual table query result, analyze the
back_tracestack that has a largealloc_bytesvalue.$addr2line -pCfe bin/observer 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x188a867a 0x188aa126 0x15c53d3f 0x16b06ae2 0x16b369ef 0x16b32d7e 0x15c8bcf8 0x15c8606a void* oceanbase::lib::ObTenantCtxAllocator::common_alloc<oceanbase::lib::ObjectMgr>(long, oceanbase::lib::ObMemAttr const&, oceanbase::lib::ObTenantCtxAllocator&, oceanbase::lib::ObjectMgr&) at /home/distcc/tmp/./deps/oblib/src/lib/alloc/ob_tenant_ctx_allocator.cpp:414 (discriminator 2) oceanbase::lib::ObTenantCtxAllocator::alloc(long, oceanbase::lib::ObMemAttr const&) at /home/distcc/tmp/./deps/oblib/src/lib/alloc/ob_tenant_ctx_allocator.cpp:35 oceanbase::lib::ObMallocAllocator::alloc(long, oceanbase::lib::ObMemAttr const&) at ??:? oceanbase::common::ob_malloc(long, oceanbase::lib::ObMemAttr const&) at ./build_sanity/deps/oblib/src/lib/../../../../../deps/oblib/src/lib/allocator/ob_malloc.h:39 ob_malloc_hook(unsigned long, void const*) at ??:? operator new(unsigned long) at ??:? operator new[](unsigned long, std::nothrow_t const&) at ??:? oceanbase::share::schema::ObSchemaMgrCache::init(long, oceanbase::share::schema::ObSchemaMgrCache::Mode) at ./build_sanity/src/share/./src/share/schema/ob_schema_mgr_cache.cpp:153 oceanbase::share::schema::ObSchemaStore::init(unsigned long, long, long) at ./build_sanity/src/share/./src/share/schema/ob_schema_store.cpp:29 oceanbase::share::schema::ObSchemaStoreMap::create(unsigned long, long, long) at ./build_sanity/src/share/./src/share/schema/ob_schema_store.cpp:131 (discriminator 2) oceanbase::share::schema::ObMultiVersionSchemaService::init_multi_version_schema_struct(unsigned long) at ./build_sanity/src/share/./src/share/schema/ob_multi_version_schema_service.cpp:218 (discriminator 52) oceanbase::share::schema::ObServerSchemaService::update_schema_mgr(oceanbase::common::ObISQLClient&, oceanbase::share::schema::ObRefreshSchemaStatus const&, oceanbase::share::schema::ObSchemaMgr&, long, oceanbase::share::schema::ObServerSchemaService::AllSchemaKeys&) at ./build_sanity/src/share/./src/share/schema/ob_server_schema_service.cpp:4232 (discriminator 54) oceanbase::share::schema::ObServerSchemaService::refresh_increment_schema(oceanbase::share::schema::ObRefreshSchemaStatus const&) at ./build_sanity/src/share/./src/share/schema/ob_server_schema_service.cpp:5663 (discriminator 4) oceanbase::share::schema::ObServerSchemaService::refresh_schema(oceanbase::share::schema::ObRefreshSchemaStatus const&) at ./build_sanity/src/share/./src/share/schema/ob_server_schema_service.cpp:5288 oceanbase::share::schema::ObMultiVersionSchemaService::refresh_tenant_schema(unsigned long) at ??:? operator() at ./build_sanity/src/share/./src/share/schema/ob_multi_version_schema_service.cpp:2328 (discriminator 4)Use the
addr2linecommand to parse the stack that has the largestalloc_bytesvalue. Contact OceanBase Technical Support to analyze whether the stack has encountered memory leaks.