Redis 是使用C语言实现的一款非关系型数据库，既象具备前辈 memcached 从内存中读写数据以实现高性能，另外还可以通过快照到磁盘实现持久化，一般作为互联网IT技术栈中的缓存层实现。

客户端

redis-cli -h <hostname or ip> -p <port>

info [keyspace] #查看keys 信息
info [memory] #查看内存占用信息
info [stats]
info #查看所有信息

High memory usage triggered on grafana(>85%)

Investigation process login on omnisaas-tiffany-prod-db-1 and run top command to check current ram usage.(redis > 3G).

checks redis keys information on grafana dashboard(keys on master increase continuously) login on omnisaas-tiffany-prod-db-1 and run redis-cli

info keyspace info memory outputs below:

127.0.0.1:6379> info keyspace

db0:keys=11923407,expires=0,avg_ttl=0 db1:keys=1412,expires=0,avg_ttl=0 db2:keys=5,expires=2,avg_ttl=10148 db3:keys=2,expires=2,avg_ttl=17755 db4:keys=4,expires=1,avg_ttl=17399

127.0.0.1:6379> info memory

used_memory:3476245288 used_memory_human:3.24G used_memory_rss:3995889664 used_memory_rss_human:3.72G used_memory_peak:3476569560 used_memory_peak_human:3.24G used_memory_peak_perc:99.99% used_memory_overhead:614264244 used_memory_startup:786592 used_memory_dataset:2861981044 used_memory_dataset_perc:82.35% total_system_memory:16374530048 total_system_memory_human:15.25G used_memory_lua:72704 used_memory_lua_human:71.00K maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction mem_fragmentation_ratio:1.15 mem_allocator:jemalloc-4.0.3 active_defrag_running:0 lazyfree_pending_objects:0 why so many keys not free up and no expiration time? so we need search more detail.

to set rbd tools

pip install rdbtools

rdb -c memory /var/lib/redis/6379/dump.rdb > /path/output.txt
then check the detail in the output.txt 

... 11833502 0,hash,Token:token-383d9ae46ac24a398c78c40edffc4c8f,262,ziplist,5,28, 11833503 0,hash,Token:token-12a928eefb9446248262580d88999f2b,262,ziplist,5,28, 11833504 0,hash,Token:token-f347fea74a874635b364cabcfc9f9422,262,ziplist,5,28, 11833505 0,hash,Token:token-5d8a723d312c436fa410af663f0f669c,262,ziplist,5,28, 11833506 0,hash,Token:token-b1f40c0d6b8a42b9bc1b44e88ec73192,262,ziplist,5,28, 11833507 0,hash,Token:token-0d1764fca13a43aaa8a8f533ab1ae8c4,262,ziplist,5,28, 11833508 0,hash,Token:token-27100fc1f8794953aa1b15e904a633aa,262,ziplist,5,28, 11833509 0,hash,Token:token-3f1badf089f54d9aafd72f8fc4df498a,262,ziplist,5,28, 11833510 0,hash,Token:token-d75792b99734420a8fb56a2bd1d487fb,262,ziplist,5,28, 11833511 0,hash,Token:token-13f20d88ee1842729dc444e28dc9743d,262,ziplist,5,28, 11833512 0,hash,Token:token-2a46a90c7c2248f78435188fac6c3338,262,ziplist,5,28, 11833513 0,hash,Token:token-029bd09d8668439498acf8d76d4d0dd5,262,ziplist,5,28, 11833514 0,hash,Token:token-3d88a6e1033d445caebbc7f69956c891,262,ziplist,5,28, 11833515 0,hash,Token:token-a46e2539eb634029be0efbe152c333a2,262,ziplist,5,28, 11833516 0,hash,Token:token-3eef2732ea1549ea98d01181c7b42a0f,262,ziplist,5,28, 11833517 0,hash,Token:token-0f598dce72fa4c638bb5c50f6d385694,262,ziplist,5,28, 11833518 0,hash,Token:token-1146df446d3248f1b9d7f50943b26f0c,262,ziplist,5,28, 11833519 0,hash,Token:token-97f4f5b711db4a4980a985e8ddf7a636,262,ziplist,5,28, 11833520 0,hash,Token:token-70a8619f364846d9be591496651b6e2e,262,ziplist,5,28, 11833521 0,hash,Token:token-0c9bc229075c440cab885198982badb8,262,ziplist,5,28, 11833522 0,hash,Token:token-227eb06d4e954f77949ee2763a66b5da,262,ziplist,5,28, 11833523 0,hash,Token:token-33130e18c496411c85e6cabcdd7c5186,262,ziplist,5,28, 11833524 0,hash,Token:token-87f5f2c3678141f7829cd41eeb587f1b,262,ziplist,5,28, 11833525 0,hash,Token:token-5b047fb2b5544b8486a4c4310b68b85f,262,ziplist,5,28, 11833526 0,hash,Token:token-a39b775b456a46d2b6315da98b3ce539,262,ziplist,5,28, 11833527 0,hash,Token:token-ab2a4d7b810f4807b45fa9730c88b715,262,ziplist,5,28, 11833528 0,hash,Token:token-673517aa7beb47c4a9e44580ba3d2da4,262,ziplist,5,28, 11833529 0,hash,Token:token-ce07ca029cc54012b0b5407f2ee9587c,262,ziplist,5,28, 11833530 0,hash,Token:token-5c4c86bbcc8d4930826a512da962247a,262,ziplist,5,28, 11833531 0,hash,Token:token-748a471ce141453995e773b8785b6320,262,ziplist,5,28, 11833532 0,hash,Token:token-32d0abef763b4fe394f565c638928673,262,ziplist,5,28, 11833533 0,hash,Token:token-dc2794a237c74e68b61d04a0e68c77c4,262,ziplist,5,28, 11833534 1,string,a2117483-0092-5f72-b7dd-4adf22f48d8e::60002401,96,string,8,8, ... from the ouput we can find there are about 11833533 keys not free up. one key/value takes up 262 byte.

we can estimate total memory usage is about 3G byte.

Handle need to cowork with developers to determine clean strategy and how to handle the existing hash data on redis.

Problem record (update in 2022-04-20) redis cluster in prod need to set maxmemory and maxmemory-policy to reclaim memory.(it would be better to adjust hz value)

developer(wei) confirmed wechat-auth-token expiration not work in redis, it’s another reason memory not free up.(wei has fix the issue now and version fixed need to be deployed to prod)