May 21, 2017 · dsp2017 tools

Monitoring Redis

Today I'll show how you can monitor Redis activity. It's very useful for low-level debugging.

Few words about Redis

In a single sentence, Redis is an in-memory key-value store. But it's also much more than that - it supports many complex data types and is blazingly fast (can perform milions operations per second).

It's mostly used via language wrappers, like redis-py for Python, but sometimes you want to take a look what exactly is happening underneath. There's a very useful redis command for that, and I'll show you how to use it.

Monitoring redis

Redis-cli is an entrypoint for most of redis tools. One of them is Monitor, that logs all executed commands. It adds some overhead (about 50% to each command), but you have full knowledge what's happening. How to use it? It's quite easy, run following command:

# use optional flags if you are not on a host with redis
redis-cli [-h hostname] [-p port] [-a password] monitor

# command for a redis Docker container
docker exec redis-container-name redis-cli monitor  

What will you see? It depends on your redis usage. In my case, I couldn't read anything - messages were appearing way too fast. So, let's dump it into a file:

redis-cli monitor > redis.log  
# wait a bit, then press Ctrl + C to stop

Now we're talking! Analyzing file is way easier. But still, in my case, result of a 10s dump has 12MB and more than 100k lines. Let's see how it looks like (redis is used in Django-channels application):

1495374947.551309 [0 lua] "expire" "asgi:9c17319ac1144e5d967e387ca4c9a159" "60"  
1495374947.551324 [0 lua] "rpush" "asgi:http.request" "asgi:9c17319ac1144e5d967e387ca4c9a159"  
1495374947.551376 [0 lua] "expire" "asgi:http.request" "61"  
1495374947.552120 [0 167.114.254.56:58364] "GET" "asgi:9c17319ac1144e5d967e387ca4c9a159"  
1495374947.552721 [0 167.114.254.63:39430] "EVALSHA" "3640886a0c8901ca9188f5f7a5f7a346145b9c5f" "14" "asgi:websocket.send!CaVgMqZFcpUk" "asgi:websocket.send!CNmEQsDcYUxm" "asgi:websocket.send!RvlvHsHxqYLa" "asgi:websocket.send!AEtVpVIjfvSf" "asgi:websocket.send!aWIQkBnAdTSJ" "asgi:websocket.send!eCmjCHjgYKtC" "asgi:websocket.send!tJwTIGtDuGbd" "asgi:websocket.send!iTCZRlrsXexr" "asgi:websocket.send!LEjMkRlBHmpf" "asgi:websocket.send!ozosqIsbEYLW" "asgi:websocket.send!frEaeJKHEHQG" "asgi:websocket.send!bhuzagrVicUj" "asgi:websocket.send!OGHHZBCFUIoJ" "asgi:websocket.send!HeDLryxfaqdI"  
1495374947.552872 [0 lua] "LPOP" "asgi:websocket.send!CaVgMqZFcpUk"  
1495374947.552885 [0 lua] "LPOP" "asgi:websocket.send!CNmEQsDcYUxm"  

Searching in a Redis log file

Now we have a big file with many lines. How can we find anything there? There are couple options, like always:

Personally I make some initial filtering of my file, and open result in editor. Here are the most common commands that I'm using:

# Show all ip addresses that are using redis (with count)
cat redis.log | grep -v 'lua' | cut -d' ' -f3 | cut -d':' -f1 | sort | uniq  
# sample output:
# 719  192.168.1.111
# 738  192.168.1.114
# 796  192.168.1.154
# 57   192.168.1.17

# Show all commands with specified type 
# and save it to file for later use
cat redis.log | grep 'SET' > redis-set.log  

That's all. I've used this technique to find some nasty bugs in my code and profile it. Share if you found this post useful!

Comments powered by Disqus