Today I'll show how you can monitor Redis activity. It's very useful for low-level debugging.
Few words about Redis
In a single sentence, Redis is an in-memory key-value store. But it's also much more than that - it supports many complex data types and is blazingly fast (can perform milions operations per second).
It's mostly used via language wrappers, like redis-py for Python, but sometimes you want to take a look what exactly is happening underneath. There's a very useful redis command for that, and I'll show you how to use it.
Monitoring redis
Redis-cli is an entrypoint for most of redis tools. One of them is Monitor, that logs all executed commands. It adds some overhead (about 50% to each command), but you have full knowledge what's happening. How to use it? It's quite easy, run following command:
# use optional flags if you are not on a host with redis
redis-cli [-h hostname] [-p port] [-a password] monitor
# command for a redis Docker container
docker exec redis-container-name redis-cli monitor
What will you see? It depends on your redis usage. In my case, I couldn't read anything - messages were appearing way too fast. So, let's dump it into a file:
redis-cli monitor > redis.log
# wait a bit, then press Ctrl + C to stop
Now we're talking! Analyzing file is way easier. But still, in my case, result of a 10s dump has 12MB and more than 100k lines. Let's see how it looks like (redis is used in Django-channels application):
1495374947.551309 [0 lua] "expire" "asgi:9c17319ac1144e5d967e387ca4c9a159" "60"
1495374947.551324 [0 lua] "rpush" "asgi:http.request" "asgi:9c17319ac1144e5d967e387ca4c9a159"
1495374947.551376 [0 lua] "expire" "asgi:http.request" "61"
1495374947.552120 [0 167.114.254.56:58364] "GET" "asgi:9c17319ac1144e5d967e387ca4c9a159"
1495374947.552721 [0 167.114.254.63:39430] "EVALSHA" "3640886a0c8901ca9188f5f7a5f7a346145b9c5f" "14" "asgi:websocket.send!CaVgMqZFcpUk" "asgi:websocket.send!CNmEQsDcYUxm" "asgi:websocket.send!RvlvHsHxqYLa" "asgi:websocket.send!AEtVpVIjfvSf" "asgi:websocket.send!aWIQkBnAdTSJ" "asgi:websocket.send!eCmjCHjgYKtC" "asgi:websocket.send!tJwTIGtDuGbd" "asgi:websocket.send!iTCZRlrsXexr" "asgi:websocket.send!LEjMkRlBHmpf" "asgi:websocket.send!ozosqIsbEYLW" "asgi:websocket.send!frEaeJKHEHQG" "asgi:websocket.send!bhuzagrVicUj" "asgi:websocket.send!OGHHZBCFUIoJ" "asgi:websocket.send!HeDLryxfaqdI"
1495374947.552872 [0 lua] "LPOP" "asgi:websocket.send!CaVgMqZFcpUk"
1495374947.552885 [0 lua] "LPOP" "asgi:websocket.send!CNmEQsDcYUxm"
Searching in a Redis log file
Now we have a big file with many lines. How can we find anything there? There are couple options, like always:
- Open file in text editor and just search for relevant lines
- Use commands to show and aggregate only interesting lines
Personally I make some initial filtering of my file, and open result in editor. Here are the most common commands that I'm using:
# Show all ip addresses that are using redis (with count)
cat redis.log | grep -v 'lua' | cut -d' ' -f3 | cut -d':' -f1 | sort | uniq
# sample output:
# 719 192.168.1.111
# 738 192.168.1.114
# 796 192.168.1.154
# 57 192.168.1.17
# Show all commands with specified type
# and save it to file for later use
cat redis.log | grep 'SET' > redis-set.log
That's all. I've used this technique to find some nasty bugs in my code and profile it. Share if you found this post useful!
