Force-evict expired keys from redis

According to redis documentation redis keys with an expiration are evicted either in a passive way when they are accessed or in an active way where every 0.1 seconds redis looks at 100 random keys and evicts the ones that are expired. If at least 25/100 keys were evicted it tries again.

Unfortunately it seems that for a write-heavy application this probabilistic algorithm is still not sufficient to keep memory usage steady, it just continued to grow over time for us. So, we were forced to manage expirations ourselves. One way would be writing keys written into a separate data structure per hour, and then periodically going through those keys to force eviction. This seems like a big overhead to store all keys twice.

Another way is to iterate through the keys in redis. Redis recently added a SCAN command that allows you to do just that, and in the process evict keys that have expired. Below is a simple python script that illustrates this approach.

import redis

r = redis.Redis()
t = r.scan()
while t[0]:
    t = r.scan(t[0])

Note: you need to install redis python package to run this.

This command is only available in redis versions 2.8 and above, before that you were stuck with KEYS command that blocks all connections while it iterates and returns all the keys. That makes it a very bad contendant for production environments.

Posted Sat 28 June 2014 by Ivan Dyedov in Python (Linux, Redis, Python)