Fork me on GitHub

Category: Python

Force-evict expired keys from redis

According to redis documentation redis keys with an expiration are evicted either in a passive way when they are accessed or in an active way where every 0.1 seconds redis looks at 100 random keys and evicts the ones that are expired. If at least 25/100 keys were evicted it tries again.

Unfortunately it seems that for a write-heavy application this probabilistic algorithm is still not sufficient to keep memory usage steady, it just continued to grow over time for us. So, we were forced to manage expirations ourselves. One way would be writing keys written into a separate data structure per hour, and then periodically going through those keys to force eviction. This seems like a big overhead to store all keys twice.

Another way is to iterate through the keys in redis. Redis recently added a SCAN command that allows you to do just that, and in the process evict keys that have expired. Below is a simple python script that illustrates this approach.

Read more

Posted Sat 28 June 2014 by Ivan Dyedov in Python (Linux, Redis, Python)


Autoreload Code in Python

While you're developing and debugging your WSGI application there's lots of ways to automatically reload your code on change out of the box. For example, if you're using werkzeug you can just pass the use_reloader flag:

run_sumple('', 5000, app, use_reloader=True)

For Flask, which actually uses werzeug internally, setting debug=True is all you need:

Django will automatically do it for you when you use: runserver

All of these examples work great while developing locally, however, they are greatly discouraged against being used in production. So the question arises, what do you do to automatically reload your code in production?

Read more

Posted Sun 07 April 2013 by Ivan Dyedov in Python (Python, uWSGI, gunicorn, WSGI)


gevent: gunicorn vs uWSGI

Following my previous benchmark I finally got around to benchmarking uWSGI with gevent and comparing its performance to gunicorn with gevent worker type. To do this you have to compile uWSGI and gevent from source, so I used the latest tagged releases at the time of the test, uWSGI 1.3 and gevent 1.0b4.

As it turns out, performance of the two servers is almost identical when using gevent...

Read more

Posted Sun 14 October 2012 by Ivan Dyedov in Python (Python, WSGI, gevent, gunicorn, uWSGI, benchmark, nginx, Ubuntu, Linux)


Benchmark uWSGI vs gunicorn for async workers

All of the WSGI benchmarks I found were pretty outdated or didn't include async results, so I decided to do some benchmarking myself.

I know there's other options for running python WSGI applications, but I settled on just 2: gunicorn, which has the advantage of being pure-python and uWSGI, which has the advantage of being pure-C.

Read more

Posted Wed 12 September 2012 by Ivan Dyedov in Python (Python, Ubuntu, WSGI, uWSGI, gunicorn, benchmark, nginx, gevent, eventlet)


Counting leaves with Python

If you have a multi-level dictionary in python looking like:

tree = {
    "first1": {
        "second1": {
            "third1": {
                "leaf1": 123,
                "leaf2": 234,
                "leaf3": 345
        "second2": {
            "leaf4": 456,
            "leaf5": 567
    "leaf6": 678

Here's a way to recursively count the number of elements, or "leaves" in this so-called dictionary tree (a single leaf being anything that's not a dictionary):

Read more

Posted Wed 03 August 2011 by Ivan Dyedov in Python (Python)