Category: Python

Force-evict expired keys from redis

According to redis documentation redis keys with an expiration are evicted either in a passive way when they are accessed or in an active way where every 0.1 seconds redis looks at 100 random keys and evicts the ones that are expired. If at least 25/100 keys were evicted it tries again.

Unfortunately it seems that for a write-heavy application this probabilistic algorithm is still not sufficient to keep memory usage steady, it just continued to grow over time for us. So, we were forced to manage expirations ourselves. One way would be writing keys written into a separate data structure per hour, and then periodically going through those keys to force eviction. This seems like a big overhead to store all keys twice.

Another way is to iterate through the keys in redis. Redis recently added a SCAN command that allows you to do just that, and in the process evict keys that have expired. Below is a simple python script that illustrates this approach.

Posted Sat 28 June 2014 by Ivan Dyedov in Python (Linux, Redis, Python)

Comments

Autoreload Code in Python

While you're developing and debugging your WSGI application there's lots of ways to automatically reload your code on change out of the box. For example, if you're using werkzeug you can just pass the use_reloader flag:

run_sumple('127.0.0.1', 5000, app, use_reloader=True)

For Flask, which actually uses werzeug internally, setting debug=True is all you need:

app.run(debug=True)

Django will automatically do it for you when you use:

manage.py runserver

All of these examples work great while developing locally, however, they are greatly discouraged against being used in production. So the question arises, what do you do to automatically reload your code in production?

Posted Sun 07 April 2013 by Ivan Dyedov in Python (Python, uWSGI, gunicorn, WSGI)

Comments

gevent: gunicorn vs uWSGI

Following my previous benchmark I finally got around to benchmarking uWSGI with gevent and comparing its performance to gunicorn with gevent worker type. To do this you have to compile uWSGI and gevent from source, so I used the latest tagged releases at the time of the test, uWSGI 1.3 and gevent 1.0b4.

As it turns out, performance of the two servers is almost identical when using gevent...

Posted Sun 14 October 2012 by Ivan Dyedov in Python (Python, WSGI, gevent, gunicorn, uWSGI, benchmark, nginx, Ubuntu, Linux)

Comments

Benchmark uWSGI vs gunicorn for async workers

All of the WSGI benchmarks I found were pretty outdated or didn't include async results, so I decided to do some benchmarking myself.

I know there's other options for running python WSGI applications, but I settled on just 2: gunicorn, which has the advantage of being pure-python and uWSGI, which has the advantage of being pure-C.

Posted Wed 12 September 2012 by Ivan Dyedov in Python (Python, Ubuntu, WSGI, uWSGI, gunicorn, benchmark, nginx, gevent, eventlet)

Comments

Counting leaves with Python

If you have a multi-level dictionary in python looking like:

tree = {
    "first1": {
        "second1": {
            "third1": {
                "leaf1": 123,
                "leaf2": 234,
                "leaf3": 345
            }
        },
        "second2": {
            "leaf4": 456,
            "leaf5": 567
        }
    },
    "leaf6": 678
}

Here's a way to recursively count the number of elements, or "leaves" in this so-called dictionary tree (a single leaf being anything that's not a dictionary):

Posted Wed 03 August 2011 by Ivan Dyedov in Python (Python)

Comments