Yawd website

nov09

How to cache your website menu with django

On large django projects it is more likely that to generate a single webpage multiple queries over large datasets and complex data processing must be involved. In such cases -were performance is an issue-, the use of caching techniques can really save the day. Django provides a solid framework for caching, allowing for the storage/retrieval of single values (e.g. the result of complex database queries), html code fragments (e.g. the 'footer' of your webpage) or even entire django views.

In today's article we'll examine a way to cache our menu (consider a tree-menu application like the one described in this yawd blog article or the django-treemenus application) in order to reduce loading times and save db server load. Although this appears to be a fairly straight-forward task, a couple of decisions can be tricky.

Note: the code used on this article is based on the tree menu system discussed in this yawd article.

Caching the entire menu

We first need to open the template responsible for generating our menu (e.g. menu.html) and use django's 'cache' template tag to wrap the code rendering the menu.

Menu generation template (e.g. menu.html)
{% load cache %}

{% cache 35000 menu menu.id %}

<div id="menu_wrapper_{{menu.name}}">
{% for item in menu.menuitems %}
...menu rendering code here...
{% endfor %}
</div>

{% endcache %}

The above is really simple, it just tells django to cache the menu for 35000 seconds, under the name 'menu'. The last argument of the 'cache' tag (in this example menu.id) means that we want separate caching for each menu. Now the menu is cached at the template level. Any queries that the view (or menu tag) performs to retrieve the menu items prior to rendering the template will still be executed. To avoid this we must edit the menu tag's code. In our example the menu is rendered by the menu_as_ul tag, which we will modify as follows (this can be applied to any menu system, usually within a view or tag):

menu tag code
@register.inclusion_tag('menu.html', takes_context=True)
def menu_as_ul(context, id):
    
    #construct the cache key
    args = md5_constructor(id)
    key = 'template.cache.menu.%s' % (args.hexdigest())
    
    #if the key exists do not query
    if cache.get(key):
        return { 'request' : context['request'], 'menu' : {'id' : id} }

    menu = get_object_or_404(Menu, id=id, published=True)

    ...construct the menu here...
    
    return {
            'id' : id,
            'request' : context['request'] 
             ....your context...
            }

On the above code snippet, we calculate the cache key for the requested menu and then search the cache memory for the constructed key. If the key was not found, the menu is not cached and we need to query the database for the template to render the menu. If the key was found in the cache we can safely skip the querying process.

Since we cache our menu for an entire day, we must also be able to clear the records each time a menuitem is being added/edited/deleted. To achieve this we add a listener to the MenuItem's post_save signal. The callback function just removes all menu cache keys.

models.py
def clear_menu_cache(**kwargs):
    objects = Menu.objects.filter(published=True)
    
    for object in objects:
        args = md5_constructor(object.name)
        key = 'template.cache.menu.%s' % (args.hexdigest())
        cache.delete(key)

post_save.connect(clear_menu_cache, sender=MenuItem, dispatch_uid="menuitem_post_save")
post_delete.connect(clear_menu_cache, sender=MenuItem, dispatch_uid="menuitem_post_delete")

Caching separate menu instances for each url

If your menu html differs for each one of your pages (e.g. additional style classes applied to current elements, or extra sub-menu items displayed based on the current menu item), then caching the entire menu just once won't be sufficient. You need to store a copy of your menu html for each page. To achieve this, we must modify our cache key to contain each page's url. As a result, 4 different records will be cached, say, for the home-page's main menu, the home-page's footer menu, the contact page's main menu and the contact page's footer menu - instead of 2 stored with our previous approach. We modify our template code as follows:

Menu generation template (e.g. menu.html)
{% load cache %}

{% cache 35000 menu request.path menu.id %}
...menu rendering code here...
{% endcache %}

As we proceed with the implementation we face a major problem. When using Django's memcached backend for caching, there is no way of obtaining and iterating over the cached keys (this is not a Django limitation, it's just the way memcache works). Therefore, to clear our menu cache records we must know the key names a priori. In our case (where keys contain the page's url) this translates to knowing all the website urls! In large dynamic systems this is practically impossible. To overcome this we can store all keys to the database (as soon as they are created) to be able to retrieve them by means of querying the db. This will add a single query to the complexity of the page, but the benefit of caching the menu pays back for sure! At first, we declare the model that will hold the key data.

models.py
class MenuCacheKeys(models.Model):
    key = models.CharField(max_length=255)
    active = models.BooleanField(default=True)
    menu = models.ForeignKey(Menu)

    def __unicode__(self):
        self.key

Now let's modify our menu tag code to take into account the page's url and store all keys to the database:

menu tag code
@register.inclusion_tag('menu.html', takes_context=True)
def menu_as_ul(context, id):
    
    #now we take the url oath into consideration when constructing the key
    args = md5_constructor(u':'.join([context['request'].path, name]))
    key = 'template.cache.menu.%s' % (args.hexdigest())
    menu = get_object_or_404(Menu, name=name, published=True)

    #store keys to the database
    try:
        item = MenuCacheKeys.objects.get(key=key,menu=menu)
        if not item.active:
            item.active = True
            item.save()
    except:
        MenuCacheKeys.objects.create(key=key,menu=menu)

    if cache.get(key):
        ....same as above...

Finally, we will modify our signal listener to delete the stored keys once a MenuItem is added/edited/deleted.

models.py
def clear_menu_cache(**kwargs):
    if kwargs['sender'] == MenuItem:
        objects = MenuCacheKeys.objects.filter(active=True,
                                        menu=kwargs['instance'].get_menu())
    else:
        objects = MenuCacheKeys.objects.filter(active=True)

    for object in objects:
        cache.delete(object.key)
        object.active = False
        object.save()

post_save.connect(clear_menu_cache, sender=MenuItem, dispatch_uid="menuitem_post_save")
post_delete.connect(clear_menu_cache, sender=MenuItem, dispatch_uid="menuitem_post_delete")

The tricky part lies in MenuCacheKeys. You need to make sure that querying the database for a cache key is not as quick as querying the database for MenuItems. If that's the case, then you probably don't need to cache your menu at all (but that cannot be true since you're reading this article :P ). Please feel free to comment!

Comments powered by Disqus