Yawd website

nov07

Tuning the yawd-elfinder Django file manager performance

The latest version of yawd-elfinder allows you to manage your remote files through the application and link them to your models using Django file storages. There exist several 3rd party storage applications (like django-storages, django-dropbox etc.) that enable connection to various remote systems. Amazon s3, ftp and sftp backends, even your dropbox public folder are just an example of what can be connected to yawd-elfinder.

However, depending on the driver, remote connections may affect the performance of yawd-elfinder. For example the dropbox storage uses a restful API to perform its operations, meaning that an http request must be sent for any required information. Although yawd-elfinder instantiates each storage only once in an effort to avoid negotiations for subsequent elfinder requests, this is not enough. To improve performance in this case you can use two options that tune the caching behavior of yawd-elfinder.

The keepAlive option

The keepAlive option determines whether the driver instance should be cached in the application memory or not. Caching the driver means that on every command that elfinder sends to the server (e.g. list directory, or create a new file etc.), the driver will not have to mount the volume again and will use the cached version instead. When mounting a volume a lot of background processing happens. While when working on a local filesystem there is no obvious performance boost, it is recommended that is option is on for remote drives.

Let’s suppose we want to attach a file to one of our django project models and be able to select files from both the local filesystem and our dropbox public folder. We first need to define a custom optionset for use with yawd-elfinder. We will define two root directories, the first using the ElfinderVolumeLocalFileSystem driver and the latter using the ElfinderVolumeStorage driver along with the django-dropbox storage.

in settings.py:
from elfinder.volumes.storage import ElfinderVolumeStorage
from elfinder.volumes.filesystem import ElfinderVolumeLocalFileSystem

ELFINDER_CONNECTOR_OPTION_SETS = {
    'multiple' : {
        'debug': False,
        'roots': [{
                'id' : 'localid',
                'driver' : ElfinderVolumeLocalFileSystem,
            },{
                'id' : 'dropboxid',
                'driver' : ElfinderVolumeStorage,
                'storageClass' : 'django_dropbox.storage.DropboxStorage',
            }]
    }
}

Now we should define our model. We will use the file_ member to store the file attachment, setting an ElfinderField using our ‘multiple’ optionset.

in models.py:
from django.db import models
from elfinder.fields import ElfinderField 

class TestModel(models.Model):
    name = models.CharField(max_length=100)
    file_ = ElfinderField(optionset='multiple', blank=True)
    def __unicode__(self):
        return self.name

If we syncdb and edit the model through the admin website we note that elfinder is taking a while to respond, even when operating on the local filesystem files. This happens because upon request both volumes must are mounted and the dropbox client is slowing down the process. To enable keepAlive and see the diference we change our optionset to the following:

in settings.py:
from elfinder.volumes.storage import ElfinderVolumeStorage
from elfinder.volumes.filesystem import ElfinderVolumeLocalFileSystem

ELFINDER_CONNECTOR_OPTION_SETS = {
    'multiple' : {
        'debug': False,
        'roots': [{
                'id' : 'localid',
                'driver' : ElfinderVolumeLocalFileSystem,
            },{
                'id' : 'dropboxid',
                'driver' : ElfinderVolumeStorage,
                'storageClass' : 'django_dropbox.storage.DropboxStorage',
                'keepAlive' : True
            }]
    }
}

Note that keepAlive is a per-root option. This means we are able to cache the dropbox driver instance and do not cache the local filesystem root, to save system memory. Visiting the admin site we will now see a significant performance boost, especially when operating on the local root.

The cache option

When a request is sent to the yawd-elfinder connector, the application performs multiple operations on disk, such as reading modification time information, calculating file and directory sizes, listing directories etc. These actions are very quick on local drives but on remote file systems they could benefit from some caching. The cache option is a root-wise option that determines the number of seconds for which yawd-elfinder will cache file and directory information. The first time yawd-elfinder generates a file stat it will store it to the cache and use the cached version on subsequent requests. The cache is cleared when files are modified through elfinder (e.g. upon file delete, contents update, directory creation etc.).

The default value for cache option is 10 minutes (600 seconds). In our example, we might like to cache our dropbox root for an hour (so that during a normal session information is not calculated multiple times) and disable caching for the local driver. To do so, we modify our optionset as follows:

in settings.py:
from elfinder.volumes.storage import ElfinderVolumeStorage
from elfinder.volumes.filesystem import ElfinderVolumeLocalFileSystem

ELFINDER_CONNECTOR_OPTION_SETS = {
    'multiple' : {
        'debug': False,
        'roots': [{
                'id' : 'localid',
                'driver' : ElfinderVolumeLocalFileSystem,
                'cache' : 0
            },{
                'id' : 'dropboxid',
                'driver' : ElfinderVolumeStorage,
                'storageClass' : 'django_dropbox.storage.DropboxStorage',
                'keepAlive' : True,
                'cache' : 3600
            }]
    }
}

This time, if we repeatedly visit dropbox folders through elfinder the response is much faster. However we must be careful with caching. If a third-party changes the filesystem it might take up to X seconds for elfinder to notice, X being the number of seconds defined through the cache option.

Comments powered by Disqus