Django: Defer a model field by default

Spinning queries into a fine thread.

Some models have one or a few large fields that dominate their per-instance size. For example, take a minimal blog post model:

from django.db import models


class Post(models.Model):
    blog = models.ForeignKey("Blog", on_delete=models.CASCADE)
    title = models.TextField()
    body = models.TextField()

body is typically many times larger than the rest of the Post. It can be a good optimization to defer() such fields when not required:

def index(request):
    posts = Post.objects.defer("body")
    ...

Deferred fields are not fetched in the main query, but will be lazily loaded upon access. Deferring large fields can noticeably reduce data transfer, and thus query time, memory usage, and total page load time. This comes with the risk that you accidentally lazy-load the deferred field, which done in a loop leads to N+1 queries.

When most usage of a model does not require the field, you might want to defer a field by default. Then you don’t need to sprinkle .defer(...) calls everywhere, and can instead use .defer(None) in the few sites where the field is used.

Defer by default with a custom base manager

To defer fields by default, follow these steps:

  1. Create a manager class that makes the appropriate defer() call in its get_queryset() method.
  2. Attach the manager to the model, ideally as objects.
  3. Make the manager the Model’s base manager by naming it in Meta.base_manager_name.

(This manager class should not apply any filtering, as noted in the base manager documentation).

For example, to adapt the above Post model to defer body:

from django.db import models


class PostManager(models.Manager):
    def get_queryset(self):
        return super().get_queryset().defer("body")


class Post(models.Model):
    blog = models.ForeignKey("Blog", on_delete=models.CASCADE)
    title = models.TextField()
    body = models.TextField()

    objects = PostManager()

    class Meta:
        base_manager_name = "objects"

Situations where the defer() applies

The field will then be deferred in most situations. We can check this by testing whether the field is in an instance’s __dict__ attribute.

Let’s look at some examples, which also use these referring models:

class Blog(models.Model):
    title = models.TextField()


class Banner(models.Model):
    post = models.ForeignKey(Post, on_delete=models.CASCADE)
    text = models.TextField()

The examples:

  1. Fetching a Post directly:

    In [1]: from example.models import *

In [2]: post = Post.objects.earliest("id")

In [3]: "body" in post.__dict__
Out[3]: False

  2. Fetching a post through a related manager:

    In [4]: blog = Blog.objects.earliest("id")

In [5]: post = blog.post_set.earliest("id")

In [6]: "body" in post.__dict__
Out[6]: False

  3. Accessing a post through a prefetched related manager:

    In [7]: blog = Blog.objects.prefetch_related("post_set").earliest("id")

In [8]: post = blog.post_set.all()[0]

In [9]: "body" in post.__dict__
Out[9]: False

  4. Lazy-loading a post through a foreign key:

    In [10]: banner = Banner.objects.earliest("id")

In [11]: post = banner.post

In [12]: "body" in post.__dict__
Out[12]: False

  5. Accessing a post through a prefetched foreign key:

    In [13]: banner = Banner.objects.prefetch_related("post").earliest("id")

In [14]: post = banner.post

In [15]: "body" in post.__dict__
Out[15]: False

Fin

Thanks to Pascal Fouque at my client Silvr for asking me to look into doing this.

May you defer fields but not your dishes,

—Adam

Learn how to make your tests run quickly in my book Speed Up Your Django Tests.

Subscribe via RSS, Twitter, Mastodon, or email:

One summary email a week, no spam, I pinky promise.

Related posts:

Tags: