Efficient Reloading in Django’s Runserver With Watchman

Setup

It’s just a few steps to get Django using watchman, covered in a couple paragraphs in the runserver documentation. Here I’ll cover the exact commands it needed me to run to set it up for DB Buddy on macOS, which is a pretty vanilla Django 3.1 project.

First, I installed watchman, following the installation instructions on its site. On macOS, this meant just running brew install watchman.

Second, I installed pywatchman, the library Django uses to interface with watchman. I use pip-compile from pip-tools to manage my requirements, so first I added pywatchman to my requirements.in:

diff --git a/requirements.in b/requirements.in
index c376421..4407458 100644
--- a/requirements.in
+++ b/requirements.in
@@ -28,2 +28,3 @@ pytest-xdist
 python-dotenv
+pywatchman
 requests

I then ran pip-compile to pin the current version of pywatchman in my requirements.txt, resulting in this change:

diff --git a/requirements.txt b/requirements.txt
index 95002f9..5d9073c 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -155,2 +155,4 @@ pytz==2020.5
     #   tzlocal
+pywatchman==1.4.1
+    # via -r requirements.in
 requests-mock==1.8.0

I then ran pip install -r requirements.txt to install the compiled requirements file.

Third, I set up a watchman configuration file in order to ignore my node_modules directory. The Django documentation hints that ignoring non-Python directories like this may be sensible to reduce load. My project has a relatively minimal JavaScript setup, but that still amounts to 14,303 files in the node_modules directory, all of which don’t need watching.

I followed the watchman configuration documentation and created a .watchmanconfig file next to my manage.py containing:

{
  "ignore_dirs": ["node_modules"]
}

Fourth, I restarted runserver. Django confirmed it’s using watchman by listing WatchmanReloader as the file watcher on the first line:

$ python manage.py runserver
Watching for file changes with WatchmanReloader
Performing system checks...

System check identified no issues (1 silenced).
January 20, 2021 - 07:49:15
Django version 3.1.5, using settings 'db_buddy.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Great!

Benchmark

I followed this up with a quick before/after comparison of CPU usage, whilst the server was completely idle and not serving any requests.

Using StatReloader, runserver took about ~1.6% of a CPU core.

With WatchmanReloader, this dropped to 0%, with the watchman process also showing 0%. That’s fantastic!

This project is also relatively small at the moment, with only 50 Python files and a few dependencies. Since the StatReloader’s work is proportional to the number of files, the difference will only be more stark on larger projects.

Fin

If you’re interested in learning more about the implementation of Django’s watchman integration, check out Tom Forbes’ EuroPython talk.

May your edit cycle be ever faster and more efficient,

—Adam