I recently created a multi-threaded background job processor, Disqualified. I knew of several alternatives, but I wanted to write my own because:
When writing a library, I tend to focus a lot on version compatibility since I know that companies can easily fall behind on keeping their library dependencies updated. I decided not to focus on that this time, but to only support my own, immediate use cases; Disqualified only supports Ruby 3.1 and Rails 7.0.
(But Super, my admin framework, supports Ruby 2.3+ and Rails 5.0+!)
One of my goals was to make this a multi-threaded background job processor, similar to Sidekiq. I decided to leverage Concurrent Ruby since it was already required by Rails.
Overall, I found it pleasant to work with Concurrent Ruby. I liked that it encouraged separating “threading code” and “logic code”.
I’ll probably blog about this some more soon. Disqualified is about to get much faster.
Like I mentioned, Disqualified was built for SQLite. It’s a great database, but it doesn’t have some features that I’m used to.
I realized that SQLite doesn’t support pessimistic locking (SELECT FOR UPDATE SKIP LOCKED), which is a feature I often use to make sure that something happens only once. With multi-threaded code, it’s pretty easy to accidentally make something never happen or happen too many times.
I needed to make sure that each job ran exactly once. I researched possibilities and decided to use a single, atomic update to “claim” a job, then to find and execute that claimed job.
The end result is basically the same. This was a pleasant reminder that you could use simple locks to build more complicated locks. It’s kinda cool!
Rails conveniently includes a way to safely run your Rails application code. (Oops, I noticed and fixed a mistake while writing this blog post—I had been using the executor although background job queues need to use the reloader.)
Although I had skipped this for my first release, I realized supporting it was the easiest way to get APM metrics.
I looked through the Rails source and found that it was easier than I thought. Most of the magic happens through ActiveJob::Base.serialize and ActiveJob::Base.execute. Reading an existing adapter is probably the easiest way to understand what to do; here’s the Sidekiq adapter.
It had lots of names—“obsolete”, “overdue”, “disqualify”, and finally “disqualified”. The name is a bit sarcastic since SQLite isn’t a very popular database used in websites. And since I only built what I needed for my website, I figured most people would find some “disqualifying” aspect of it.
But also—diSQuaLified—it has the letters s, q, and l, in that order!
Anyway, check out Disqualified! It now works with ActiveJob. It’s built with SQLite in mind, but it should work with any database supported by ActiveRecord, including Postgres. It might be a good fit for you if you want a lightweight, polling, Rails-only, multi-threaded background job processor.