The test setup that went bump in the night
|
|
We don't think about our tests enough.
Tests are often the garbage bin of the codebase. If we're lucky enough that they exist at all. They get less attention, less love, and less refactoring. We treat tests as a checklist item to be checked and then forgotten about.
This leads to 30 minute test suites. I've said before that I think that all tests can run at 100 assertions (or expects) per second or faster. But most suites are considerably slower than that?
Why? It's tempting to say "the database", but actually, that's not it. It's factories.
Many test suites spend 50% or more of their time in FactoryBot. How does this happen? Is it really because we need to `create()` thousands of records to run our suite?
The advice is frequently to "just replace create() with build() or build_stubbed()". Yes, we can do that, sometimes, and the performance benefits are good. For example, consider this real-world test from Rubygems.org:
class RubygemTest < ActiveSupport::TestCase
context "with a saved rubygem" do
setup do
@rubygem = create(:rubygem, name: "SomeGem")
end
subject { @rubygem }
should have_many(:owners).through(:ownerships)
end
end
If you've never seen this syntax before, they're using shoulda-context.
Think about the behavior under test here - do we need a persisted Rubygem record to test whether or not that relationship exists? Nope. That relationship exists regardless, because it's defined when the class is defined.
Rubygem.new.owners
=> #<ActiveRecord::Associations::CollectionProxy []>
Unfortunately, what I showed you isn't the real file. The real file looks more like this:
setup do
@rubygem = create(:rubygem, name: "SomeGem")
end
subject { @rubygem }
should have_many(:owners).through(:ownerships)
should have_many(:ownerships).dependent(:destroy)
should have_many(:subscribers).through(:subscriptions)
should have_many(:subscriptions).dependent(:destroy)
# so on and so forth, for about 2 dozen lines.
Now, when we run this suite, how many Rubygem records will be created?
I find that many Rubyists don't understand that test setup runs before every test. We understand that the state is the same for every test, but we don't understand that the setup block runs before every test to make that happen.
So, for the four tests above I quoted, we've created a Rubygem record four times (one time per test), each time unnecessarily.
I think many test-writer's mental model seems to be "I wrote setup once, so it runs once." Not true. It's setup-test, setup-test, setup-test.
In this way, inefficient test setup compounds across every test. In this case, the fix is simple. We don't need to involve FactoryGirl at all. Let's just initialize a new object - much simpler, and the exact class of the object under test is now perfectly clear:
setup do
@rubygem = Rubygem.new(name: "SomeGem")
end
That change sped up the entire test class by 10%. A nice gain.
When writing test setup, think about the behavior under test. Minimize setup wherever possible.
Until next week,
-Nate
|
|
|
|
