Jake Worth

• bugs
• testing

“How did this feature break? I thought we were writing tests.” –Frustrated Boss

We spent the time writing tests, and yet, a bug survived. Should we just stop writing tests?

No, but we should maybe write better tests, and think about them differently.

In this post, I’d like to explain a mental model I have that helps me write and maintain tests: thinking of bugs, and the tests that cover them, in classes.

Bugs and Tests in Classes

Every test has limitations. Unit tests miss the big picture, end-to-end (E2E) tests miss the details. Rather than chasing the impossible, that the test suite catches all bugs, I’ve found it’s preferable to think about bugs and tests in classes, like the classifications of the animal kingdom.

Here are a few common classes of bugs:

• 🐛 Low-level bugs: typos, compilation errors, incorrect but compilable syntax
• 🐞 Logical bugs: sum(2, 2) returning undefined, “You have NaN new messages!”
• 🐜 Integration bugs: cancelMeeting called instead of createMeeting
• 🦋 End-to-end bugs: from midnight to 1 AM, the API endpoint returns 422

These are classes of bugs, coverable only by the right classes of tests. Some examples:

• 🐛 Low-level bugs: linting and formatting, static analysis, type-checking
• 🐞 Logical bugs: unit tests
• 🐜 Integration bugs: component tests
• 🦋 End-to-end bugs: E2E tests

Linting, formatting, static analysis, and type checking are great at catching dumb mistakes. Unit tests catch logical mistakes. Component tests catch integration mistakes, such as the wrong function being called when a button is clicked. And E2E tests catch, at a high cost, the fault lines between systems.

And none of these are good at catching anything else. Linting won’t catch a broken production API. E2E tests will pass when a string constructing function is replaced with hard-coded copy. You need the right class of test to catch the right the class of bug.

Benefits & Applications

Thinking in classes has a few benefits and applications.

First, it’s empowering! We replace “everything is broken” with “our date function doesn’t handle null.” The latter prescribes its remedy– write a test where the input is null. Thinking of bugs in classes makes a big problem (software has bugs) smaller (untested inputs produce unpredictable behavior).

Second, it’s a testing strategy that works. Kent C. Dodds calls this breakdown “The Testing Trophy”. If you want a solid test suite in a real-world application, it needs to at least cover the four big classes I described above. In a JavaScript codebase, this could mean:

• TypeScript type checking and Prettier running on Git pre-commit and CI
• Unit tests running on file write and CI
• Component tests running on file write and CI
• Cypress integration tests sparingly covering the most crucial paths through the software
• Green builds required to merge code

Even with these classes of tests, bugs still slip through. We tested what we thought mattered, missed things, learned, and added more cases. Thinking in classes gives you a set of tools to extend those cases with confidence.

What are your thoughts on classifying bugs? Let me know!