By Theresa Eades

Jul 19, 2022

Introduction

The first time I encountered it, Atul Gawande’s The Checklist Manifesto made intuitive sense to me. The book's thesis is straightforward: using checklists can lead to greater efficiency, consistency, and safety in the execution of complex tasks. Gawande, a surgeon, writes

our great struggle in medicine these days is not just with ignorance and uncertainty. It's also with complexity: how much you have to make sure you have in your head and think about. There are a thousand ways things can go wrong.

Gawande’s focus is on his effort to define a standard surgery checklist. This checklist would eliminate whole categories of all too common mistakes and errors of omission. Even experts need help, Gawande says. “It’s just too easy for an otherwise competent doctor to miss a step, or forget to ask a key question or, in the stress and pressure of the moment, to fail to plan properly for every eventuality.”

By focusing a doctor's attention on basic, unmemorable tasks – have we checked whether the patient has any known allergies? – Gawande's safe surgery checklist has been a resounding success. Using the checklist has reduced surgical complications by a third, and at no extra cost. I’m a software engineer, not a doctor, but I also execute complex, multi-step routines under time and sometimes resource pressure. How many times had I planned out my work, thought through various edge cases, and then forgotten a fundamental task when implementation was almost complete? Two years ago, when my team worked on adding video to the listing page, we had to restart the experiment because we forgot to include specific event logging for our feature. Last year we implemented the key functionality for an experiment with the "add to cart" button, but neglected to get accessibility and text strings translated early, costing us a delay till the translations could complete. If I had started with a checklist, we could have knocked out event logging and translation without having to think twice.

On my team, engineering work begins with a task called a “spike.” We receive specs from our product manager and designer and then spend time digging into the engineering requirements of the work. A fellow engineer created a loose template doc for spikes and that was when inspiration struck. I recalled Gawande’s manifesto and realized that this was the perfect occasion for a checklist! I reflected on the pitfalls I had encountered before in our process and on the common tasks required for product experiments at Etsy: population eligibility, translations, and event logging for starters. Now add in feature implementation, testing, and accessibility considerations and it becomes too much to keep in your head. If using something as simple a checklist can decrease surgery failures by a third, what could they do for my product experiments?

Without further ado, here is my own Checklist Manifetsy (at Etsy, we love our puns)!

Checklist

Scope

• Is the feature meant to be very polished and finished or are we just trying to get user feedback as an MVP?
• If we are running a MVP, is the current feature a true MVP? How can we simplify or cut scope?

Eligibility

• What populations should be included or excluded from the experiment? When should users see this feature? (Which pages, signed in/signed out, mobile, desktop, etc.)
• Where/when should bucketing occur?
• Will the experiment conflict with any other experiments? Do the experiments need to run exclusively?
• What countries should the experiment run in (can impact translations)?

A11Y

• Is there any special accessibility work this feature will require? If extra work is anticipated, check in early with our a11y team.
• When testing and developing we should keep two users in mind - a keyboard user and a voice over user, do we need to add other code for these users?

Translations

• Are there any strings to be translated that should be submitted ASAP?
• Do we need to translate any labels for a11y?

Observability

• How will we know that the feature is working? Are there existing graphs we can use or do we need new ones?
• Should any of these metrics have a threshold or alerting?
• Are we missing any key events to obtain user feedback?
• How will we compare our control and variant?

Performance

• Is there anything in my experiment that could degrade performance of the site?
• Do I need an operational experiment to verify that I’m not impacting performance?

Error States

• Do we have designs for loading states?
• Do we have designs for unsuccessful requests and error handling?
• Do we have informative logging when there are errors?

QA

• What set of browsers and devices should we test our new feature against?
• Which user perspectives do we need to test?

Ramping

• What will our ramping strategy be?

Takeaways

A checklist is a generic tool, but mine has become personal over time. As it evolves, it serves as a reminder that I’m growing in my craft. (I could actually tell you a story about each question!) The checklist has become an extension of my practice. When I’m planning out a track of work, I’ll make a new copy and start taking notes. Sometimes I can dismiss a whole section with a simple no. Sometimes I don’t know an answer and need to dig deeper, or ask for a design update. After the experiment is complete, I’ll reflect on how it went. If we made a preventable mistake, I’ll revisit the checklist and add in my learnings. This ensures that we remain nimble and adaptable in our process.

This specific checklist won’t work for all teams and all projects. It’s one that I’ve honed over a few years in product engineering at Etsy. Even though my checklist is specific to me, it’s a great way to get new team members up to speed on our experimentation process. If a teammate uses the checklist and doesn’t understand an item, it becomes an opportunity to tell a story and socialize knowledge. Checklists are not new to Etsy! There are a few internal ones that have inspired me in my own list. In particular, looking at lists that other teams use has been an efficient way to learn about my own knowledge gaps. I hope this has inspired you to use checklists in your work, the same way The Checklist Manifesto inspired me!