Menu

Subscribe

Scaling Engineering Teams via RFCs: Writing Things Down

I have recently been talking at small and mid-size companies, sharing engineering best practices I see us use at Uber, which I would recommend any tech company adopt as they are growing. The one topic that gets both the most raised eyebrows, as well the most "aha!" moments is the one on how the planning process for engineering has worked since the early years of Uber.

When working at large companies like Microsoft or smaller ones like Skyscanner, there have been two things related to planning that have always bugged me. First, the lack of visibility on others building or having built the same thing as my team. Second, the tech and architecture debt accumulated due to different teams building things very differently, both approach-wise and quality-wise.

What if I said there is a way to tackle both these issues pretty well, using a few simple steps? A word of warning, one of the steps will sound a little crazy. Here they are:

  1. Do planning before building something new. This can be in-person whiteboarding or just talking it through with the team members, as long as you're all clear on how you will get things done.
  • Capture this plan in a short, written document. Once it is clear to the team how and what you do, it should be relatively quick to write down the "how". Don't go overboard.
  • Have a few, select people approve this plan before starting work. Similar to how it's a good quality gate to only merge a pull request after someone does a review on it, it makes a huge difference if before starting work on a project, a few, relevant people validate the planned work. These can be senior engineers, people on a team who will use the feature and so on.
  • Send this planning document out to all engineers in the company and let anyone and everyone comment on it. Yes, this the step that probably sounds crazy.
  • Have everyone follow the above steps for every project that is beyond some super trivial complexity and iterate on a lightweight, RFC-like process so it works well for your org or company.

As unlikely it might sound, the above process both works and scales really well, from a handful of engineers to teams of thousands. It addresses not only issues on visibility or reducing tech/architecture debt, but also spreading knowledge and having engineers be more engaged day to day. This is the one, simple process I recommend any small or medium tech team to do, especially if they are in a growth phase. It's also the process we've successfully used and iterated on at Uber, going from tens of engineers to a couple of thousand ones.

The power of writing things down

Writing and sharing that writing with others creates accountability. It also almost always leads to more thorough decisions. A simple way to increase code quality? Do code review in writing, before merging. A simple way to have a meeting be less of a waste of time? Have a written agenda before the meeting, then write up and send out decisions and actions afterwards. A simple way to run projects with fewer surprises? Have the team write down what they are planning to do and share it with others.

Us, engineers hate time wasters. Documentation is often seen as one of these time wasters, mostly because it is boring to do. Planning is often seen as a kind of documentation, so there is a natural tendency to just skip this step for efficiency. I like to flip this argument around on saving time, though.

If everyone agrees how the project should be done then writing the approach down should be a piece of cake. It just needs to capture the shared understanding on what the approach is, what architecture changes will be made, what the tricky parts are. Someone from the team should be able to do this in a few hours, the other team members giving a thumbs-up after reading through it. Usually, I see this being less straightforward. "This is not what I meant when we talked." or "What about this important edge case we forgot?" and "If we change this here, it could break this other part of the system" are things that often come up when writing the plan down. It's great to have these discussions before having the same realizations when the project is halfway done.

What about when people don't agree with how the project should be done or there are just a lot of changes? That already sounds like a project that will take a lot longer than people think - writing down things at least should give a clearer picture.

Reviewers and spreading knowledge across the organization

While writing is a good way to organize one's thoughts, it is a very different exercise when we write for someone else to understand. It's a good idea to specify who needs to read this document and give a thumbs-up. Once the plan is written-up, the surest way to make sure the people actually read the doc is to require them confirming just this - in writing, via e.g. a comment.

The part of sending out the plan on how this project will be built, to all the engineering organization, via email, is something that can sound crazy. People often worry about this creating a lot of noise. This is both true and not. The noise levels will definitely increase. Thankfully, email filters are easy to create - just make it easy to filter for these kinds of emails. At the same time, there are a lot less project plans being sent out, even with an engineering size in the hundreds or thousands, than people typically expect.

The type of information pushed to people in an organization shapes the culture considerably. Having an organization with a culture to push engineering plans to everyone - for example, via email - and invite anyone to comment sets a tone of trust and responsibility. During my first few months at Uber, I used to obsess on reading planning docs from a variety of teams, chiming into ones where I had relevant domain knowledge e.g. on the technology. In these first few months, I learned a lot more about what other teams worked on, than I did in years working at a large company where this information was not visible. To this day, I still look through plans that teams make, getting a sense of what is going on outside my little bubble.

Finally, allowing anyone and everyone to chime in is a key part of keeping a consistent engineering bar across the organization. At Uber, I have seen multiple cases of people on one side of the organization realize that another team on the other side of the organization is planning to do something similar to what they have done, but with a very different approach, often with a different quality bar. For example, a US team building a new feature might have not considered other parts of the world and a team in India pointing out gaps in their localization approach. The teams having transparency, self-balancing and self-correction happens quite naturally.

Tailoring the process via iteration

At Uber, a process described here has been used from the early years. Details of how this planning and review process has worked has been refined as the company grew and matured. What started out as an email to every engineer changed into mailing lists per domain (backend, mobile, web) and templates built by engineers to help convey information in a more consistent matter. As the company is growing to the thousands of engineers, more tooling is being built to make the process of searching and approving even easier.

An interesting part of the iteration I've seen is the evolution of templates. People who review many engineering proposals often had the same type of questions. Questions like "What is the motivation to do this work?" or "How will this be tested?" or "Will any archtiecture changes be made here?" were very common questions. Seeing the repeat on these, engineers came up with templates that keep being updated to make reading and writing these plans be easy. To give an example, this is roughly how the backend and frontend (mobile/web) templates evolved about a year ago, since when we switched to generated templates:

Backend

  • List of approvers
  • Abstract (what is the project about?)
  • Architecture changes
  • Service SLAs
  • Service dependencies
  • Load & performance testing
  • Multi data-center concerns
  • Security considerations
  • Testing & rollout
  • Metrics & monitoring
  • Customer support considerations

Mobile/Web

  • List of approvers
  • Abstract (what is the project about?)
  • UI & UX
  • Architecture changes
  • Network interactions detailed
  • Library dependencies
  • Security concerns
  • Testing & rollout
  • Analytics
  • Customer support considerations
  • Accessibility

Iterating and customizing to the needs of the engineering team is key. In our case, the templates started to include important things we cared about. Things like reliability, scale, security. At Uber, we've built many smaller services, so things to consider when doing so - like load and performance testing or SLAs - are part of that domain. When accessibility became a big focus for mobile, that section made it into the template. You get the idea.

A process that scales

At Uber, we called this process RFC - Request for Comments, given the many similarities it has with the Request For Comments publication process in the tech community. The process stuck because it was started early enough - when the engineering team was small - and it helped scale knowledge and eliminate silos, as the company kept growing rapidly. It has scaled remarkably well, from tens of engineers to the low thousands. Starting at a few thousand engineers, new challenges arise with this kind of process, which is a good problem to have by itself.

To keep all of this from being overly theoretical, you can see our RFC process happen in the open by checking some of our open source projects, such as BaseUI. BaseUI is web a design system comprised of modern, responsive, living components. All RFCs are published and reviewed here and here. Development only starts after the RFC is approved.

So is this process unique to Uber? Far from it. Larger tech companies, like Google, Facebook, Microsoft or Amazon all came up with something that resembles some level of RFC process. Google calls it design docs and here's a good overview of how it works. Architecture Decision Records - ADRs - are also a good format to use.

The scale on how thorough the documentation should be, who the docs are pushed to, how strictly it is reviewed or how standardized it is across different orgs is different for all companies. If your at a tech company where planning is still adhoc, without a structured way on recording/distributing it, one of the things that will help engineering scale greatly is having a type of RFC process in place early on. Just give it a try and iterate on how you do it. That's how it always starts.

Featured Pragmatic Engineer Jobs

The above jobs score at least 9/12 on The Pragmatic Engineer Test. Browse more senior engineer and engineering leadership roles with great engineering cultures, or add your own on The Pragmatic Engineer Job board and apply to join The Pragmatic Engineer Talent Collective.