Existing by coincidence, programming deliberately
Historically, the Firefox Accounts (FxA) codebase was organised as separate repositories because it's deployed as separate microservices in production. However, that arrangement caused us some nagging issues as developers, so we decided to migrate the code to a monorepo instead. This post discusses why and how we did that, and how things turned out in the end.
There were three main problems with the many-repo approach.
Firstly,
it was hard to share common code
between services.
Generally we addressed that
by extracting the re-usable logic
to yet another repository
and publishing it from there with npm.
But each additional repository
increased the mental load
of working on FxA
and updating the common code
became a tiresome process
that involved opening pull requests
in multiple locations
to grab the latest dependency.
Because different subsets of the team worked in different repositories, coding conventions gradually diverged between them. We could have averted this situation with stricter lint rules shared between repos, but by the time we realised there was a problem, there was a problem. The respective owners of each codebase were not enthusiastic about making wide-ranging formatting changes to their code, so anyone working horizontally was forced to chop and change between local customs instead.
The most serious issues were in CI. It was possible for services to break downstream consumers of their interfaces, without causing any builds to fail. This would sometimes lead to downstream developers discovering failures while working on unrelated tasks and lacking any context to help them understand what was wrong.
With the imminent arrival of new recruits on the FxA team, we decided to take the hit and pull all of the code in to one place.
Our hard requirements for the move were that all of the commit histories must be preserved, the most recent tag for each service should be copied too and that CI should run tests for all downstream consumers when their upstream dependencies were changed. Danny Coates took the bull by the horns and wrote a script to do the actual work. We played with that until we persuaded ourselves we'd thought things through properly, then we fixed a date to migrate the code for real.
There could be no going back after that point, as issues were to be moved, pull requests re-opened, code landed and old repositories archived. We opted to do the move immediately after cutting the release for FxA train 134, to give ourselves some breathing space to work through any problems before cutting train 135. Or at least, that was the theory. In practice, we had to cut point releases for train 134 so a lot of that breathing space was lost.
After the code was moved, the next priority was to ensure we had a release script that worked from the new monorepo. This meant working for both JavaScript and Rust services, so probably warrants a separate blog post in its own right, but you can find the finished script here.
It's fair to say we underestimated how many problems might occur as a result of this migration. For example, some of the unexpected issues were:
The git hooks we had in our old repos were forced to compete against each other in the new monorepo. Lacking the time to make them co-exist happily, we just deleted them.
The version endpoint for one of our services
broke because the relative path
to the .git directory
changed.
Fixing it was straightforward.
Some of our tests cloned the database repo in their setup. That repo is now archived and becoming ever staler with each passing commit, so we quickly applied a sticking-plaster fix to copy the local db directory instead but are yet to land a long-term fix that we're really happy with.
Additionally, we haven't yet done some of the things the monorepo was designed to enable, such as standardise a mechanism for cross-package dependencies or unify the lint rules for all our packages.
Ultimately, it's become clear the migration will be a gradual process carried out over a number of months rather than an instant cutover. The long-term benefits are all still available but the journey to reach them is ongoing!