Spending too much money is actually a good thing (or at least a better thing)
|
|
Any rapidly expanding business is trying to learn things as quickly as possible. What works, what doesn't work, what lines of business are profitable and a good avenue to invest more in, and which are not and should be abandoned.
One easy place engineering leadership can catch itself on a snag is a premature focus on cost when learning and speed is actually the primary goal.
Take, for example, a startup trying to survive on ramen noodles and ice cream bars. The technical co-founder reads an article on Hacker News about some whizbang new framework, datastore or other piece of tech stack and thinks, well golly, we could save 50% off of our Heroku bill if we "just used X".
Cut to 6 months later, the transition to X is in a half-usable state with elements of the thing X was supposed to replace still hanging around, the cost savings never quite materialized, and now 6 months of runway has been spent and 6 months of features that could deliver value to the customer were never delivered. Was the right decision made?
Or here's another story: a startup that's been doing alright. Things are going well, they've got a business model that they think is working OK. Then, a big new client is signed or a big sale occurs. The site explodes into flame, and the big client is lost or the sale doesn't go off well.
Both of these companies suffered from the same failure of prioritization.
For almost any web application, one of your first goals should be a deployment architecture that can scale from 0 to ~10,000 requests per minute with almost no human intervention. The cost at which this happens is not material. The ability to scale at any price is more important than the ability to scale at a low price.
Spending more money on technical resources is one of the easiest possible release valves you have as an engineering leader. It can buy you a quarter or even a year of engineering time to focus on other concerns. But you need to build that release valve. So many teams I've worked with can't pull this lever, and so end up micro-optimizing their stacks to save a buck here and there, or just to prevent the site from going down again.
Instead, build your web application so that:
- Compute can scale horizontally without human intervention. A dyno slider, an autoscaling group, a horizontal pod autoscaler, who cares. But it needs to be something that works automatically.
- Databases can scale vertically. Any application can reach into the mid-to-high 5 digits of requests-per-minute with a single-node SQL database and a key-value store. So don't overthink it - just pay Daddy Bezos to up yoru RDS plan again.
The great thing about spending on server resources is that you can always optimize it later. Unlike tech debt, which accrues interest, server spending rarely ever compounds. It's simply a cost that can be addressed later.
Also, once you've actually grown the feature and started spending lots of money to keep it running, that optimization work now is clearly high-leverage. It's easy to do the math on "well, if we optimized this feature a bit, we'd reduce our AWS spending by 50% which is $XX,XXX." But it's much harder, nigh impossible, to predict the future cost savings of tech decisions you're making.
What are the alternatives to spending first and then optimizing later?
- Underbaked architecture that doesn't scale. This is putting your head in the sand and hoping you can get by with one server compute node forever. It can work, but only for a while. Eventually, changes in traffic outstrip your capacity to keep up with the manual labor required to make this happen, and then revenue gets impacted as the site goes down under the load.
- Losing time spent learning from the customer. Engineers love to sit in their ivory towers and build The Perfect System that no one will ever need.
Spending too much money is a much better problem to have than not having any money to spend at all.
Until next time,
-Nate
|
|
|
|