One of the great mysteries is how we as a species are able to to get anything done. Is it because we are so good at tricking ourselves to believe it’s going to be easy?

Some things are very difficult and very expensive (however you measure cost). Many structures built centuries ago took centuries to build. For example. the Colonge Cathedral wasn’t finished until 600 years after construction started. Which projects starting this year are likely to be finished after 2222?

Perhaps one of the reasons why anything gets built at all is that our brains need to underestimate the work involved at the beginning of a project and once we discover how hard things are, we’re too emotionally invested to give up. My book would never have been started if I had known the costs involved up-front, nor would it have been completed if I hadn’t already done so much work on it and promised so many people that it would be done. Even when part of my brain tried to tell me that the sunk cost fallacy was real, I kept moving forward.

Software engineers appear to particularly bad at this. Or, at least, they’re much more vocal about how terrible other people’s software is and how much cleaner their implementation would be. I mean, curl could be written in a weekend. Twitter could be written in a weekend. This one is so common, it’s almost a meme.

It’s terrible that we don’t immediately stop ourselves when we begin to believe things like this. We wonder why we can’t make accurat estimates. And yet I keep entertaining ideas like rewriting Apache Kafka, RabbitMQ or a database. This week it’s been an itch to write an ActivityPub server.

Formally, this phenomenon is known as the Planning Fallacy. Even people who study its effects are prey. It was one of the first cognitive biases discussed by Daniel Kahneman and Amos Tversky in 1979. To counter-act it, here are two strategies:

• write a plan and commit - once you’ve done the analysis and understood the space, you’re actually less likely to fall behind [doi:10.3758/MC.36.4.791]
• smaller tasks are easier to estimate - try to split the project into as many tasks as possible [doi:10.1002/1099-0992(200011/12)30:6%3C873::AID-EJSP22%3E3.0.CO;2-U]